On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley <dashley@gmail.com> wrote:> >Without FPU support, assuming that the processor has basic integer >multiplication instructions, integer operations are ALWAYS faster than >floating-point operations. Usually _far_ faster. And always more >precise.Floating point instructions MUL/DIV are trivial, just multiply/divide the mantissa and add/sub the exponent. With FP add/sub you have to denormalize one operand and then normalize the result, which can be quite time consuming, without sufficient HW support. This can be really time consuming, if the HW is designed by an idiot.
Floating point vs fixed arithmetics (signed 64-bit)
Started by ●March 26, 2012
Reply by ●March 27, 20122012-03-27
Reply by ●March 27, 20122012-03-27
On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote:>On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley ><dashley@gmail.com> wrote: > >> >>Without FPU support, assuming that the processor has basic integer >>multiplication instructions, integer operations are ALWAYS faster than >>floating-point operations. Usually _far_ faster. And always more >>precise. > >Floating point instructions MUL/DIV are trivial, just multiply/divide >the mantissa and add/sub the exponent. > >With FP add/sub you have to denormalize one operand and then normalize >the result, which can be quite time consuming, without sufficient HW >support. > >This can be really time consuming, if the HW is designed by an idiot.Your observations are valid. But I have yet to see a practical example of something that can be done faster and with equal accuracy in floating point vs. using integer operations. I concur with your observations. After reading your first paragaph ... yeah, floating-point multiplication is pretty simple so long as the floating point format is sane. Before reading your post, I my mental model was that floating-point operations might be 20 times as slow as integer operations. Now I'm thinking maybe 2-3 times. DTA.
Reply by ●March 27, 20122012-03-27
"David T. Ashley" wrote:> On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote: > > >On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley > ><dashley@gmail.com> wrote: > > > >> > >>Without FPU support, assuming that the processor has basic integer > >>multiplication instructions, integer operations are ALWAYS faster than > >>floating-point operations. Usually _far_ faster. And always more > >>precise. > > > >Floating point instructions MUL/DIV are trivial, just multiply/divide > >the mantissa and add/sub the exponent. > > > >With FP add/sub you have to denormalize one operand and then normalize > >the result, which can be quite time consuming, without sufficient HW > >support. > > > >This can be really time consuming, if the HW is designed by an idiot. > > Your observations are valid. But I have yet to see a practical > example of something that can be done faster and with equal accuracy > in floating point vs. using integer operations. > > I concur with your observations. After reading your first paragaph > ... yeah, floating-point multiplication is pretty simple so long as > the floating point format is sane. > > Before reading your post, I my mental model was that floating-point > operations might be 20 times as slow as integer operations. Now I'm > thinking maybe 2-3 times.I did a fixed point support package for our 8 bit embedded systems compilers and one interesting metric came out of the project. Given a number of bits in a number and similar error checking fixed or float took very similar amounts of execution time and code size in applications. For example 32 bit float and 32 bit fixed point. They are not exact but they are close. In the end much to my surprise the choice is dynamic range or resolution. There are other factors IEEE754 has potentially much more error checking but not all libraries a written to support it, and not applications need it. Regards, w.. -- Walter Banks Byte Craft Limited http://www.bytecraft.com
Reply by ●March 27, 20122012-03-27
On Tue, 27 Mar 2012 13:56:49 -0500, Walter Banks wrote:> "David T. Ashley" wrote: > >> On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote: >> >> >On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley >> ><dashley@gmail.com> wrote: >> > >> > >> >>Without FPU support, assuming that the processor has basic integer >> >>multiplication instructions, integer operations are ALWAYS faster >> >>than floating-point operations. Usually _far_ faster. And always >> >>more precise. >> > >> >Floating point instructions MUL/DIV are trivial, just multiply/divide >> >the mantissa and add/sub the exponent. >> > >> >With FP add/sub you have to denormalize one operand and then normalize >> >the result, which can be quite time consuming, without sufficient HW >> >support. >> > >> >This can be really time consuming, if the HW is designed by an idiot. >> >> Your observations are valid. But I have yet to see a practical example >> of something that can be done faster and with equal accuracy in >> floating point vs. using integer operations. >> >> I concur with your observations. After reading your first paragaph ... >> yeah, floating-point multiplication is pretty simple so long as the >> floating point format is sane. >> >> Before reading your post, I my mental model was that floating-point >> operations might be 20 times as slow as integer operations. Now I'm >> thinking maybe 2-3 times. > > I did a fixed point support package for our 8 bit embedded systems > compilers and one interesting metric came out of the project. > > Given a number of bits in a number and similar error checking fixed or > float took very similar amounts of execution time and code size in > applications. > > For example 32 bit float and 32 bit fixed point. They are not exact but > they are close. In the end much to my surprise the choice is dynamic > range or resolution. > > There are other factors IEEE754 has potentially much more error checking > but not all libraries a written to support it, and not applications need > it.That's interesting, because in my experience fixed-point fractional arithmetic (i.e., 0x7fffffff = 1 - 2^-31, 0x80000001 = -1 + 2^-31), with saturation-on-add, is significantly faster (3x to 10x) than floating point on all the machines I've tried it except for those with floating- point hardware. I have a portable version that works on just about anything that's ANSI-C compatible, and when I really need speed I rewrite the arithmetic routines in assembly for about a 2x increase. The only processor that came close to matching it was the TMS320F2812, where we used the ANSI-C compatible version that was just about matched by the floating-point package that came with the tool set (and I _know_ that TI cut corners with that floating point package). That's the _only_ processor in my experience where the floating point could keep up with the ANSI-C version, and I would expect that had I written an assembly version it would have been faster yet. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
Reply by ●March 27, 20122012-03-27
Tim Wescott wrote:> On Tue, 27 Mar 2012 13:56:49 -0500, Walter Banks wrote: > > > "David T. Ashley" wrote: > > > >> On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote: > >> > >> >On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley > >> ><dashley@gmail.com> wrote: > >> > > >> > > >> >>Without FPU support, assuming that the processor has basic integer > >> >>multiplication instructions, integer operations are ALWAYS faster > >> >>than floating-point operations. Usually _far_ faster. And always > >> >>more precise. > >> > > >> >Floating point instructions MUL/DIV are trivial, just multiply/divide > >> >the mantissa and add/sub the exponent. > >> > > >> >With FP add/sub you have to denormalize one operand and then normalize > >> >the result, which can be quite time consuming, without sufficient HW > >> >support. > >> > > >> >This can be really time consuming, if the HW is designed by an idiot. > >> > >> Your observations are valid. But I have yet to see a practical example > >> of something that can be done faster and with equal accuracy in > >> floating point vs. using integer operations. > >> > >> I concur with your observations. After reading your first paragaph ... > >> yeah, floating-point multiplication is pretty simple so long as the > >> floating point format is sane. > >> > >> Before reading your post, I my mental model was that floating-point > >> operations might be 20 times as slow as integer operations. Now I'm > >> thinking maybe 2-3 times. > > > > I did a fixed point support package for our 8 bit embedded systems > > compilers and one interesting metric came out of the project. > > > > Given a number of bits in a number and similar error checking fixed or > > float took very similar amounts of execution time and code size in > > applications. > > > > For example 32 bit float and 32 bit fixed point. They are not exact but > > they are close. In the end much to my surprise the choice is dynamic > > range or resolution. > > > > There are other factors IEEE754 has potentially much more error checking > > but not all libraries a written to support it, and not applications need > > it. > > That's interesting, because in my experience fixed-point fractional > arithmetic (i.e., 0x7fffffff = 1 - 2^-31, 0x80000001 = -1 + 2^-31), with > saturation-on-add, is significantly faster (3x to 10x) than floating > point on all the machines I've tried it except for those with floating- > point hardware. > > I have a portable version that works on just about anything that's ANSI-C > compatible, and when I really need speed I rewrite the arithmetic > routines in assembly for about a 2x increase. > > The only processor that came close to matching it was the TMS320F2812, > where we used the ANSI-C compatible version that was just about matched > by the floating-point package that came with the tool set (and I _know_ > that TI cut corners with that floating point package). That's the _only_ > processor in my experience where the floating point could keep up with > the ANSI-C version, and I would expect that had I written an assembly > version it would have been faster yet.What you saw was what I was expecting. My points in the post was to be careful in assuming that fixed is going to be dramatically better. At least for 8 bits the variable size in bits is a significant factor when all math is multiprecision. One of the keys in our metrics was the target was 8 bit processors and there was an exchange between precision and dynamic range but the bit sizes remained the same. Real applications are probably dominated by scaling and precision reducing the number of bits used by fixed point for the same application. It didn't make sense until I realized that it was 8 bit processors using software mults and divides and 32 bit floating point uses for the most part 24bit mults and divides and a few adds/subtracts for the exponents. 32 bit fixed point uses 32 bit mults/divides adding to the cycle count. My experience with 32 bit processors is similar to yours although I don't have metrics to back it up. Walter..
Reply by ●March 28, 20122012-03-28
On Tue, 27 Mar 2012 15:35:14 -0500, Walter Banks wrote:> Tim Wescott wrote: > >> On Tue, 27 Mar 2012 13:56:49 -0500, Walter Banks wrote: >> >> > "David T. Ashley" wrote: >> > >> >> On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote: >> >> >> >> >On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley >> >> ><dashley@gmail.com> wrote: >> >> > >> >> > >> >> >>Without FPU support, assuming that the processor has basic integer >> >> >>multiplication instructions, integer operations are ALWAYS faster >> >> >>than floating-point operations. Usually _far_ faster. And always >> >> >>more precise. >> >> > >> >> >Floating point instructions MUL/DIV are trivial, just >> >> >multiply/divide the mantissa and add/sub the exponent. >> >> > >> >> >With FP add/sub you have to denormalize one operand and then >> >> >normalize the result, which can be quite time consuming, without >> >> >sufficient HW support. >> >> > >> >> >This can be really time consuming, if the HW is designed by an >> >> >idiot. >> >> >> >> Your observations are valid. But I have yet to see a practical >> >> example of something that can be done faster and with equal accuracy >> >> in floating point vs. using integer operations. >> >> >> >> I concur with your observations. After reading your first paragaph >> >> ... yeah, floating-point multiplication is pretty simple so long as >> >> the floating point format is sane. >> >> >> >> Before reading your post, I my mental model was that floating-point >> >> operations might be 20 times as slow as integer operations. Now I'm >> >> thinking maybe 2-3 times. >> > >> > I did a fixed point support package for our 8 bit embedded systems >> > compilers and one interesting metric came out of the project. >> > >> > Given a number of bits in a number and similar error checking fixed >> > or float took very similar amounts of execution time and code size in >> > applications. >> > >> > For example 32 bit float and 32 bit fixed point. They are not exact >> > but they are close. In the end much to my surprise the choice is >> > dynamic range or resolution. >> > >> > There are other factors IEEE754 has potentially much more error >> > checking but not all libraries a written to support it, and not >> > applications need it. >> >> That's interesting, because in my experience fixed-point fractional >> arithmetic (i.e., 0x7fffffff = 1 - 2^-31, 0x80000001 = -1 + 2^-31), >> with saturation-on-add, is significantly faster (3x to 10x) than >> floating point on all the machines I've tried it except for those with >> floating- point hardware. >> >> I have a portable version that works on just about anything that's >> ANSI-C compatible, and when I really need speed I rewrite the >> arithmetic routines in assembly for about a 2x increase. >> >> The only processor that came close to matching it was the TMS320F2812, >> where we used the ANSI-C compatible version that was just about matched >> by the floating-point package that came with the tool set (and I _know_ >> that TI cut corners with that floating point package). That's the >> _only_ processor in my experience where the floating point could keep >> up with the ANSI-C version, and I would expect that had I written an >> assembly version it would have been faster yet. > > What you saw was what I was expecting. My points in the post was to be > careful in assuming that fixed is going to be dramatically better. At > least for 8 bits the variable size in bits is a significant factor when > all math is multiprecision. > > One of the keys in our metrics was the target was 8 bit processors and > there was an exchange between precision and dynamic range but the bit > sizes remained the same. > > Real applications are probably dominated by scaling and precision > reducing the number of bits used by fixed point for the same > application. > > It didn't make sense until I realized that it was 8 bit processors using > software mults and divides and 32 bit floating point uses for the most > part 24bit mults and divides and a few adds/subtracts for the exponents. > 32 bit fixed point uses 32 bit mults/divides adding to the cycle count. > > My experience with 32 bit processors is similar to yours although I > don't have metrics to back it up.Ah. I see your point. 9 multiplies and some shifting during addition vs. 16 multiplies might well turn out to be a wash. The first serious control loop I did was quite starved for clock cycles, and used a 24-bit accumulator, but with an 8 x 16 (or 8 x 8) multiply, and had 16-bit data paths other than that. -- Tim Wescott Control system and signal processing consulting www.wescottdesign.com
Reply by ●March 28, 20122012-03-28
On 27/03/2012 20:56, Walter Banks wrote:> > > "David T. Ashley" wrote: > >> On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote: >> >>> On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley >>> <dashley@gmail.com> wrote: >>> >>>> >>>> Without FPU support, assuming that the processor has basic integer >>>> multiplication instructions, integer operations are ALWAYS faster than >>>> floating-point operations. Usually _far_ faster. And always more >>>> precise. >>> >>> Floating point instructions MUL/DIV are trivial, just multiply/divide >>> the mantissa and add/sub the exponent. >>> >>> With FP add/sub you have to denormalize one operand and then normalize >>> the result, which can be quite time consuming, without sufficient HW >>> support. >>> >>> This can be really time consuming, if the HW is designed by an idiot. >> >> Your observations are valid. But I have yet to see a practical >> example of something that can be done faster and with equal accuracy >> in floating point vs. using integer operations. >> >> I concur with your observations. After reading your first paragaph >> ... yeah, floating-point multiplication is pretty simple so long as >> the floating point format is sane. >> >> Before reading your post, I my mental model was that floating-point >> operations might be 20 times as slow as integer operations. Now I'm >> thinking maybe 2-3 times. > > I did a fixed point support package for our 8 bit embedded systems > compilers and one interesting metric came out of the project. > > Given a number of bits in a number and similar error checking fixed > or float took very similar amounts of execution time and code size > in applications.That's not a big surprise - with floating point, the actual arithmetic is 24-bit, which will be quite a lot faster than 32-bit on a small 8-bit machine (especially if it doesn't have enough registers or data pointers).> > For example 32 bit float and 32 bit fixed point. They are not exact > but they are close. In the end much to my surprise the choice is > dynamic range or resolution. > > There are other factors IEEE754 has potentially much more error > checking but not all libraries a written to support it, and not > applications need it. > > > Regards, > > > w.. > -- > Walter Banks > Byte Craft Limited > http://www.bytecraft.com > > >
Reply by ●March 28, 20122012-03-28
On 27/03/2012 19:02, David T. Ashley wrote:> On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote: > >> On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley >> <dashley@gmail.com> wrote: >> >>> >>> Without FPU support, assuming that the processor has basic integer >>> multiplication instructions, integer operations are ALWAYS faster than >>> floating-point operations. Usually _far_ faster. And always more >>> precise. >> >> Floating point instructions MUL/DIV are trivial, just multiply/divide >> the mantissa and add/sub the exponent. >> >> With FP add/sub you have to denormalize one operand and then normalize >> the result, which can be quite time consuming, without sufficient HW >> support. >> >> This can be really time consuming, if the HW is designed by an idiot. > > Your observations are valid. But I have yet to see a practical > example of something that can be done faster and with equal accuracy > in floating point vs. using integer operations. >It depends on the chip, the type of floating point hardware it has, the operations you need, the compiler, and the code quality. For a lot of heavy calculations done with integer arithmetic, you need a number of "extra" instructions as well as the basic add, subtract, multiply and divides. You might need shifts for scaling, mask operations, extra code to get the signs right, etc. And the paths for these are likely to be highly serialised, with each depending directly on the results of the previous operation, which slows down pipelining. With hardware floating point, you have a much simpler instruction stream, which can result in faster throughput even if the actual latency for the calculations is the same. This effect increases with the size and complexity of the processor. Obviously it is dependent on the processor having floating point hardware for the precision needed (single or double), but once you have any sort of hardware floating point you should re-check all your assumptions about speed differences. You could be wrong in either direction.> I concur with your observations. After reading your first paragaph > ... yeah, floating-point multiplication is pretty simple so long as > the floating point format is sane. > > Before reading your post, I my mental model was that floating-point > operations might be 20 times as slow as integer operations. Now I'm > thinking maybe 2-3 times. > > DTA.
Reply by ●March 28, 20122012-03-28
On Wed, 28 Mar 2012 09:17:14 +0200, David Brown wrote:> On 27/03/2012 19:02, David T. Ashley wrote: >> On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote: >> >>> On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley >>> <dashley@gmail.com> wrote: >>> >>> >>>> Without FPU support, assuming that the processor has basic integer >>>> multiplication instructions, integer operations are ALWAYS faster >>>> than floating-point operations. Usually _far_ faster. And always >>>> more precise. >>> >>> Floating point instructions MUL/DIV are trivial, just multiply/divide >>> the mantissa and add/sub the exponent. >>> >>> With FP add/sub you have to denormalize one operand and then normalize >>> the result, which can be quite time consuming, without sufficient HW >>> support. >>> >>> This can be really time consuming, if the HW is designed by an idiot. >> >> Your observations are valid. But I have yet to see a practical example >> of something that can be done faster and with equal accuracy in >> floating point vs. using integer operations. >> >> > It depends on the chip, the type of floating point hardware it has, the > operations you need, the compiler, and the code quality. For a lot of > heavy calculations done with integer arithmetic, you need a number of > "extra" instructions as well as the basic add, subtract, multiply and > divides. You might need shifts for scaling, mask operations, extra code > to get the signs right, etc. And the paths for these are likely to be > highly serialised, with each depending directly on the results of the > previous operation, which slows down pipelining. With hardware floating > point, you have a much simpler instruction stream, which can result in > faster throughput even if the actual latency for the calculations is the > same. > > This effect increases with the size and complexity of the processor. > Obviously it is dependent on the processor having floating point > hardware for the precision needed (single or double), but once you have > any sort of hardware floating point you should re-check all your > assumptions about speed differences. You could be wrong in either > direction.The key point is "it is dependent on the processor having floating point hardware for the precision needed". And, I might add, on other things -- see Walter Banks's comments in another sub-thread about 32-bit floating point vs. 32-bit integer math. In my experience with signal processing and control loops, having a library that implements fixed-point, fractional arithmetic with saturation on addition and shift-up is often faster that floating point _or_ "pure" integer math, and sidesteps a number of problems with both. It's at the cost of a learning curve with anyone using the package, but it works well. On all the processors I've tried it except for x86 processors, there's been a 3-20x speedup once I've hand-written the assembly code to do the computation (and that's without understanding or trying to accommodate any pipelines that may exist). But on the x86 -- which is the _only_ processor that I've tried it that had floating point -- 32-bit fractional arithmetic is slower than 64-bit floating point. So, yes -- whether integer (or fixed point) arithmetic is going to be faster than floating point depends _a lot_ on the processor. So instead of automatically deciding to do everything "the hard way" and feeling clever and virtuous thereby, you should _benchmark_ the performance of a code sample with floating point vs. whatever fixed-point poison you choose. Then, even if fixed point is significantly faster, you should look at the time consumed by floating point and ask if it's really necessary to save that time: even cheapo 8-bit processors run pretty fast these days, and can implement fairly complex control laws at 10 or even 100Hz using double-precision floating point arithmetic. If floating point will do, fixed point is a waste of effort. And if floating point is _faster_, fixed point is just plain stupid. So, benchmark, think, make an informed decision, and then that virtuous glow that surrounds you after you make your decision will be earned. -- My liberal friends think I'm a conservative kook. My conservative friends think I'm a liberal kook. Why am I not happy that they have found common ground? Tim Wescott, Communications, Control, Circuits & Software http://www.wescottdesign.com
Reply by ●March 28, 20122012-03-28
On Tue, 27 Mar 2012 18:52:09 +0300, upsidedown@downunder.com wrote:>On Tue, 27 Mar 2012 11:28:18 -0400, David T. Ashley ><dashley@gmail.com> wrote: > >> >>Without FPU support, assuming that the processor has basic integer >>multiplication instructions, integer operations are ALWAYS faster than >>floating-point operations. Usually _far_ faster. And always more >>precise. > >Floating point instructions MUL/DIV are trivial, just multiply/divide >the mantissa and add/sub the exponent.Assuming we are doing 64 bit double precession mul/div with an 8 bit processor, the mantissa is 48-56 bits and hence a single cycle 8x8=16 bit multiply instruction helps a lot. In addition, the lowest part of mantissa result (96-112 bits) is interesting only to see if this will generate a carry to the most significant 48-56 bits.>With FP add/sub you have to denormalize one operand and then normalize >the result, which can be quite time consuming, without sufficient HW >support.The denormalization of the smaller value can be done quite effectively if the hardware supports shift right by N bits in a single instruction. In fact it makes sense to first perform the right shift by multiple of 8 bits by byte copy and then do the 1..7 bit right shift by shift right instructions. Unfortunately , the normalization after FP add/sub gets ugly. While you can do the multiple of 8 shift with byte test and byte copying, you still have to do the final left shift with a loop 1-7 times with shift into carry and branch if carry set. Again, if the hardware supports something like FindFirstBitSet instruction in a single cycle, this will help the normalization a lot.>This can be really time consuming, if the HW is designed by an idiot.In the old days, I have seen a lot of designs, in which the designs is made based on available gates, not by the required functionality.
