We Keep Coding

Truncate significant figures [duplicate]

Why do some numbers lose accuracy when stored as floating point numbers?
For example, the decimal number 9.2 can be expressed exactly as a ratio of two decimal integers (92/10), both of which can be expressed exactly in binary (0b1011100/0b1010). However, the same ratio stored as a floating point number is never exactly equal to 9.2:
32-bit "single precision" float: 9.19999980926513671875
64-bit "double precision" float: 9.199999999999999289457264239899814128875732421875
How can such an apparently simple number be "too big" to express in 64 bits of memory?

In most programming languages, floating point numbers are represented a lot like scientific notation: with an exponent and a mantissa (also called the significand). A very simple number, say 9.2, is actually this fraction:
5179139571476070 * 2 -49
Where the exponent is -49 and the mantissa is 5179139571476070. The reason it is impossible to represent some decimal numbers this way is that both the exponent and the mantissa must be integers. In other words, all floats must be an integer multiplied by an integer power of 2.
9.2 may be simply 92/10, but 10 cannot be expressed as 2n if n is limited to integer values.
Seeing the Data
First, a few functions to see the components that make a 32- and 64-bit float. Gloss over these if you only care about the output (example in Python):
def float_to_bin_parts(number, bits=64):
if bits == 32: # single precision
int_pack = 'I'
float_pack = 'f'
exponent_bits = 8
mantissa_bits = 23
exponent_bias = 127
elif bits == 64: # double precision. all python floats are this
int_pack = 'Q'
float_pack = 'd'
exponent_bits = 11
mantissa_bits = 52
exponent_bias = 1023
else:
raise ValueError, 'bits argument must be 32 or 64'
bin_iter = iter(bin(struct.unpack(int_pack, struct.pack(float_pack, number))[0])[2:].rjust(bits, '0'))
return [''.join(islice(bin_iter, x)) for x in (1, exponent_bits, mantissa_bits)]
There's a lot of complexity behind that function, and it'd be quite the tangent to explain, but if you're interested, the important resource for our purposes is the struct module.
Python's float is a 64-bit, double-precision number. In other languages such as C, C++, Java and C#, double-precision has a separate type double, which is often implemented as 64 bits.
When we call that function with our example, 9.2, here's what we get:
>>> float_to_bin_parts(9.2)
['0', '10000000010', '0010011001100110011001100110011001100110011001100110']
Interpreting the Data
You'll see I've split the return value into three components. These components are:
Sign
Exponent
Mantissa (also called Significand, or Fraction)
Sign
The sign is stored in the first component as a single bit. It's easy to explain: 0 means the float is a positive number; 1 means it's negative. Because 9.2 is positive, our sign value is 0.
Exponent
The exponent is stored in the middle component as 11 bits. In our case, 0b10000000010. In decimal, that represents the value 1026. A quirk of this component is that you must subtract a number equal to 2(# of bits) - 1 - 1 to get the true exponent; in our case, that means subtracting 0b1111111111 (decimal number 1023) to get the true exponent, 0b00000000011 (decimal number 3).
Mantissa
The mantissa is stored in the third component as 52 bits. However, there's a quirk to this component as well. To understand this quirk, consider a number in scientific notation, like this:
6.0221413x1023
The mantissa would be the 6.0221413. Recall that the mantissa in scientific notation always begins with a single non-zero digit. The same holds true for binary, except that binary only has two digits: 0 and 1. So the binary mantissa always starts with 1! When a float is stored, the 1 at the front of the binary mantissa is omitted to save space; we have to place it back at the front of our third element to get the true mantissa:
1.0010011001100110011001100110011001100110011001100110
This involves more than just a simple addition, because the bits stored in our third component actually represent the fractional part of the mantissa, to the right of the radix point.
When dealing with decimal numbers, we "move the decimal point" by multiplying or dividing by powers of 10. In binary, we can do the same thing by multiplying or dividing by powers of 2. Since our third element has 52 bits, we divide it by 252 to move it 52 places to the right:
0.0010011001100110011001100110011001100110011001100110
In decimal notation, that's the same as dividing 675539944105574 by 4503599627370496 to get 0.1499999999999999. (This is one example of a ratio that can be expressed exactly in binary, but only approximately in decimal; for more detail, see: 675539944105574 / 4503599627370496.)
Now that we've transformed the third component into a fractional number, adding 1 gives the true mantissa.
Recapping the Components
Sign (first component): 0 for positive, 1 for negative
Exponent (middle component): Subtract 2(# of bits) - 1 - 1 to get the true exponent
Mantissa (last component): Divide by 2(# of bits) and add 1 to get the true mantissa
Calculating the Number
Putting all three parts together, we're given this binary number:
1.0010011001100110011001100110011001100110011001100110 x 1011
Which we can then convert from binary to decimal:
1.1499999999999999 x 23 (inexact!)
And multiply to reveal the final representation of the number we started with (9.2) after being stored as a floating point value:
9.1999999999999993
Representing as a Fraction
9.2
Now that we've built the number, it's possible to reconstruct it into a simple fraction:
1.0010011001100110011001100110011001100110011001100110 x 1011
Shift mantissa to a whole number:
10010011001100110011001100110011001100110011001100110 x 1011-110100
Convert to decimal:
5179139571476070 x 23-52
Subtract the exponent:
5179139571476070 x 2-49
Turn negative exponent into division:
5179139571476070 / 249
Multiply exponent:
5179139571476070 / 562949953421312
Which equals:
9.1999999999999993
9.5
>>> float_to_bin_parts(9.5)
['0', '10000000010', '0011000000000000000000000000000000000000000000000000']
Already you can see the mantissa is only 4 digits followed by a whole lot of zeroes. But let's go through the paces.
Assemble the binary scientific notation:
1.0011 x 1011
Shift the decimal point:
10011 x 1011-100
Subtract the exponent:
10011 x 10-1
Binary to decimal:
19 x 2-1
Negative exponent to division:
19 / 21
Multiply exponent:
19 / 2
Equals:
9.5
Further reading
The Floating-Point Guide: What Every Programmer Should Know About Floating-Point Arithmetic, or, Why don’t my numbers add up? (floating-point-gui.de)
What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg 1991)
IEEE Double-precision floating-point format (Wikipedia)
Floating Point Arithmetic: Issues and Limitations (docs.python.org)
Floating Point Binary

This isn't a full answer (mhlester already covered a lot of good ground I won't duplicate), but I would like to stress how much the representation of a number depends on the base you are working in.
Consider the fraction 2/3
In good-ol' base 10, we typically write it out as something like
0.666...
0.666
0.667
When we look at those representations, we tend to associate each of them with the fraction 2/3, even though only the first representation is mathematically equal to the fraction. The second and third representations/approximations have an error on the order of 0.001, which is actually much worse than the error between 9.2 and 9.1999999999999993. In fact, the second representation isn't even rounded correctly! Nevertheless, we don't have a problem with 0.666 as an approximation of the number 2/3, so we shouldn't really have a problem with how 9.2 is approximated in most programs. (Yes, in some programs it matters.)
Number bases
So here's where number bases are crucial. If we were trying to represent 2/3 in base 3, then
(2/3)10 = 0.23
In other words, we have an exact, finite representation for the same number by switching bases! The take-away is that even though you can convert any number to any base, all rational numbers have exact finite representations in some bases but not in others.
To drive this point home, let's look at 1/2. It might surprise you that even though this perfectly simple number has an exact representation in base 10 and 2, it requires a repeating representation in base 3.
(1/2)10 = 0.510 = 0.12 = 0.1111...3
Why are floating point numbers inaccurate?
Because often-times, they are approximating rationals that cannot be represented finitely in base 2 (the digits repeat), and in general they are approximating real (possibly irrational) numbers which may not be representable in finitely many digits in any base.

While all of the other answers are good there is still one thing missing:
It is impossible to represent irrational numbers (e.g. π, sqrt(2), log(3), etc.) precisely!
And that actually is why they are called irrational. No amount of bit storage in the world would be enough to hold even one of them. Only symbolic arithmetic is able to preserve their precision.
Although if you would limit your math needs to rational numbers only the problem of precision becomes manageable. You would need to store a pair of (possibly very big) integers a and b to hold the number represented by the fraction a/b. All your arithmetic would have to be done on fractions just like in highschool math (e.g. a/b * c/d = ac/bd).
But of course you would still run into the same kind of trouble when pi, sqrt, log, sin, etc. are involved.
TL;DR
For hardware accelerated arithmetic only a limited amount of rational numbers can be represented. Every not-representable number is approximated. Some numbers (i.e. irrational) can never be represented no matter the system.

There are infinitely many real numbers (so many that you can't enumerate them), and there are infinitely many rational numbers (it is possible to enumerate them).
The floating-point representation is a finite one (like anything in a computer) so unavoidably many many many numbers are impossible to represent. In particular, 64 bits only allow you to distinguish among only 18,446,744,073,709,551,616 different values (which is nothing compared to infinity). With the standard convention, 9.2 is not one of them. Those that can are of the form m.2^e for some integers m and e.
You might come up with a different numeration system, 10 based for instance, where 9.2 would have an exact representation. But other numbers, say 1/3, would still be impossible to represent.
Also note that double-precision floating-points numbers are extremely accurate. They can represent any number in a very wide range with as much as 15 exact digits. For daily life computations, 4 or 5 digits are more than enough. You will never really need those 15, unless you want to count every millisecond of your lifetime.

Why can we not represent 9.2 in binary floating point?
Floating point numbers are (simplifying slightly) a positional numbering system with a restricted number of digits and a movable radix point.
A fraction can only be expressed exactly using a finite number of digits in a positional numbering system if the prime factors of the denominator (when the fraction is expressed in it's lowest terms) are factors of the base.
The prime factors of 10 are 5 and 2, so in base 10 we can represent any fraction of the form a/(2b5c).
On the other hand the only prime factor of 2 is 2, so in base 2 we can only represent fractions of the form a/(2b)
Why do computers use this representation?
Because it's a simple format to work with and it is sufficiently accurate for most purposes. Basically the same reason scientists use "scientific notation" and round their results to a reasonable number of digits at each step.
It would certainly be possible to define a fraction format, with (for example) a 32-bit numerator and a 32-bit denominator. It would be able to represent numbers that IEEE double precision floating point could not, but equally there would be many numbers that can be represented in double precision floating point that could not be represented in such a fixed-size fraction format.
However the big problem is that such a format is a pain to do calculations on. For two reasons.
If you want to have exactly one representation of each number then after each calculation you need to reduce the fraction to it's lowest terms. That means that for every operation you basically need to do a greatest common divisor calculation.
If after your calculation you end up with an unrepresentable result because the numerator or denominator you need to find the closest representable result. This is non-trivil.
Some Languages do offer fraction types, but usually they do it in combination with arbitary precision, this avoids needing to worry about approximating fractions but it creates it's own problem, when a number passes through a large number of calculation steps the size of the denominator and hence the storage needed for the fraction can explode.
Some languages also offer decimal floating point types, these are mainly used in scenarios where it is imporant that the results the computer gets match pre-existing rounding rules that were written with humans in mind (chiefly financial calculations). These are slightly more difficult to work with than binary floating point, but the biggest problem is that most computers don't offer hardware support for them.

converting c# float to IEEE single precision floating point bytes

I am implementing RFC4506 (XDR) in c#.
Does the binary format for floats in c# (BitConverter.GetBytes) use the IEEE standard?
How would I convert a float in c# into the XDR binary format for an IEEE single precision floating point number, I could do the legwork manually, but I would like to know if there is an existing method to do this?
The standard defines the floating-point data type "float" (32 bits or
4 bytes). The encoding used is the IEEE standard for normalized
single-precision floating-point numbers [IEEE]. The following three
fields describe the single-precision floating-point number:
S: The sign of the number. Values 0 and 1 represent positive and
negative, respectively. One bit.
E: The exponent of the number, base 2. 8 bits are devoted to this
field. The exponent is biased by 127.
F: The fractional part of the number's mantissa, base 2. 23 bits
are devoted to this field.
Therefore, the floating-point number is described by:
(-1)**S * 2**(E-Bias) * 1.F
the exact layout is as follows.
+-------+-------+-------+-------+
|byte 0 |byte 1 |byte 2 |byte 3 | SINGLE-PRECISION
S| E | F | FLOATING-POINT NUMBER
+-------+-------+-------+-------+
1|<- 8 ->|<-------23 bits------>|
<------------32 bits------------>
Just as the most and least significant bytes of a number are 0 and 3,
the most and least significant bits of a single-precision floating-
point number are 0 and 31. The beginning bit (and most significant
bit) offsets of S, E, and F are 0, 1, and 9, respectively. Note that
these numbers refer to the mathematical positions of the bits, and
NOT to their actual physical locations (which vary from medium to
medium).

.NET uses IEEE representation as well so BitConverter.GetBytes gets you the bytes you need.
It is however a half-baked standard, it only specifies the bit interpretation but does not demand a physical representation. The RFC is infected with Unix preferences from the previous century, like all networking standards are, byte order is big-endian.
On most any machine that can run .NET you need to reverse the bytes. Thus:
public static byte[] XdrFloat(float value) {
byte[] bytes = BitConverter.GetBytes(value);
if (BitConverter.IsLittleEndian) Array.Reverse(bytes);
return bytes;
}

Displaying IEEE-754 quadruple-precision (binary128) floating point values in scientific notation in C#

I'm trying to translate the raw binary data from a thread context into a human-readable format, and have come up empty when trying to translate quadruple-precision floating point values into a readable format in C#.
Ultimately, I'd like to display it in standard scientific notation, e.g. 1.234567×1089. I'm not worried about loss of precision in the process - I just want a reasonable idea of what the value is.
My first thought was to manually compute the value as a double by raising the exponent, but of course I'm going to exceed the maximum value in many cases. I don't mind losing precision, but not being able to display it at all isn't acceptable.
Is there some kind of simple mathematical hack I can use for this?

You could install a third-party library that handles that. For example it looks like QPFloat gives you a new struct called System.Quadruple which overrides ToString, so you could try that.
(I wonder when .NET will support something like System.Quadruple.)

So here's an answer to expand on the comment I made earlier. I hope you don't mind that I'm using Python, since I know where to find everything I need in that language; maybe someone else can translate this into a suitable answer in C#.
Suppose that you've got a sequence of 128 bits representing a number in IEEE 754 binary128 format, and that we've currently read those 128 bits in in the form of an unsigned integer x. For example:
>>> x = 0x4126f07c18386f74e697bd57a865a9d0
(I guess this would be a bit messier in C#, since as far as I can tell it doesn't have a 128-bit integer type; you'd need to either use two 64-bit integers for the high and low words, or use the BigInteger type.)
We can extract the exponent and significand via bit operations as usual (I'm assuming that you already got this far, but I wanted to include the computation for completeness):
>>> significand_mask = (1 << 112) - 1
>>> exponent_mask = (1 << 127) - (1 << 112)
>>> trailing_significand = x & significand_mask
>>> significand = 1.0 + float(trailing_significand) / (2.0**112)
>>> biased_exponent = (x & exponent_mask) >> 112
>>> exponent = biased_exponent - 16383
Note that while the exponent is exact, we've lost most of the precision of significand at this point, keeping only 52-53 bits of precision.
>>> significand
1.9393935334951098
>>> exponent
295
So the value represented is around 1.9393935334951098 * 2**295, or around 1.234567e+89. But you can't do the computation directly at this stage because it might overflow a Double (in this case it doesn't, but if the exponent were bigger you'd have a problem). So here's where the logs come in: let's compute the natural log of the value represented by x:
>>> from math import log, exp
>>> log_of_value = log(significand) + exponent*log(2)
>>> log_of_value
205.14079357778544
Then we can divide by log(10) to get the exponent and mantissa for the decimal part: the quotient of the division gives the decimal exponent, while the remainder gives the log of the significand, so we have to apply exp to it to retrieve the actual significand:
>>> exp10, mantissa10 = divmod(log_of_value, log(10))
>>> exp10
89.0
>>> significand10 = exp(mantissa10)
>>> significand10
1.234566999999967
And formatting the answer nicely:
>>> print("{:.10f}e{:+d}".format(significand10, int(exp10)))
1.2345670000e+89
That's the basic idea: to do this generally you'd also need to handle the sign bit and the special bit patterns for zeros, subnormal numbers, infinities and NaNs. Depending on the application, you may not need all of those.
There's some precision loss involved firstly in the conversion of the integer significand to a double precision float, but also in taking logs and exponents. The worst case for precision loss occurs when the exponent is large, since a large exponent magnifies the absolute error involved in the log(2) computation, which in turn contributes a larger relative error when taking exp to get the final significand. But since the (unbiased) exponent doesn't exceed 16384, it's not hard to bound the error. I haven't done the formal computations, but this should be good for around 12 digits of precision across the range of the binary128 format, and precision should be a bit better for numbers with small exponent.

there are few hacks for that...
compute hex string for number
mantissa and exponent are in binary so there should be no problem just do not forget to add zero for each 2^4 exponent part and shift the mantissa by exponent&3 bits. Negative exponents need few tweaks but are very similar.
All of this can be done by bit and shift operations so no precision loss if coded right ...
convert hex string to dec string
there are quite a few examples also here on SO here is mine. You can also tweak it a little to skip zero processing for more speed...
now scan the dec string
if you look at mine dec2hex and hex2dec conversions in link above then the scan is already there you need to find:
the position of first nonzero decimal from left and right
position of decimal point
from these you easily compute exponent
convert dec string to mantissa * 10^exponet form
it is quite straight forward just remove zeros ... and translate decimal point to its new position then add exponent part ...
add sign for mantissa
you can add it directly in bullets #1,#2 but if you do it in the end then it will spare you a few ifs ...
Hope this helps ...

Is there any practical difference between the .net decimal values 1m and 1.0000m?

Is there any practical difference between the .net decimal values 1m and 1.0000m?
The internal storage is different:
1m : 0x00000001 0x00000000 0x00000000 0x00000000
1.0000m : 0x000186a0 0x00000000 0x00000000 0x00050000
But, is there a situation where the knowledge of "significant digits" would be used by a method in the BCL?
I ask because I'm working on a means of compressing the space required for decimal values for disk storage or network transport and am toying with the idea of "normalizing" the value before I store it to improve it's compressability. But, I'd like to know if it is likely to cause issues down the line. I'm guessing that it should be fine, but only because I don't see any methods or properties that expose the precision of the value. Does anyone know otherwise?

The reason for the difference in encoding is because the Decimal data type stores the number as a whole number (96 bit integer), with a scale which is used to form the divisor to get the fractional number. The value is essentially
integer / 10^scale
Internally the Decimal type is represented as 4 Int32, see the documentation of Decimal.GetBits for more detail. In summary, GetBits returns an array of 4 Int32s, where each element represents the follow portion of the Decimal encoding
Element 0,1,2 - Represent the low, middle and high 32 bits on the 96 bit integer
Element 3 - Bits 0-15 Unused
Bits 16-23 exponent which is the power of 10 to divide the integer by
Bits 24-30 Unused
Bit 31 the sign where 0 is positive and 1 is negative
So in your example, very simply put when 1.0000m is encoded as a decimal the actual representation is 10000 / 10^4 while 1m is represented as 1 / 10^0 mathematically the same value just encoded differently.
If you use the native .NET operators for the decimal type and do not manipulate/compare the bit/bytes yourself you should be safe.
You will also notice that the string conversions will also take this binary representation into consideration and produce different strings so you need to be careful in that case if you ever rely on the string representation.

The decimal type tracks scale because it's important in arithmetic. If you do long multiplication, by hand, of two numbers — for instance, 3.14 * 5.00 — the result has 6 digits of precision and a scale of 4.
To do the multiplication, ignore the decimal points (for now) and treat the two numbers as integers.
3.14
* 5.00
------
0000 -- 0 * 314 (0 in the one's place)
00000 -- 0 * 314 (0 in the 10's place)
157000 -- 5 * 314 (5 in the 100's place)
------
157000
That gives you the unscaled results. Now, count the total number of digits to the right of the decimal point in the expression (that would be 4) and insert the decimal point 4 places to the left:
15.7000
That result, while equivalent in value to 15.7, is more precise than the value 15.7. The value 15.7000 has 6 digits of precision and a scale of 4; 15.7 has 3 digits of precision and a scale of 1.
If one is trying to do precision arithmetic, it is important to track the precision and scale of your values and results as it tells you something about the precision of your results (note that precision isnt' the same as accuracy: measure something with a ruler graduated in 1/10ths of an inch and the best you can say about the resulting measurement, no matter how many trailing zeros you put to the right of the decimal point is that it is accurate to, at best, a 1/10th of an inch. Another way of putting it would be to say that your measurement is accurate, at best, within +/- 5/100ths of the stated value.

The only reason I can think of is so invoking `ToString returns the exact textual representation in the source code.
Console.WriteLine(1m); // 1
Console.WriteLine(1.000m); // 1.000

If byte is 8 bit integer then how can we set it to 255?

The byte keyword denotes an integral
type that stores values as indicated
in the following table. It's an Unsigned 8-bit integer.
If it's only 8 bits then how can we assign it to equal 255?
byte myByte = 255;
I thought 8 bits was the same thing as just one character?

There are 256 different configuration of bits in a byte
0000 0000
0000 0001
0000 0010
...
1111 1111
So can assign a byte a value in the 0-255 range

Characters are described (in a basic sense) by a numeric representation that fits inside an 8 bit structure. If you look at the ASCII Codes for ascii characters, you'll see that they're related to numbers.
The integer count a bit sequence can represent is generated by the formula 2^n - 1 (as partially described above by #Marc Gravell). So an 8 bit structure can hold 256 values including 0 (also note TCPIP numbers are 4 separate sequences of 8 bit structures). If this was a signed integer, the first bit would be a flag for the sign and the remaining 7 would indicate the value, so while it would still hold 256 values, but the maximum and minimum would be determined by the 7 trailing bits (so 2^7 - 1 = 127).
When you get into Unicode characters and "high ascii" characters, the encoding requires more than an 8 bit structure. So in your example, if you were to assign a byte a value of 76, a lookup table could be consulted to derive the ascii character v.

11111111 (8 on bits) is 255: 128 + 64 + 32 + 16 + 8 + 4 + 2 + 1
Perhaps you're confusing this with 256, which is 2^8?

8 bits (unsigned) is 0 thru 255, or (2^8)-1.
It sounds like you are confusing integer vs text representations of data.

i thought 8 bits was the same thing as
just one character?
I think you're confusing the number 255 with the string "255."
Think about it this way: if computers stored numbers internally using characters, how would it store those characters? Using bits, right?
So in this hypothetical scenario, a computer would use bits to represent characters which it then in turn used to represent numbers. Aside from being horrendous from an efficiency standpoint, this is just redundant. Bits can represent numbers directly.

255 = 2^8 − 1 = FF[hex] = 11111111[bin]

range of values for unsigned 8 bits is 0 to 255. so this is perfectly valid
8 bits is not the same as one character in c#. In c# character is 16 bits. ANd even if character is 8 bits it has no relevance to the main question

I think you're confusing character encoding with the actual integral value stored in the variable.
A 8 bit value can have 255 configurations as answered by Arkain
Optionally, in ASCII, each of those configuration represent a different ASCII character
So, basically it depends how you interpret the value, as a integer value or as a character
ASCII Table
Wikipedia on ASCII

Sure, a bit late to answer, but for those who get this in a google search, here we go...
Like others have said, a character is definitely different to an integer. Whether it's 8-bits or not is irrelevant, but I can help by simply stating how each one works:
for an 8-bit integer, a value range between 0 and 255 is possible (or -127..127 if it's signed, and in this case, the first bit decides the polarity)
for an 8-bit character, it will most likely be an ASCII character, of which is usually referenced by an index specified with a hexadecimal value, e.g. FF or 0A. Because computers back in the day were only 8-bit, the result was a 16x16 table i.e. 256 possible characters in the ASCII character set.
Either way, if the byte is 8 bits long, then both an ASCII address or an 8-bit integer will fit in the variable's data. I would recommend using a different more dedicated data type though for simplicity. (e.g. char for ASCII or raw data, int for integers of any bit length, usually 32-bit)