Binary Representation of Decimal Values

More decimal issues, but unrelated to the SQL stuff. The problem being that numbers like 1/10 can be easily displayed in base 10 as 0.1. But, in base 10 we have problems with numbers like 1/3, .3333333333. Well, in binary, they have problems with some of our decimal numbers that you wouldn’t expect. e.g. 0.4 – 0.1 = 0.30000000000000000000004
while 1.4-1.1 = 0.29999999999999999999998.

The problem here is with rounding. I wrote a quick JS function to round these numbers to a significant number of digits.

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s