Binary Representation of Decimal Values#
More decimal issues, but unrelated to the SQL stuff. The problem being that numbers like 1/10 can be easily displayed in base 10 as 0.1. But, in base 10 we have problems with numbers like 1/3, .3333333333. Well, in binary, they have problems with some of our decimal numbers that you wouldn't expect. e.g. 0.4 - 0.1 = 0.30000000000000000000004 while 1.4-1.1 = 0.29999999999999999999998.

The problem here is with rounding. I wrote a quick JS function to round these numbers to a significant number of digits.
Categories: Code Links
Thursday, September 29, 2005 7:50:28 AM (Central Daylight Time, UTC-05:00) #    Comments [0]  |  Tracked by:
"SQL Server Decimal/Money/Float datatypes" (www.ChrisMay.org) [Trackback]

 

Name
E-mail
(will show your gravatar icon)
Home page

Comment (HTML not allowed)  

Enter the code shown (prevents robots):
Live Comment Preview
All content © 2010, Christopher May, Inc
Open Job Positions
On this page
Google Ads
This site
Syndicate
Calendar
<September 2005>
SunMonTueWedThuFriSat
28293031123
45678910
11121314151617
18192021222324
2526272829301
2345678
Archives
 Full Archives By Category
March, 2010 (3)
February, 2010 (4)
January, 2010 (4)
December, 2009 (1)
November, 2009 (6)
October, 2009 (4)
September, 2009 (2)
August, 2009 (3)
July, 2009 (3)
May, 2009 (2)
April, 2009 (5)
March, 2009 (10)
February, 2009 (6)
January, 2009 (6)
December, 2008 (3)
November, 2008 (3)
October, 2008 (11)
September, 2008 (5)
August, 2008 (1)
July, 2008 (15)
June, 2008 (14)
May, 2008 (2)
April, 2008 (6)
March, 2008 (12)
February, 2008 (7)
January, 2008 (10)
December, 2007 (2)
November, 2007 (13)
October, 2007 (10)
September, 2007 (6)
August, 2007 (4)
July, 2007 (12)
June, 2007 (25)
May, 2007 (19)
April, 2007 (14)
March, 2007 (18)
February, 2007 (18)
January, 2007 (20)
December, 2006 (18)
November, 2006 (22)
October, 2006 (12)
September, 2006 (16)
August, 2006 (15)
July, 2006 (13)
June, 2006 (8)
May, 2006 (18)
April, 2006 (53)
March, 2006 (13)
December, 2005 (5)
October, 2005 (1)
September, 2005 (23)
August, 2005 (20)
July, 2005 (18)
June, 2005 (19)
May, 2005 (15)
April, 2005 (32)
March, 2005 (22)
February, 2005 (6)
January, 2005 (12)
December, 2004 (13)
November, 2004 (6)
September, 2004 (3)
August, 2004 (4)
July, 2004 (10)
June, 2004 (10)
May, 2004 (29)
April, 2004 (20)
March, 2004 (16)
February, 2004 (10)
January, 2004 (20)
December, 2003 (20)
November, 2003 (12)
October, 2003 (17)
September, 2003 (35)
August, 2003 (17)
July, 2003 (32)
June, 2003 (27)
May, 2003 (14)
Sitemap
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Blogroll OPML
[Feed] Chris May
Disclaimer

Powered by: newtelligence dasBlog 2.3.9074.18820

The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

Send mail to the author(s) E-mail

Theme design by Jelle Druyts


Pick a theme: