data types

mccmikesandy3
09-05-2003, 07:39 AM
I wrote VB code to search thru raw data to pick out values at specific points. I solved the problem but I don't know what is happening to my values. The variable (Gap) is a single data type. If the gap excedes a limit the loop ends. If it doesn't excede the limit it is incremented. The increment is 0.010. The gaps are three significant digit numbers (ie. 0.215). What happens is the gap is incremented to 0.2249999999 and it won't reach the value until the next iteration of the loop. I did solve the problem by putting a 'correction' into the loop equation (ie. If Gap < MaxGap+0.001) but this doesn't answer what is going on. I tried double data type but I seem to get the same thing.

Thinker
09-05-2003, 07:59 AM
It is the conversion error between fractional numbers stored in binary
and represented as decimal. If you don't need a lot of precision, you
could try a scaled decimal type like currency.

EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum