No announcements

Adding 2 floating point numbers in query produces wrong result

I have a query tha tads 2 FP (single 2) numbers (0.0 and 89.4) and the result is 89.4000. see query ...
| 14 Replies | 627 Views | Created by Steve Staab - Monday, April 25, 2016 11:45 PM | Last reply by Albert D. Kallal - Sunday, May 8, 2016 5:59 PM

Trailing digits when adding certain floating point numbers in javascript

I'm was making a simple input/output gadget when I ran into a strange problem in Javascript.  Essentially when adding two floating point numbers ...
| 0 Replies | 5329 Views | Created by acraige5 - Sunday, August 16, 2009 2:18 AM

tncating floating point numbers

takes up way too much screen-estate displaying all the uncessary detail. I would like to have maybe 2 or 3 numbers after the decimal point and i'm wondering which ...
| 2 Replies | 2829 Views | Created by amp1928 - Monday, May 31, 2010 11:54 PM | Last reply by amp1928 - Tuesday, June 1, 2010 12:37 AM

Wrong floating point assembler generated

What are you expecting anyway, floating point numbers aren't accurate. There are a few reasons why you could have this type of change in the results and all of them ...
| 2 Replies | 2828 Views | Created by rumpelstilzchen2010 - Thursday, June 9, 2011 4:51 PM | Last reply by Darran Rowe - Thursday, June 9, 2011 11:49 PM

Floating-point numbers and equality

Floating-point numbers should never be directly compared for equality, because the IEEE floating-point representation does not support an exact equality ...
| 1 Replies | 3108 Views | Created by Rainmater - Friday, November 2, 2012 8:41 AM | Last reply by Sehnsucht_Fr - Friday, November 2, 2012 11:12 AM

Floating Point, Additions and Random Numbers?

In the addition of real numbers like 0.1234 + 0.1234 + 0.1234 + ........ appears random number to the right of the point like this result 12.34000000000005
| 1 Replies | 3621 Views | Created by Determinant - Wednesday, June 11, 2008 11:39 PM | Last reply by WayneAKing - Thursday, June 12, 2008 1:17 AM

Int/Floating-point numbers conversion

integer and floating-point calculation.  I found it very hard because there are common years as well as leap years. And each month has different ...
| 7 Replies | 1747 Views | Created by VBui - Wednesday, January 4, 2012 8:21 AM | Last reply by Barry-Schwarz - Thursday, January 5, 2012 5:00 AM

loop for floating point numbers

0.3 in all situations.  If you want exact calculations on a "floating-point" type field, then consider using a "currency" type ...
| 3 Replies | 275 Views | Created by IntenseNJ - Monday, May 16, 2016 3:42 PM | Last reply by Viorel_ - Monday, May 16, 2016 5:35 PM

Matrix transform produces wrong result (SL4)

* mt.Matrix.M22 + mt.Matrix.OffsetY; I get this result: X = 425.011518239975 Y = 48.025378704071 I ...
| 3 Replies | 1017 Views | Created by asthomas - Sunday, September 11, 2011 1:40 PM | Last reply by asthomas - Thursday, September 15, 2011 9:28 AM

skeleton in x64 Asm floating point numbers in Visual C++

Dear Community Is this skeleton of program.cpp is completely in order to addressing 64 bit floating point number in Asm x64 ...
| 1 Replies | 619 Views | Created by RB _1 - Wednesday, November 19, 2014 7:04 PM | Last reply by davewilk - Wednesday, November 19, 2014 8:02 PM

Floating point errors with irrational numbers

Yes. Please study What Every Computer Scientist Should Know About Floating Point Arithmetic And I do mean study. Don't just ...
| 1 Replies | 346 Views | Created by GadMolkho - Thursday, January 22, 2015 8:41 PM | Last reply by Brian Muth - Thursday, January 22, 2015 9:41 PM

Floating Point Precision issue

I am working on a multithreaded application in VC++ that does a lot of floating point computation. I was wondering  whether it is possible that the order ...
| 2 Replies | 3883 Views | Created by rKarthik - Monday, August 14, 2006 12:57 AM | Last reply by rKarthik - Monday, August 14, 2006 2:22 AM

Floating point

term floating point refers to the fact that the radix point (decimal point, or, more commonly in computers, binary point) can "float": ...
| 7 Replies | 3404 Views | Created by Dave Pf - Friday, July 18, 2008 4:13 PM | Last reply by SMD- - Wednesday, July 23, 2008 12:41 PM

Library with Floating Point numbers with multiprecision arithmetic

"floating point numbers and not for integer" Use the J# BigDecimal class.
| 4 Replies | 3101 Views | Created by Oscarfh - Wednesday, January 14, 2009 12:04 PM | Last reply by Oscarfh - Wednesday, January 14, 2009 2:51 PM

tabletable.compute("Var()", filter ) produces wrong result

Just a gues, what result value type are you using, that we don't see? That it does things half does not mean that the answer from Martin is incorrect, there ...
| 7 Replies | 4297 Views | Created by topcan5 - Thursday, May 14, 2009 11:54 PM | Last reply by topcan5 - Wednesday, May 27, 2009 2:48 PM

floating point control register

I am having a problem with floating point exceptions after creating an instance of a COM object within in a Windows form application. The result of the ...
| 0 Replies | 3129 Views | Created by Robert.Jack - Tuesday, July 18, 2006 4:24 PM

Floating point precision

How do I find out the floating point precision of my machine?
| 1 Replies | 2820 Views | Created by mpt_fz - Wednesday, July 29, 2009 5:25 PM | Last reply by Reed Copsey, Jr - Wednesday, July 29, 2009 5:34 PM

Wrong query result

=20         The red filters satisfy the result ...
| 1 Replies | 6114 Views | Created by ardmore - Wednesday, April 16, 2008 3:35 PM | Last reply by Kalman Toth - Wednesday, April 16, 2008 3:47 PM