* significant digits
significant digits
I have a field I want to set significant digits on it 2 decimals and I want it to be a number
so
510.1
should be
510.10
and of course I can make that number, it is trivial, but LUA immediately changes it back to 510.1
am I missing a trick of some kind to set significant digits?
so
510.1
should be
510.10
and of course I can make that number, it is trivial, but LUA immediately changes it back to 510.1
am I missing a trick of some kind to set significant digits?
FH V.6.2.7 Win 10 64 bit
- Jane
- Site Admin
- Posts: 8514
- Joined: 01 Nov 2002 15:00
- Family Historian: V7
- Location: Somerset, England
- Contact:
Re: significant digits
Lua uses floating point numbers and as such does not use fixed decimals. If you want to display the number with fixed decimals you will need to output as a formatted string.
See http://lua-users.org/wiki/FormattingNumbers
See http://lua-users.org/wiki/FormattingNumbers
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."
- tatewise
- Megastar
- Posts: 28409
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: significant digits
In most applications trailing decimal fraction zeros are always removed, e.g. Excel spreadsheet.
Although you are storing digits that represent a number, any field in FH/GEDCOM will actually be a string.
So I suggest you use the Lua string.format library function that uses C printf formatting codes:
e.g.
local number = 510.1
local string = string.format( "%.2f", number )
That will save as "510.10"
To convert that string to a number use:
local number = tonumber( string )
Although you are storing digits that represent a number, any field in FH/GEDCOM will actually be a string.
So I suggest you use the Lua string.format library function that uses C printf formatting codes:
e.g.
local number = 510.1
local string = string.format( "%.2f", number )
That will save as "510.10"
To convert that string to a number use:
local number = tonumber( string )
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Re: significant digits
it is what i am doing now. mostly its just a 'feature' of fhoutputresultsetcolumn... not the first time I have issues with the numbers as text sort.
FH V.6.2.7 Win 10 64 bit
- tatewise
- Megastar
- Posts: 28409
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: significant digits
fhOutputResultSetColumn only offers text or integer options for ColumnType and NOT number.
So any numbers that are not integer must be formatted appropriately into text strings.
If you want them to sort correctly, then they must also be fixed length text strings with leading spaces.
Otherwise, the leading digit will determine the alphanumeric sort order regardless of whether it is unit, ten, hundred, etc.
So use:
local string = string.format( "%9.2f", number )
to force string to always be 9 characters long including decimal point and 2 fraction digits, i.e. 6 chars before decimal point.
It will pad with leading space chars where necessary, so that sorting will work correctly.
So any numbers that are not integer must be formatted appropriately into text strings.
If you want them to sort correctly, then they must also be fixed length text strings with leading spaces.
Otherwise, the leading digit will determine the alphanumeric sort order regardless of whether it is unit, ten, hundred, etc.
So use:
local string = string.format( "%9.2f", number )
to force string to always be 9 characters long including decimal point and 2 fraction digits, i.e. 6 chars before decimal point.
It will pad with leading space chars where necessary, so that sorting will work correctly.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Re: significant digits
function fmtGPS(gps)
gps = (string.format('%4.7f', gps))
return gps
end
function fmtGPS(gps)
gps = (string.format('%12.7f', gps))
return gps
end
gps argument
-111.872509
gps returned
-111.8725087
either way
I was expecting
-111.8725090
how so?
gps = (string.format('%4.7f', gps))
return gps
end
function fmtGPS(gps)
gps = (string.format('%12.7f', gps))
return gps
end
gps argument
-111.872509
gps returned
-111.8725087
either way
I was expecting
-111.8725090
how so?
FH V.6.2.7 Win 10 64 bit
- tatewise
- Megastar
- Posts: 28409
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: significant digits
The obvious answer is that gps was actually -111.8725087
How do you know gps was -111.872509 ?
Where is that being displayed?
That may be a 6 decimal point rounded value, which would be correct for an actual value of -111.8725087
Try debugging:
function fmtGPS(gps)
gps = (string.format('%12.7f', gps))
return gps
end
local ans = fmtGPS(-111.8725087)
With breakpoint on first line of fmtGPS function in the debugger shows gps is -111.872509 but we know it is actually -111.8725087, i.e. it is rounded in the debug display.
How do you know gps was -111.872509 ?
Where is that being displayed?
That may be a 6 decimal point rounded value, which would be correct for an actual value of -111.8725087
Try debugging:
function fmtGPS(gps)
gps = (string.format('%12.7f', gps))
return gps
end
local ans = fmtGPS(-111.8725087)
With breakpoint on first line of fmtGPS function in the debugger shows gps is -111.872509 but we know it is actually -111.8725087, i.e. it is rounded in the debug display.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Re: significant digits
function fmtGPS(gps)
(a) gps = (string.format('%12.7f', gps))
(r) return gps
end
(a) = -111.872509
(r) = -111.8725087
(a) is before format
(b) is before return
you are saying that (a) is not the actual value, just is a display value for debug?
and that (b) is correct?
(a) gps = (string.format('%12.7f', gps))
(r) return gps
end
(a) = -111.872509
(r) = -111.8725087
(a) is before format
(b) is before return
you are saying that (a) is not the actual value, just is a display value for debug?
and that (b) is correct?
FH V.6.2.7 Win 10 64 bit
- tatewise
- Megastar
- Posts: 28409
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: significant digits
Yes, like I said, try an experiment with:
function fmtGPS(gps)
x100 = gps * 100
(a) gps = (string.format('%12.7f', gps))
(r) return gps
end
local ans = fmtGPS(-111.87250872915)
We know gps at (a) is actually -111.87250872915
But it is displayed at (a) in debug as -111.872509 i.e. it is rounded to 6 digits the same as %12.6f
Note that x100 is displayed as -11187.250873 which proves my point.
function fmtGPS(gps)
x100 = gps * 100
(a) gps = (string.format('%12.7f', gps))
(r) return gps
end
local ans = fmtGPS(-111.87250872915)
We know gps at (a) is actually -111.87250872915
But it is displayed at (a) in debug as -111.872509 i.e. it is rounded to 6 digits the same as %12.6f
Note that x100 is displayed as -11187.250873 which proves my point.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry