Page 1 of 1
table lookups (when we was fab) or memoization (gangnam style)
Posted: 15 Jun 2019 13:19
by Ron Melby
you have all been hammered to death by my matRLT (materialize relation)
Code: Select all
function matRLT(iptr, ix)
local ix = ix or 1
tblRLT = tblRLT or {}
cwrlt = cwrlt or 0
--[[ relationship ]]
local rlt = ''
--[[ pre = pre replacement ]]
local pre = ''
--[[ pst = post replacement ]]
local pst = ''
--[[ rpl = replacement area ]]
local rpl = ''
--[[ tho = throw away parts ]]
local tho = ''
rlt = fhCallBuiltInFunction('Relationship', ptrRoot, iptr, 'TEXT', ix)
if rlt == '' then
return rlt
end
-- correct rltdata
rlt = rlt:gsub( '%-removed', ' removed' ) -- fix error (removed and -removed 1st cousins
rlt = rlt:gsub( 'once', '1 times' ) -- text to numeric
rlt = rlt:gsub( 'twice','2 times' ) -- text to numeric
rlt = rlt:gsub( '^(%d[^%d])(.*)', ' %1%2' ) -- align leading numerics
rlt = rlt:gsub( '%(x(%d)%)', '(x0%1)' ) -- (xd) padded to (x0d)
rlt = rlt:gsub( 'great great%-', 'great (x02) ' ) -- reduce great great-
pre, rpl, pst = string.match(rlt, '(.*[^%d])(%d+ times)(.*)') -- d or dd times to (xdd)
if rpl ~= nil then
rpl = tonumber(string.match(rpl, '(%d+)'))
if rpl < 10 then
rpl = ('0'..rpl)
end
rlt = (pre .. '(x'.. rpl .. ')' .. pst)
end
pre, rpl, tho, pst = string.match(rlt, '(.*) %(x(%d+)%) (great%-)(.*)') -- remove great- add 1 to x(dd)
if tho ~= nil then
rpl = (tonumber(rpl) + 1)
if rpl < 10 then
rpl = ('0'.. rpl)
end
rpl = '(x' .. rpl .. ')'
rlt = pre .. ' ' .. rpl .. ' ' .. pst
end
rlt = rlt:gsub( 'great%-', 'great ' ) -- change great- to great
rlt = rlt:gsub( 'great', 'Great' ) -- change great- to great
rlt = rlt:gsub( 'grand', 'Grand' ) -- change great- to great
rlt = rlt:gsub( 'uncle', 'Uncle' ) -- change great- to great
rlt = rlt:gsub( 'aunt', 'Aunt' ) -- change great- to great
return rlt
end --fn matRLT
in and of itself, I know it works well. However it is repetitive, nothing new under the sun, most of the time. its dirty, leaving temporary string snips all over the place by the score, and doing many relatively expensive operations for each iteration. This, in my geezer days of computer geeking cried out for table lookups:
tblRLT:
[4th cousin once removed], [ 4th cousin (x01) removed]
...
...
I suppose I could add a counter of how often I see the relation, sort it descending, at end of program so
for k, v in pairs(tblRLT) does an even faster lookup
so, the closest thing I can find that does that, is Map Life Facts.
I got tablex is needed for loading and saving.
local Folder = fhGetContextInfo('CI_APP_DATA_FOLDER') -- global to all projects
local Filename = Folder .. '\\Plugin Data\\tblRLT.dat'
local tblRLT = { 'grandfather', 'Grandfather' } and so on.
table.save(tblRLT, Filename) -- save the table to file
tblRLT = table.load(Filename) -- load the table from file
(thats all the simple part....)
Now the one program I know does this and does it in a way I want it to is that Map Life Facts.
I cannot find the logic (because the program is too advanced for me) if the table does not exist, make it, and load it. if it exists, use it, and update as necessary. sort of condensed so I can make small feeding functions to matRLT.
where is that simple logic condensed in some snippet or program?
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 16 Jun 2019 10:31
by tatewise
I guess the idea is to take the table building offline, like you have done for other speed improvements.
So you can use the tablex modules for loading and saving as you say.
The function matRLT needs to be modified to check and update the table tblRLT.
local rel = fhCallBuiltInFunction('Relationship', ptrRoot, iptr, 'TEXT', ix)
rlt = tblRLT[rel] -- lookup relationship translation via table
if rlt then return rlt end
rlt = rel
-- process rlt as currently --
tblRLT[rel] = rlt -- save translation against original relationship
return rlt
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 16 Jun 2019 17:50
by Ron Melby
tblRLT = ({fhrlt,swrlt, 1})
Yes, so for now, my problem is one of statement compostion
tblRLT needs three things
fhrlt [index, which is the relation fh returns]
swrlt [value, which is the relation my software transforms]
rltocr [a number which I will add one to, every time I am returned a fhrlt for sorting later)
of course now I am still shaky on working with tables
do I?
table.insert(fhrlt, swrlt, rltocr or 1)
or?
table.insert(tblRLT[fhrlt], swrlt, rltocr or 1)
the only way to get to that statement is I want to create a table entry.tl
beforehand when I look at the table to see if its already in there,
if it is, I add 1 to rltocr
how do I update the table?
table. ??? tblRLT.rltocur (although I havent named it..rltocur = rltocur + 1
I think the answer is slightly different than what I have here.
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 16 Jun 2019 19:10
by tatewise
You are making it far too complicated.
Yes, only three data items are involved.
tblRLT is the translation table.
fhrlt is the original FH relationship provided by fhCallBuiltInFunction('Relationship', ptrRoot, iptr, 'TEXT', ix)
swrlt is the translated relationship you require.
So fhrlt is the table index and swrlt is the table value.
To save that translation use tblRLT[fhrlt] = swrlt
To lookup a translation use swrlt = tblRLT[fhrlt]
That is all there is to it. That is how basic Lua translation tables work, where the index is a text string (not an integer).
You do NOT use table.insert as that only employs integer keys that are useless for this purpose.
You do NOT need multiple table fields. No sorting or counting is necessary.
If fhrlt is "4th cousin once removed" and swrlt is "4th cousin (x01) removed" then
tblRLT["4th cousin once removed"] contains "4th cousin (x01) removed"
So when fhCallBuiltInFunction('Relationship', ptrRoot, iptr, 'TEXT', ix) returns "4th cousin once removed"
tblRLT["4th cousin once removed"] will return "4th cousin (x01) removed"
Let me give a really simple example using tblCurrency
tblCurrency = { USA="dollar"; UK="pound"; Japan="yen"; Russia="ruble"; }
So tblCurrency["USA"] returns "dollar"
and tblCurrency["UK"] returns "pound"
To add to the table you would use:
tblCurrency["Europe"] = "euro"
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 16 Jun 2019 20:01
by Ron Melby
behind the scene it must be something that matches the text, and its in some arrival order.
if one put all the most common up front with a rltocur descending sort before it is put away, it would be faster no?
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 16 Jun 2019 20:58
by tatewise
No, it is in no particular order, and you cannot influence the order.
The order that the index entries are created is not preserved.
So
for k,v in pairs (tblRTL) do will not traverse in the same order as the index entries were created.
Just use the table lookup as it is. It is a standard associative array.
It will be faster than your function software, and faster than any sorting & counting technique.
Lua uses a hash algorithm to lookup the index efficiently.
I have read somewhere that text index lookup is just as efficient as integer index lookup.
See
https://www.lua.org/gems/sample.pdf that has lots of efficiency advice.
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 17 Jun 2019 00:42
by Ron Melby
-- initiate variables for save procedure
local tables,lookup = { tbl },{ [tbl] = 1 }
file:write( "return {"..charE )
for idx, t in ipairs( tables ) do
if filename and filename ~= true and filename ~= 1 then
file:write( "-- Table: {"..idx.."}"..charE )
end
file:write( "{"..charE )
local thandled = {}
for i, v in ipairs( t ) do <<<<<<<<<<<<<<<<bad argument #1 to ipairs, table expected got string.i
this is code from Map Life Facts
@Module: +fh+tablex_v1
@Author: Mike Tate
@Version: 1.2
@LastUpdated: 22 May 2018
@Description: A Table Load Save Module.
@V1.2: Added local definitions of _ to ensure nil gets returned on error.
@V1.1: ?
@V1.0: Initial version 0.94 is Lua 5.1 compatible.
I only have this copied out into my program, no other modules.
when called by:
save(table, Filename)
table = 'tblRLT' has entries they look good (local table .260)
Filename = ' C:\ProgramData\Calico Pie\Family Historian\tblRLT.dat'
path built by:
function rtvProjectTBL(table)
local Folder = fhGetContextInfo('CI_APP_DATA_FOLDER') -- to share between projects
local Filename = Folder .. '\\' .. table .. '.dat'
return Filename
end
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 17 Jun 2019 09:41
by tatewise
Ron, you are way beyond debugging such trivial errors.
What is the parameter
table that you are saving?
It should be the
table tblRLT but you are supplying the
string 'tblRLT'
I strongly advise that you do NOT save in
C:\ProgramData\Calico Pie\Family Historian\
Instead use
C:\ProgramData\Calico Pie\Family Historian\Plugin Data\
i.e.
Code: Select all
function rtvProjectTBL(table)
local Folder = fhGetContextInfo('CI_APP_DATA_FOLDER') -- share between projects
local Filename = Folder .. '\\Plugin Data\\' .. table .. '.dat'
return Filename
end
local Filename = rtvProjectTBL( 'tblRLT' )
save( tblRLT, Filename )
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 17 Jun 2019 16:58
by Ron Melby
its up and running.
I am now in search if a way to code the impossible in LUA,
function rtvObjNameText(tblRLT)
the magic = ShaPow!!!
return 'tblRLT'
end
I have tried some tricks and there is nothing doing...it would be very useful, very useful to be able to do that programatically.
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 17 Jun 2019 19:24
by tatewise
I don't think there is any such magic.
The easiest solution is to simply define a string variable at the same point you define the original variable:
tblRLT = { }
strRLT = 'tblRLT'
Assuming you only want it to work for tables, and never need to use index 0, then another trick is:
tblRLT = { }
tblRLT[0] = 'tblRLT'
Then wherever you want the table name use tblRLT[0]
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 17 Jun 2019 20:09
by Ron Melby
the tbl[0] might work with a lot of monkeying around, but there is a cart before the horse problem.
this is part of why I messed up the tblRLT and 'tblRLT'
nevertheless the thought would be...
create a table of tables I have opened in a program from load
either they are new or existing tables.
register the tblRLT, 'tblRLT' entries, and maybe the #t (but that could be iffy, because I could change a table and still have it have the same #t
then, when its time to save, go thru the table, and save the tables to disk.
one problem is when table is loaded if it already exists, one can use t[0] but not if its new
function rtvProjectTBL(table)
local Folder = fhGetContextInfo('CI_APP_DATA_FOLDER') -- share between projects
local Filename = Folder .. '\\Plugin Data\\' .. table .. '.dat' <<<<< here table needs to be a string
return Filename
end
local Filename = rtvProjectTBL( 'tblRLT' ) <<<<< here table needs to be a string
save( tblRLT, Filename ) <<<<<<<<<< here it needs to be the object.
so if
stbl = vartostring(tblRLT) instead of it returning the pointedobject or some other clever LOL........
stbl = fhGetVarNameAsText(tblRLT)
but of course in generalizing, its name is destroyed completely....
rtvVarNam(table)
debug.getinfo
debug.getlocal
run thru _G
all the other ideas......
nope.........
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 17 Jun 2019 21:57
by tatewise
If you are prepared to work with a table of tables, then there is a possible solution.
In different Plugins the
tblAll table of tables can use different tables as long as the names match the files.
Code: Select all
tblAll = { tblRTL = { }; tblABC = { }; tblXYZ = { }; }
function rtvProjectTBL(name)
local Folder = fhGetContextInfo('CI_APP_DATA_FOLDER') -- share between projects
local Filename = Folder .. '\\Plugin Data\\' .. name .. '.dat'
return Filename
end
function loadTables(tblLoad) -- Load the shared table files
for name, tbl in pairs (tblLoad) do
tbl, err = load( rtvProjectTBL(name) )
tblLoad[name] = tbl or { }
end
end
function saveTables(tblSave) -- Save the shared table files
for name, tbl in pairs (tblSave) do
save( tbl, rtvProjectTBL(name) )
end
end
loadTables(tblAll)
tblAll.tblABC = { ABC=2 }
tblAll.tblXYZ = { XYZ=3 }
tblAll.tblRTL = { RTL=1 } -- Use tblAll.tblRTL where used tblRTL before, e.g.
tblAll.tblRLT[fhrlt] = swrlt -- Create memoization
swrlt = tblAll.tblRLT[fhrlt] -- Lookup memoization
saveTables(tblAll)
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 18 Jun 2019 01:29
by Ron Melby
9got the best possibilities,
can I have a tableall. TBLRLT{fh, sw), meta{tbltxt='tblRLT'; path=Filename; A=''; B=''; ...N='?'...} )
and so on?
or do I have to have associative tables(arrays), and how do I associate them?
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 18 Jun 2019 09:26
by tatewise
Probably, yes.
But you need to provide some valid table definition syntax and examples of usage.
Your
tableall example has a crazy mix of
{ } ( ) unmatched brackets, so I don't understand.
You know what you want to do, but without examples I don't understand what you want.
e.g.
How is
memoization supported?
Which parts of
tableall are saved and loaded?
Why do you need both
tbltxt and
path?
What does "
and so on" mean?
Remember, for
memoization to work efficiently you MUST use the associative table concept:
tblMemo[fhrlt] = swrlt -- Create memoization entry
swrlt = tblMemo[fhrlt] -- Lookup memoization entry
where
tblMemo is any table such as
tblRLT or
tblAll.tblRLT or
tableall.TBLRLT
This might be what you want, but I am guessing:
Code: Select all
local tblAll = {
RLT = {
Memo={};
Meta={
Name='tblRLT';
Path='C:\\ProgramData\\Calico Pie\\Family Historian\\Plugin Data\\tblRLT.dat';
A='1'; B='a'; -- etc
};
};
ABC = {
Memo={};
Meta={
Name='tblABC';
Path='C:\\ProgramData\\Calico Pie\\Family Historian\\Plugin Data\\tblABC.dat';
A='2'; B='b'; -- etc
};
};
XYZ = {
Memo={};
Meta={
Name='tblXYZ';
Path='C:\\ProgramData\\Calico Pie\\Family Historian\\Plugin Data\\tblXYZ.dat';
A='3'; B='c'; -- etc
};
};
}
function loadTables(tblLoad) -- Load the shared table files
for Key, Value in pairs (tblLoad) do
local Table, Error = load( Value.Meta.Path )
if Table then
tblLoad[Key].Memo = Table.Memo or {}
tblLoad[Key].Meta = Table.Meta or Value.Meta
else
-- Report Error ?
end
end
end
function saveTables(tblSave) -- Save the shared table files
for Key, Value in pairs (tblSave) do
save( Value, Value.Meta.Path )
end
end
loadTables(tblAll) -- Load all tables
tblRLT = tblAll.RLT.Memo
tblABC = tblAll.ABC.Memo -- Copy memoization
tblXYZ = tblAll.XYZ.Memo
-- Perform all the clever stuff ...
fhrlt = "4th cousin once removed"
swrlt = "4th cousin (x01) removed"
tblRLT[fhrlt] = swrlt -- Create memoization
swrlt = tblRLT[fhrlt] -- Lookup memoization
tblABC = { ABC=2 }
tblXYZ = { XYZ=3 }
tblAll.RLT.Memo = tblRLT
tblAll.ABC.Memo = tblABC -- Copy memoization
tblAll.XYZ.Memo = tblXYZ
saveTables(tblAll) -- Save all tables
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 18 Jun 2019 13:20
by Ron Melby
because once its a project table its most likely going to be always a project table, save the path, doesnt need to rebuild it time after time.
so i might have had unmatched whatevers.
so tb lRLT[fhrlt] [sfwrlt] then an array meta lets say of some values a.....z that can be accessed.
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 18 Jun 2019 20:28
by tatewise
The
path is only built once per Plugin usage, and is very quick compared to the
save and
load functions.
So it seems unnecessary to have that meta data, which has to be loaded and saved time after time.
Also the
load function must be more complex to cope with the meta data, particularly the first time when no file exists.
e.g.
Code: Select all
function loadTables(tblLoad) -- Load the shared table files
for Key, Value in pairs (tblLoad) do
local Table, Error = load( Value.Meta.Path )
if Table then
tblLoad[Key].Memo = Table.Memo or {}
tblLoad[Key].Meta = Table.Meta or Value.Meta
end
end
end
whereas without meta data it is:
Code: Select all
function loadTables(tblLoad) -- Load the shared table files
for Key, Value in pairs (tblLoad) do
local Table, Error = load( rtvProjectTBL(Key) )
tblLoad[Key] = Table or { }
end
end
function rtvProjectTBL(name)
local Folder = fhGetContextInfo('CI_APP_DATA_FOLDER') -- share between projects
local Filename = Folder .. '\\Plugin Data\\' .. name .. '.dat'
return Filename
end
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 19 Jun 2019 01:29
by Ron Melby
Yes, you know, you are right,even at say 100 tables, you are right.
I had thought, I will have several tables that are used in a program. for the purpose of this discussion we shall call them A, B,.. , a..z
so I could in each program:
LKUP = (a, A, b, d, z)
in another it might be:
LKUP = (c,f,K)
function loadtblset(LKUP)
and there might be some reason to wipe out a table, and rebuild it (just spitballing) and the original build would be there.
and run thru those simplifiying that end of it, but it makes all the other functions more complex, the idea will go on the scrapheap of codefail.
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 19 Jun 2019 08:41
by tatewise
Another important point I forgot to mention is that fhGetContextInfo('CI_APP_DATA_FOLDER') is portable.
If you ever offered your Plugins to other users, the path on their PC may be different to the path on your PC.
So if you hard code the path in your Plugins then they may not work on another PC.
Whereas fhGetContextInfo('CI_APP_DATA_FOLDER') always provides the current PC path.
e.g.
Modern Windows use C:\ProgramData\Calico Pie\Family Historian\
Old Windows XP used C:\Documents and Settings\All Users\Application Data\Calico Pie\Family Historian\
Emulators on Mac use C:\Users\Public\Application Data\Calico Pie\Family Historian\
You can still use that LKUP concept with different tables in different Plugins as I explained earlier.
( You might even find a use for the meta data idea, but don't code the file path into the meta data. )
This is the basic solution without meta data, such as is needed for memoization:
LKUP = { a = { }; A = { }; b = { }; d = { }; z = { }; } in one Plugin
LKUP = { c = { }; f = { }; k = { }; } in another Plugin
-- Load tables c, f & k (similarly for a, A, b, d & z)
loadTables(LKUP)
tblc = LKUP.c
tblf = LKUP.f
tblk = LKUP.k
-- Do all the tricky stuff --
-- Save tables c, f & k (similarly for a, A, b, d & z)
LKUP.c = tblc
LKUP.f = tblf
LKUP.k = tblk
saveTables(LKUP)
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 22 Jun 2019 13:31
by Ron Melby
still having troubles with syntax:
function rtvTBL(stbl, styp)
styp = styp or '*project'
if styp == '*project' then
Filename = rtvProjectTBL(stbl)
end
if styp == '*global' then
Filename = rtvGlobalTBL(stbl)
end
otbl = (load(Filename)) or {} -- load the table from file *** issue here
return otbl
end
I got here by:
tblRLT = rtvTBL('tblRLT', '*project')
the Filename is correct and it has 260 entries but I cant make a general load of a table
somehow tablex does it, but I cant follow it.
I know its simple and I should know it, but I do not.
otbl NEVER shows up on debug so I know my stuff is crazy, but no idea how to say what I am trying to say in LUA
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 22 Jun 2019 14:22
by tatewise
I have just copied your script, added the functions
rtvProjectTBL &
rtvGlobalTBL plus
load &
save library.
It runs just fine, so what is the *** issue?
It will help debugging if you make
Filename and
otbl both
local, and also
tblRLT temporarily.
How many times do we have to repeat that advice
Global tables such as
otbl are not shown in debug pane unless you tick
Debug > Options > Display Global Types > Table
e.g.
Code: Select all
function rtvGlobalTBL(name)
local Folder = fhGetContextInfo('CI_APP_DATA_FOLDER')
local Filename = Folder .. '\\Plugin Data\\' .. name .. '.dat'
return Filename
end
function rtvProjectTBL(name)
local Folder = fhGetContextInfo('CI_PROJECT_DATA_FOLDER')
local Filename = Folder .. '\\Plugin Data\\' .. name .. '.dat'
return Filename
end
function rtvTBL(stbl, styp)
local Filename
styp = styp or '*project'
if styp == '*project' then
Filename = rtvProjectTBL(stbl)
end
if styp == '*global' then
Filename = rtvGlobalTBL(stbl)
end
local otbl = (load(Filename)) or {} -- load the table from file
return otbl
end
local tblRLT = rtvTBL('tblRLT', '*project')
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 22 Jun 2019 14:33
by Ron Melby
Duh, yes it was working, I could not see otbl is all, which I expected to see. the problem is there is no problem.
re locals i fix it all up after I get it working, and I try to local everything I can.
function rtvTBL(stbl, styp)
styp = styp or '*project'
if styp == '*project' then
Filename = rtvProjectTBL(stbl)
end
if styp == '*global' then
Filename = rtvGlobalTBL(stbl)
end
local otbl = (load(Filename)) -- load the table from file
return otbl or {}
end
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 22 Jun 2019 15:01
by tatewise
It is far, far better to use
local to start with, especially for tables, otherwise, guess what, you can't see them
Re: table lookups (when we was fab) or memoization (gangnam style)
Posted: 23 Jun 2019 20:00
by Ron Melby
the memoization didnt speed it up that much, but now I know how to do it.