* Export Gedcom / FH7 problem?

[kempons]

(Below was posted on FH mailing list - moving discussion here for debugging with Mike)

Hi,

I am experiencing an issue with producing a gedcom of my FH file via Export Gedcom plugin for export to my TNG website since upgrading to FH v7.

The same gedcom successfully exported via Export Gedcom plugin under FH v6, however now since upgrading to v7 I am experiencing an "out of memory" issue. I have checked and upgraded to the latest Export Gedcom plugin v4.1, however that has not solved the problem.

My FH file is quite large, approximately 465K individuals & 140K families if that is relevant.


Below is the error message produced (if this helps):

[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:4343: not enough memory for buffer allocation
stack traceback:
[C]: in function 'string.gsub'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:4343: in upvalue 'doPrune'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:8435: in local 'useRules'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:9488: in function 'ExportGedcom'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:9199: in function <[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:9189>
(...tail calls...)
[C]: in function 'iuplua.MainLoop'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."] in field 'ShowDialogue'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:9370: in function 'GUI_MainDialogue'
[string "C:\ProgramData\Calico Pie\Family Historian\Pl..."]:9546: in main chunk

Any ideas please? I would like to update my website before the end of the year.
Thanks.

Regards,
Andrew

[tatewise]

Sorry about that.
I have changed the way the FH v7 plugin processes long Notes with CONT/CONC lines in order to handle ‘rich text’ conversions. Thus it can end up with very long text strings if you have very long Notes.
However, it still is a bit odd that it should run out of memory handling strings.

Can you please edit the Plugin script to see if it is the 'string.gsub' function that is the cause?

In the Tools > Plugin dialogue select the plugin and click Edit... then about line 4343 make the following changes by inserting double hyphen -- at the start of each line to hide the statements which are only necessary to produce 100% strict GEDCOM format that most products don't care about:
-- strVal = strVal:gsub("^([23] AGE [<>]) +","%1") -- Remove invalid AGE and head & tail space chars -- V4.0
-- strVal = strVal:gsub("^ *(%d [_%u]+) *","%1&") -- Preserve one space after tag even if on tail -- V4.0
-- strVal = strVal:gsub("^ *(.-) *$","%1") -- :gsub(" +$","") -- V4.0
-- strVal = strVal:gsub("^(%d [_%u]+)&","%1 ")

Does that now run without the memory buffer error?

[kempons]

Mike,

After commenting out those lines and re-running Export Gedcom plugin unfortunately I experience exactly the same error.

It appears the issue is elsewhere in the updated Export Gedcom plugin. Let me know what to try next and we can hopefully debug this issue.

Thanks.
Andrew

[tatewise]

Andrew, thank you, but when providing feedback please provide all you know, like the actual error message.

I have reviewed the memory garbage collection features of Lua and have got an idea.

Try inserting the collectgarbage("step",0) statement on the line before the return strVal statement at about line 4346:

Code: Select all

		strVal = strVal:gsub("^(%d [_%u]+)&","%1 ")
		collectgarbage("step",0)
		return strVal
	end -- local function doPrune

[kempons]

Mike,

Success. By adding the garbage collection line, the gedcom successfully exported.
Now to update my website.

Thanks.
Andrew

[tatewise]

Where we've inserted that collectgarbage("step",0) statement is probably not ideal but you were familiar with that area.

It probably only needs to be run once per record rather than for every text field.

Let me know when you have finished updating your online TNG website and we can experiment with alternative positions.

[kempons]

Mike,

I have just updated the TNG website now so I can assist you in further debugging of the Export Gedcom plugin.

Kindly send through instructions and I will make the changes and send you the results. Thanks for all your help with this.

[tatewise]

Thank you for your help. Please edit the following changes into the Export Gedcom File script.

Restore the lines 4341 - 4347 (approx) to their original state:

Code: Select all

		end
		strVal = strVal:gsub("^([23] AGE [<>]) +","%1")							-- Remove invalid AGE and head & tail space chars -- V4.0
		strVal = strVal:gsub("^ *(%d [_%u]+)  *","%1&")							-- Preserve one space after tag even if on tail -- V4.0
		strVal = strVal:gsub("^ *(.-) *$","%1")									-- :gsub(" +$","") -- V4.0
		strVal = strVal:gsub("^(%d [_%u]+)&","%1 ")
		return strVal
	end -- local function doPrune

Then insert the collectgarbage("step",0) statement at about line 9441 and also line 9499 as shown here between the if progbar.Stop() and the end statement in both cases:

Code: Select all

			if progbar.Stop() then break end									-- Cancel Analysing?
			collectgarbage("step",0)
		end

Run the plugin as before and hopefully it will not run out of memory but the collectgarbage() overhead is reduced.

[Ron Melby]

Mike,
this may or not be useful, I expect it is going to vary depending on machine attr, fh version, os, size of GEDCOM and what particular codepaths you are taking for any given set of user circumstances, however, the default gc in lua is ... more spiv than toff.

I have been trying to debug, figure memory, optimize my hog slaughtering _famOBJ program.

require '_STD_MAT'
require '_STD_RTV'
require '_STD_FMT'
require '_STD_SAVRST'
require '_STD_SYS'
require '_STD_SORT'

fhSetStringEncoding('UTF-8')

-- sets arg as value for pause of collector. Returns prev value for pause.
local _ = collectgarbage('setpause', 50)
-- sets arg as value for step multiplier of collector. Returns prev value for step.
local _ = collectgarbage('setstepmul', 300)

.... and program code goes here i.e:

local function writeIndent(file, level)
for _ = 1, level do
file:write(' ')
end

this has worked best for me, a balance between thrashing and doing nothing at all to help. maybe the global gc settings will not work in some of your programs. I have tried the step, 0 in certain --- bit gut left over from gsub and so on, and it didnt really help the issue no matter what. in debug I did a save of the table to disk, and read it back after a few quick statements ---out of memory. same with my way, but if I wait a couple three seconds, memory is reclaimed and I can see the tables and peruse them in debug. the step, 0 didnt seem to change it.

[kempons]

Mike,

After reverting the previous changes and moving the collectgarbage("step",0) to the if progbar.Stop() and the end statements as requested, I can confirm the successful export of my gedcom.

Thank you again.

[tatewise]

Thank you Mike, that will be built into the next version of the plugin.
Please confirm that it had no major impact on the plugin speed of exporting.

[kempons]

Not that I sit and watch the export as it takes some time, but the export appeared to be faster with the 2nd option you proposed for garbage collection.

[tatewise]

The Export Gedcom File V4.1.2 (18578) incorporates the garbage collection along with other new features.

[kempons]

Thank you Mike

[kempons]

Hi Mike,

Unfortunately I am again experiencing the "out of memory" error when exporting a gedcom via the Export Gedcom File plugin.

This happens when exporting the notes (~93% through the export)

Using FH v7.0.3 and the latest version of the Export Gedcom File plugin.

Can I turn on any flag to get more logs for the error to help troubleshoot?

Thanks
Andrew

[tatewise]

I have forced the memory garbage collection to be performed every time an exported GEDCOM line is changed instead of only once per exported record. Experiments in other Plugins also suggest that FH V7 needs more garbage collection than FH V6. However, adding extra garbage collection commands does not seem to significantly impact run time.
There are a few other minor updates in this version of the Plugin too but they should not affect you.

Try the attached Export Gedcom File plugin Version 4.4.1 Date 31 Mar 2021 that needs to be extracted from the ZIP file as it is too large to attach directly.

[kempons]

Hi Mike,

Unfortunately the updated ExportGedcom plugin does not fix the issue. I am still experiencing the "out of memory" error. In fact, it now appears a lot earlier in the export process, when exporting the individuals.

BTW, like the new features very much.
Thanks for your ongoing help assisting with this.

Regards,
Andrew

[tatewise]

When you say the problem happens at ~93% when exporting notes do you mean Research Notes that are near the end?
I have noticed that they are slower to handle in my tests but not sure why and it is on my list to investigate.
Please when reporting can you post the actual error message as you did last December as that gives a line number that helps me identify the problem script.

However, the Version 4.4.1 prototype causes the problem much earlier so that is not a solution.
If you revert to the Plugin Store Version 4.4 does that reliably say 'out of memory' at the same Research Note record?
What is the actual error message?
Is that or the next Research Note particularly large or unusual in any way?

Are you using default settings for the (TNG) The Next Generation export mode?

If it is Research Notes that are the cause then on the Extra Options tab try setting the Research Note option bottom right to Remove entirely. Does that help? Can you manage without those Research Notes in TNG?

My general observation is that FH V7 Lua 5.3 seems to be more memory error prone than FH V6 Lua 5.1.

[kempons]

Hi Mike,

There are no Research notes within my FH project as yet - they were notes from an FTM gedcom I recently imported and merged. Since first reporting this error, I have utilised the Clean Up Notes plugin to convert these to Local notes.

As requested, I have reverted to the v4.4 plugin of ExportGedcom from the store and again run through the export gedcom process. Attached are various screen shots showing the settings, extra options page and the error message observed.

Do you know of a way to force stack traces to output when there is an issue with a plugin? I have not been able to obtain anything to assist you further other than "Out of memory" as shown in the attached.

Happy to assist further in debugging this issue with you.

Regards,
Andrew

[tatewise]

I guess whatever was in the Note records causing the failure last Wednesday is now in the Individual record local Notes.
That may make it more difficult to find.

Is the current "not enough memory" error the same as the "out of memory" error you reported last Wednesday?
I have never seen an Lua Error like that!

They are very different from the original "not enough memory for buffer allocation" reported in December.

Anyway, I have been experimenting with more aggressive memory garbage collection and have a prototype for you to try.
Please try the attached Export Gedcom File plugin Version 4.4.4 Date 03 Apr 2021 in a ZIP file.
If it completes then the Result Set lists memory Usage in KB by Line No at the top every so often.
If we cannot get the Plugin to complete then it may need modifying to report usage another way.

[kempons]

Hi Mike,

After further investigations, troubleshooting and trial & error, I have managed to "fix" the "not enough memory" error and successfully run through your new ExportGedcom prototype.

I observed that the error was consistently occurring on a specific record, so I edited and then removed the note entry. Problem solved - the gedcom export ran successfully through to the end.

FYI, below is the relevant memory usage output:

"-9" "0" "Media Object" "Media Conversion ( TNG )" "0 @I543587@ INDI" "8587825" "Collect Garbage Memory" "Usage=167452.00488281"
"-9" "0" "Place Record" "Delete Record Ident ( RMT TNG )" "0 @P6757@ _PLAC Parish Church, Sittingbourne, Kent, England, United Kingdom" "9973080" "Collect Garbage Memory" "Usage=215380.79394531"
"-9" "0" "Place Record" "Delete Record Ident ( RMT TNG )" "0 @P16950@ _PLAC Buckland, Buckshire, England" "10041495" "Collect Garbage Memory" "Usage=241138.11621094"
"-9" "0" "Place Record" "Delete Record Ident ( RMT TNG )" "0 @P27693@ _PLAC Fitzroy North, Victoria, Australia" "10053660" "Collect Garbage Memory" "Usage=218491.96875"
"-9" "0" "Place Record" "Delete Record Ident ( RMT TNG )" "0 @P39212@ _PLAC Hutton, Alberta, Canada" "10066144" "Collect Garbage Memory" "Usage=238380.82617188"

I attach the relevant note record that was causing all the grief. I am not aware of any particular limit I was hitting in size of notes, however I will now break this down into specific notes relevant to the census entries.

Thank you again for your assistance in troubleshooting this issue.

Kind regards,
Andrew

[tatewise]

Excellent Andrew, that has revealed a bug in the Export Gedcom File plugin when handling that Note text.

It is caused by the 4th line having hundreds of space characters in the middle of the name Mary . . . Ann WHITE
That upsets the algorithm that ensures each GEDCOM line is less than 255 characters but does not break on a space.
If you remove those spaces so it is simply Mary Ann WHITE then otherwise all should be OK.
There is no need to break the Note down at all.

I will devise a solution to that particular problem in the next version of the plugin.

[kempons]

Thank you Mike for all your help with this issue.

Kind regards,
Andrew