The OP has kindly been testing various refinements of the memory management to improve robustness with very large datasets. Most of the memory usage is in building and comparing large tables, and introducing additional garbage collection to these steps was relatively ineffective as Lua already handles tables very well, so there was little to collect.
The plugin is structured with an orchestrating function that controls when tables are built and compared, and the best strategy was to have no garbage collection during these individual processes, but to ensure that memory was immediately freed once the tables were finished with, either when they went out of scope, or by deliberately setting them to nil. Another advantage was that relatively few collections were required, so impact on performance was minimal.
However, there is one issue that I cannot resolve. The test plugin writes a log of its memory use during the various steps, and the first attachment shows a typical report from my largest test dataset of 65k individuals and around half a million facts.
- RAM use.PNG (32.35 KiB) Viewed 1458 times
- Lua Error.png (39.39 KiB) Viewed 1458 times
My qualitative hypothesis is that fhSQL() (or most probably the ADOBD object it creates) somehow has its own memory space that is not being cleared properly between runs, but we’re very much getting into the nitty gritty plumbing here that is above my pay grade!
Any suggestions for how to resolve this? I don’t know if fhSQL() was written by FHUG contributors or purely in-house by CP. I’ll raise it with them if we can’t resolve it here, but I suspect there may not be much they can do about it if it’s a limitation in a non-FH control or library.
.
