{"id":10721,"date":"2020-07-08T11:54:11","date_gmt":"2020-07-08T11:54:11","guid":{"rendered":"https:\/\/fhug.org.uk\/kb\/?post_type=kb_article&p=10721"},"modified":"2020-12-11T14:09:18","modified_gmt":"2020-12-11T14:09:18","slug":"run-time-and-memory-use-advice","status":"publish","type":"kb_article","link":"https:\/\/www.fhug.org.uk\/kb\/kb-article\/run-time-and-memory-use-advice\/","title":{"rendered":"Run Time and Memory Use Advice"},"content":{"rendered":"

Introduction<\/h2>\n

Some Plugins perform intensive repetitive operations, which on a large database in excess of 10,000 Individuals, may take a long time or need large amounts of memory.<\/p>\n

This article suggests how these resources can be minimised by using a few simple techniques. If the Plugin run time is measured in minutes rather than seconds, then even a 10% saving becomes significant.<\/p>\n

Global v Local<\/h2>\n

As a general rule local variables are more efficient than global variables, but there are exceptions.<\/p>\n

When a function requires a lookup table of constants, such as below, then it is faster if it is global.<\/p>\n

TblLookup = { A=1,E=2,I=3,O=4,U=5 }<\/pre>\n
This is because the global table is only created once, whereas a local table is created every time the function is called.<\/p>\n
It can also help to define local variables that reference global variables, especially an indexed table entry. e.g.<\/p>\n
local tblCode = TblLookup\r\nlocal intNumb = IntNumber\r\nlocal tblMode = TblMode[intNumb]<\/pre>\n
Where the same table lookup or other complex operation is required multiple times, then assign the result to a local variable and use it multiple times instead. e.g.<\/p>\n
if tblCode[strA] > 1 and tblCode[strA] < intLast then\r\n  intSum = intSum + tblCode[strA]\r\nend<\/pre>\n
becomes<\/p>\n
local intCode = tblCode[strA]\t\t-- Look up the value once \r\nif intCode > 1 and intCode < intLast then\r\n  intSum = intSum + intCode\t-- Use the local variable three times\r\nend<\/pre>\n
Some of the above techniques are illustrated in the\u00a0Soundex<\/a> (code snippet)\u00a0examples, where the\u00a0Global Variable Version\u00a0runs about 5 times faster than the\u00a0Local Variable Version, and the\u00a0Function Prototype Version\u00a0takes things one step further.<\/p>\n
Although a\u00a0local function\u00a0within another\u00a0function\u00a0clarifies its scope of use, a global\u00a0function\u00a0is faster. Maybe it is due to the global function being defined only once, whereas a local function is defined every time its container function is called.<\/p>\n
Progress Bar<\/h2>\n
The\u00a0Progress Bar<\/a> (code snippet)\u00a0provides useful feedback and a cancel option for long running Plugins. However, if the\u00a0Global Variable Version\u00a0is called too frequently, its own code can significantly extend the run time.<\/p>\n
Therefore, it may be better to avoid calling the\u00a0ProgressDisplay.Step\u00a0function on every loop step. e.g.<\/p>\n
ProgressDisplay.Start(\"Loop Progress\",9000)\r\nintSteps = 0\r\nfor i = 1, 9000 do\r\n  intSteps = intSteps + 1\r\n  if intSteps == 100 then\t\t-- Note that\tif intSteps % 100 == 0 then\tis slower\r\n    intSteps = 0\r\n    ProgressBar.Step(100)\t-- Only update Progress Bar every 100 steps\r\n  end\r\n  if ProgressBar.Stop() then break end\r\nend<\/pre>\n
This problem has been mitigated in the\u00a0Function Prototype Version\u00a0by only updating the display when necessary instead of every Step.<\/p>\n
Do not make the\u00a0ProgressBar.Stop()\u00a0function conditional on the\u00a0intSteps\u00a0count, otherwise it may make interrupting the loop and other interactions less responsive.<\/p>\n
Large Files<\/h2>\n
Sometimes it is necessary to process the contents of large files line by line. For this it is much faster to use\u00a0table.insert<\/code>\u00a0and\u00a0table.concat<\/code>\u00a0than string concatenation\u00a0strText = strText..strLine. e.g.<\/p>\n
local tblText = {}\r\nfor strLine in io.lines(strFile) do\t\t-- Read through the file line by line\r\n  strLine = strLine:gsub(\"abc\",\"xyz\")\r\n  table.insert(tblText,strLine)\t\t-- Insert the line of text as the next table entry\r\nend\r\nlocal strText = table.concat(tblText,\"\\n\")\t-- Concatenate the lines of text separated by newline\r\nSaveStringToFile(strText,strFile)\t\t-- See the Save String To File (code snippet)<\/pre>\n
Large Tables<\/h2>\n
Very large tables of data can arise, say when keeping results of each\u00a0Individual Record\u00a0in the database compared with every other\u00a0Individual Record. For smaller databases up to\u00a010,000 Individuals, this amounts to less than\u00a050,000,000\u00a0entries, but quickly escalates for larger databases, and can exhaust available memory.<\/p>\n
To avoid this problem the table of results should be sorted and the lowest entries pruned off. e.g.<\/p>\n
if intScore >= intMinimum then\t\t\t\t-- Continue if score is above lowest retained Results entry\r\n  table.insert(tblResults,{ Score=intScore, ... })\r\n  if #tblResults >= 2000 then\t\t\t-- Prune low scores from Results to avoid exhausting memory\r\n    table.sort( tblResults, function(tblA,tblB) return tblA[\"Score\"] > tblB[\"Score\"] end )\r\n    for i = 1 , #tblResults \/ 2 do\r\n      table.remove(tblResults)\t-- Remove the lower 50% of the sorted Results\r\n    end\r\n    intMinimum = tblResults[#tblResults][\"Score\"]\r\n  end\r\nend<\/pre>\n
Data Tables<\/h2>\n
When testing for alternative data values it is tempting to use\u00a0if \u2026 then \u2026 else \u2026<\/code>\u00a0structures, but when there are more than a few values it can become inefficient. Consider the following where each data reference tag is tested several times:<\/p>\n
ptrIndi = fhNewItemPtr()\r\nptrIndi:MoveToFirstRecord(\"INDI\")\r\nwhile ptrIndi:IsNotNull() do\r\n  local ptrData = fhNewItemPtr()\r\n  ptrData:MoveToFirstChildItem(ptrIndi) \r\n  while ptrData:IsNotNull() do\r\n    local strTag = fhGetTag(ptrData)\r\n    if strTag == \"NAME\" then\r\n      -- Handle names\r\n    elseif strTag == \"FAMS\" then\r\n      -- Handle spouse\r\n    elseif strTag == \"SOUR\" then\r\n      -- Handle source\r\n    end\r\n    ptrData:MoveNext()\r\n  end\r\n  ptrIndi:MoveNext()\r\nend<\/pre>\n
The following data table method is more efficient, as each data reference is only tested once. It becomes even more efficient as the number of alternatives increases. The only condition is that all the functions must support the same parameters.<\/p>\n
function HandleNames(ptrData)\r\n  -- Handle names here\r\nend\r\n\r\nfunction HandleSpouse(ptrData)\r\n  -- Handle spouse here\r\nend\r\n\r\nfunction HandleSource(ptrData)\r\n  -- Handle source here\r\nend\r\n\r\nfunction Null()\r\n  -- Handle anything else\r\nend\r\n\r\ntblWhat = {\t\t-- Translate data tag to function\r\n  NAME = HandleNames;\r\n  FAMS = HandleSpouse;\r\n  SOUR = HandleSource;\r\n}\r\n\r\nptrIndi = fhNewItemPtr()\r\nptrIndi:MoveToFirstRecord(\"INDI\")\r\nwhile ptrIndi:IsNotNull() do\r\n  local ptrData = fhNewItemPtr()\r\n  ptrData:MoveToFirstChildItem(ptrIndi) \r\n  while ptrData:IsNotNull() do\r\n    local strTag = fhGetTag(ptrData)\r\n    local action = tblWhat[strTag] or Null\r\n    action(ptrData)\t\t-- Call one of the functions above\r\n    ptrData:MoveNext()\r\n  end\r\n  ptrIndi:MoveNext()\r\nend<\/pre>\n
 <\/p>\n","protected":false},"template":"","fh_version":[14,15,739],"skill_level":[18,17],"topic":[73],"class_list":["post-10721","kb_article","type-kb_article","status-publish","hentry","fh_version-v5","fh_version-v6","fh_version-v7","skill_level-advanced","skill_level-intermediate","topic-writing-plugins"],"_links":{"self":[{"href":"https:\/\/www.fhug.org.uk\/kb\/wp-json\/wp\/v2\/kb_article\/10721","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fhug.org.uk\/kb\/wp-json\/wp\/v2\/kb_article"}],"about":[{"href":"https:\/\/www.fhug.org.uk\/kb\/wp-json\/wp\/v2\/types\/kb_article"}],"wp:attachment":[{"href":"https:\/\/www.fhug.org.uk\/kb\/wp-json\/wp\/v2\/media?parent=10721"}],"wp:term":[{"taxonomy":"fh_version","embeddable":true,"href":"https:\/\/www.fhug.org.uk\/kb\/wp-json\/wp\/v2\/fh_version?post=10721"},{"taxonomy":"skill_level","embeddable":true,"href":"https:\/\/www.fhug.org.uk\/kb\/wp-json\/wp\/v2\/skill_level?post=10721"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.fhug.org.uk\/kb\/wp-json\/wp\/v2\/topic?post=10721"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}