* Problem with code snippet

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Problem with code snippet

Post by ColeValleyGirl »

This code snippet (taken from plugins:code_snippets:zip_file_create|> Zip File Create & Extract (code snippet)) causes a plugin to hang:

Code: Select all

function extractZip(zipFile,folder)
    --  Create com object to work with Files and Folders
    local shell = luacom.CreateObject("Shell.Application")
    local source = shell:NameSpace(zipFile)  -- Get the zip file
    local items = source:items()
    local dest = shell:NameSpace(folder)   -- Get the destination folder file
    dest:CopyHere(items)    --  Copy the Zip to the Folder
    repeat                  --  Wait for copy to Zip to complete
    fhSleep(1000,500)
    until items.count == dest:items().count
end
The zip file is extracted correctly (with the normal Windows progress but the plugin hangs tereafter.

The problem goes away if I remove this code:

Code: Select all

    repeat                  --  Wait for copy to Zip to complete
    fhSleep(1000,500)
    until items.count == dest:items().count
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

I think Jane wrote that snippet, and I've experimented with it.

It seems that items.count is the number of files within the ZIP file, and that dest:items().count is the number of files in the target folder. So if you are extracting into a folder that is not initially empty then the counts never match.

By counting number of files already in folder beforehand and including that in repeat until loop it works OK.

Code: Select all

function extractZip(zipFile,folder)
    --  Create com object to work with Files and Folders
    local shell = luacom.CreateObject("Shell.Application")
    local source = shell:NameSpace(zipFile)  -- Get the zip file
    local items = source:items()
    local dest = shell:NameSpace(folder)   -- Get the destination folder file
    local files = dest:items().count       -- Count of files already in folder
    dest:CopyHere(items)    --  Copy the Zip to the Folder
    repeat                  --  Wait for copy to Zip to complete
    fhSleep(1000,500)
    until files + items.count >= dest:items().count
end
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

That works, and makes sense. Thanks, Mike.
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

The only slight snag is when the files in the target folder are essentially the same as in the ZIP file.
i.e. You are extracting subsequent versions of similar ZIP files into the same folder and overwriting the earlier files.
Then the counts test will succeed almost immediately.
The only absolutely satisfactory solution is to always use an empty target folder.
So I am not sure how best to update the Code Snippet?
Perhaps it needs to advise that it only works for empty target folders, and discuss the workaround I posted here?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

To be honest, the sleep statement is somewhat redundant... as far as I can tell, isn't invoked until after the extract completes -- unless you can see otherwise.
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

It is needed to check the extract actually worked, because there appears to be no error response from extraction process.
c.f. You Research Planner 2 which fails to extract without any error message.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

Best publish it with a warning then... However, the snippet doesn't report an error in any circumstances, so I'm not convinced it's useful.
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

You are correct, the snippet is flawed, because its repeat loops never terminate if there is an error.
So it needs 'improving' further to explain that, or include an error message escape from the loop, perhaps when the dest:items().count stops increasing?
Maybe Jane can give us her thoughts?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

This alteration to the repeat until loops seems to work.
If the dest:items().count reaches the items.count then it returns true.
Every 1 second it checks if count has stuck, and if so then returns false.
Does this look OK to you?

Code: Select all

require "luacom"

function extractZip(zipFile,folder)
    --  Create com object to work with Files and Folders
    local shell = luacom.CreateObject("Shell.Application")
    local source = shell:NameSpace(zipFile)  -- Get the zip file
    local items = source:items()
    local dest = shell:NameSpace(folder)   -- Get the destination folder file
    dest:CopyHere(items)    --  Copy the Zip to the Folder
    repeat                  --  Wait for copy to Zip to complete
        local count = dest:items().count
        if count == items.count then return true end
        fhSleep(1000,500)
    until count == dest:items().count
    return false
end

function buildZip(zipFile,folder)
    -- Create Empty Zip File
    local strZipHeader = 'PK'..string.char(5,6)..string.rep(string.char(0), 18)
    zip = assert(io.open(zipFile, 'w'))
    zip:write(strZipHeader)
    zip:close()
    --  Create com object to work with Files and Folders
    local shell = luacom.CreateObject("Shell.Application")
    local source = shell:NameSpace(folder)  -- Get the source folder
    local items = source:items()
    local dest = shell:NameSpace(zipFile)   -- Get the destination Zip file
    dest:CopyHere(items)    --  Copy the Source folder to the Zip file 
    repeat                  --  Wait for copy to Zip to complete
        local count = dest:items().count
        if count == items.count then return true end
        fhSleep(1000,500)
    until count == dest:items().count
    return false
end

if not extractZip(strzipfile,strfolder) then fhMessageBox("Extract Zip File Failed.") end

if not buildZip(strzipfile,strfolder) then fhMessageBox("Create Zip File Failed.") end

Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

Mike, that last code sniippet unerringly returns 'false' when extracting the Research Planner Help files...
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

That is odd, as it works fine for me if I substitute it into Research Planner V2.0 in place of its extractZip(...).

[ EDIT: However, on closer inspection, my proposed code only works reliably if the target folder is empty (as specified in the code snippet advice), and in the Research Planner that is not the case as that folder is Research Planner that already contains the .cfg and .fhf files, so your variant involving the files count is necessary. However, all it is actually checking is that the Research Planner Help 2.0 sub-folder gets created and not the actual contents.
I wonder if there is a way to check that sub-folder contents by using shell:NameSpace(subfolder) where subfolder is the ...\Research Planner\Research Planner Help 2.0 sub-folder? ]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

This is weird -- it fails consistently for me if I do that.

The extraction completes before the loop is entered -- is that not the case for you?

If the directory is initially empty except for the zip file itself, dest:items().count is calculated after the extraction completes *including* the Help directory and the zip file , therefore 2 ; which is (a) always greater than items.count (1) and (b) never changes during the course of the loop, so appears to be stuck. So the extraction succeeds but claims it didn't.
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

We came to the same conclusion at the same time...
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

I believe I have a solution.
Part of the problem was that the code snippet did not take account of existing items in the target folder.
But that is further confounded by the Research Planner Plugin creating the Research Planner Help 2.0 subfolder before extracting the ZIP file. Therefore, extracting the ZIP adds no items to the target folder Research Planner.dat and so there is nothing to count in that folder to check the extraction has worked.

BTW: The earlier version of function extractZip(...), and the original code snippet, did not correctly detect or return a true/false indication of ZIP file extraction.

I have modified the function extractZip(...) code snippet to account for existing items.
I have deleted the lfs.mkdir(HelpFileDirectory) from local function GetHelpFile().
Now extractZip(...) correctly returns true or false depending on whether the extraction adds any item(s), which in this case is just the Research Planner Help 2.0 subfolder.
I will see if the function buildZip(...) needs a similar treatment.

Code: Select all

function extractZip(zipFile,folder)
    --  Create com object to work with Files and Folders
    local shell = luacom.CreateObject("Shell.Application")
    local source = shell:NameSpace(zipFile)  -- Get the zip file
    local items = source:items()
    local dest = shell:NameSpace(folder)   -- Get the destination folder file
    local files = dest:items().count + items.count -- Count of top level items already in folder plus those in zip file
    dest:CopyHere(items)    --  Copy the Zip to the Folder
    repeat                  --  Check that copy of Zip completed
        local count = dest:items().count
        if count == files then return true end
        fhSleep(1000,500)
    until count == dest:items().count
    return false
end
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

Thanks for that Mike -- it works.

Until I find the next bug :)

No, seriously, thank you.
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

I believe that the function buildZip(...) posted last Friday afternoon is satisfactory.

So I've updated the plugins:code_snippets:zip_file_create|> Zip File Create & Extract (code snippet) with the new Code scripts, and added testing of the returned values to the Usage section, but removed the requirement that destination folders must be empty.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

Mike, unfortunately the extract snippet doesn't work in some (unknown) configuration under Wine. I might try the luazip library...
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

Yes, some internal Windows features are not always fully implemented in emulators such as Wine and Crossover.

I've just looked at the plugins:getting_started|> Getting Started Writing Plugins link to LuaZip is broken.
Similarly on http://lua-users.org/wiki/CompressionAndArchiving the [LuaZip] link is broken.
i.e. http://www.keplerproject.org/luazip/ no longer exists!
Have you a reference link?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

No, but I was going to try the code here: https://stackoverflow.com/a/7140474/1943174

Of course, it will be another loadrequire -- which will involve a manual download and extraction for users if Win, Crossover etc. (similar to the penlight situation) so whether it's an improvement for them I'm not sure.

Edited: there's a Github page with documentation
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

Yes, it seems to have moved from kepler to mpeterv and the link is http://mpeterv.github.io/luazip/.
I think its advantage is it is already built into FH and should not need such a complex download and installation.
Yes, it works with plugins:code_snippets:module_require_with_load|> Module Require With Load (code snippet) as advertised, but still may need manual download for Wine & Crossover.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

It will only handle reading, not writing, but for simple use cases that should be enough, and -- as you say-- all that will be needed is a loadrequire.
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

Mike, does this work for you -- I believe it handles all subdirectories properly -- anything else you would change? It does use Penlight as is my wont, but that wouldn't be hard to eliminate.

Code: Select all

do
    require ("luacom") --Microsoft's Component Object Model
    require ("lfs")
    function fhloadrequire(module,extended)
        local function httpRequest(url)
            local http = luacom.CreateObject("winhttp.winhttprequest.5.1")
            http:Open("GET",url,false)
            http:Send()
            http:WaitForResponse(30)
            return http
        end -- local function httpRequest
        if not(extended) then extended = module end
        local function installmodule(module,filename,size)
            local bmodule = false
            if not(filename) then
                filename = module..'.mod'
                bmodule = true
            end
            local storein = fhGetContextInfo('CI_APP_DATA_FOLDER')..'\\Plugins\\'
            -- Check if subdirectory needed
            local path = string.match(filename, "(.-)[^/]-[^%.]+$")
            if path ~= "" then
                path = path:gsub('/','\\')
                -- Create sub-directory
                lfs.mkdir(storein..path)
            end
            local attr = lfs.attributes(storein..filename)
            if attr and attr.mode == 'file' and attr.size == size then return true end
            -- Get file down and install it
            local url = "http://www.family-historian.co.uk/lnk/getpluginmodule.php?file="..filename
            local isOK, reply = pcall(httpRequest,url)
            if not isOK then
                fhMessageBox(reply.."\nLoad Require module finds the Internet inaccessible.")
                return false
            end
            local http = reply
            local status = http.StatusText
            if status == 'OK' then
                --           local length = http:GetResponseHeader('Content-Length')
                local data = http.ResponseBody
                if bmodule then
                    local modlist = loadstring(http.ResponseBody)
                    for x,y in pairs(modlist()) do
                        if type(x) == 'number' and type(y) == 'string' then
                            x = y -- revert to original 'filename' ipairs modlist
                            y = 0
                        end -- otherwise use ['filename']=size pairs modlist
                        if not(installmodule(module,x,y)) then
                            break
                        end
                    end
                else
                    local function OpenFile(strFileName,strMode)
                        local fileHandle, strError = io.open(strFileName,strMode)
                        if not fileHandle then
                            error("\n Unable to open file in \""..strMode.."\" mode. \n "..
                                strFileName.." \n "..tostring(strError).." \n")
                        end
                        return fileHandle
                    end -- OpenFile
                    local function SaveStringToFile(strString,strFileName)
                        local fileHandle = OpenFile(strFileName,"wb")
                        fileHandle:write(strString)
                        assert(fileHandle:close())
                    end -- SaveStringToFile
                    SaveStringToFile(data,storein..filename)
                end
                return true
            else
                fhMessageBox('An error occurred in Download please try later')
                return false
            end
        end
        local function requiref(module)
            require(module)
        end
        local _, err = pcall(requiref,extended)
        if err then
            if  err:match("module '"..extended:gsub("(%W)","%%%1").."' not found") then
                local ans = fhMessageBox(
                    'This plugin requires '..module..' support, please click OK to download and install the module',
                    'MB_OKCANCEL','MB_ICONEXCLAMATION')
                if ans ~= 'OK' then
                    return false
                end
                if installmodule(module) then
                    package.loaded[extended] = nil -- Reset Failed Load
                    require(extended)
                else
                    return false
                end
            else
                fhMessageBox('Error from require("'..module..'") command:\n'..(err or ''))
                return false
            end
        end
        return true
    end
    if not fhloadrequire("pl","pl.init") then return end -- Load required Penlight modules
    pl = require("pl.import_into")
    if not fhloadrequire("zip") then return end
end

function ExtractZip(zipPath, zipFilename, destinationPath)
    local zfile, err = zip.open(zipPath.."\\"..zipFilename) --open the zip file for processing

    local function CopyFile(file)
        local currFile, err = zfile:open(file.filename) --open a file within the zipfile
        local currFileContents = currFile:read("*a") -- read entire contents of current file
        local hBinaryOutput = io.open(destinationPath .."\\".. file.filename, "wb") --open an outputfile
        -- write current file inside zip to a file outside zip
        if hBinaryOutput then
            hBinaryOutput:write(currFileContents) --write the new file as a copy of the file within the zipfile
            hBinaryOutput:close() --close the new file
        end
        currFile:close() --close the file within the zipfile
    end

    --get the identity of any subdirectories and create them in destinationPath
    for file in zfile:files() do --file is a complete path and file name
        local newdir = path.dirname(file.filename)
        if not path.exists(destinationPath.."\\"..newdir) then dir.makepath(destinationPath.."\\"..newdir) end
    end
    -- iterate through each file inside the zip file
    for file in zfile:files() do
        if path.basename(file.filename) == "" then --this is a directory
            --do nothing -- directory creation was handled earlier
        else --this is a file
            CopyFile(file)
        end
    end

    zfile:close()
end

local cstrPluginDir = fhGetPluginDataFileName("LOCAL_MACHINE",true)
local DestDirectory = cstrPluginDir.."\\Test"
local zipname = "Test"
ExtractZip(cstrPluginDir, zipname..".zip", cstrPluginDir)

User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

That essentially works for me, but needs a couple of tweaks.

The last line ExtractZip( needs DestDirectory) as last parameter.

Inside function ExtractZip does local zfile, err = zip.open need an err handler if zfile is nil?

We don't know what Windows services this LuaZip library relies upon, so may still have issues with Wine/Crossover.

Are we satisfied that the Help.zip file download is always working correctly with Wine/Crossover?
If not, then maybe changing the zip extractor is not the solution.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with code snippet

Post by ColeValleyGirl »

Yes, re test harness destination directory but for publication as a snippet it should be as simple as possible and avoid the verbiage in the last 4 lines (I just wanted a test harness that could be isolated from everything else).

Yes, re error handler but only sensible thing to do in a code snippet is to report the error? (The Research Planner acts on download errors and doesn't invoke the extraction if the file doesn't exist and a download failed).

Next steps would be to ask Davidf and Valkrider to test in their environment so we know that it works for them?

The Help file download may be subject to the luacom problems in some circumstances... unless you've got any bright ideas about circumventing it, documentation and a manual download is my best idea. I strongly believe that offline help is the best option for complex plugins.

Were there any special steps you took to make libraries available at how_to:family_historian_v6_on_crossover_wine#plugin_library_modules|> Plugin Library Modules? It will need to be updated to include luazip before I ask anyone not using Windows to test. Or did you just use the dll that came with FH on Windows?
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with code snippet

Post by tatewise »

All agreed.

If offline help is a requirement, then download is a necessity and must live with Wine/Crossover issues.
As you know, I prefer online help.

The library modules are each just a compressed ZIP of whatever is in the Plugins folder.
Most are in a subfolder, but LuaZip appears to be just a zip.dll file.
I can add that to the others if you like?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply