* LUA: howto upload binary file

Writing and using plugins for Version 5 and above.
Post Reply
User avatar
coret
Newbie
Posts: 4
Joined: 03 Nov 2019 18:45
Family Historian: V6
Contact:

LUA: howto upload binary file

Post by coret » 03 Nov 2019 19:07

Hi,

I'd like to make a plugin for Family Historian which uses the Genealogie Online Publication API. This basically is a HTTP request (POST) with file upload. But I can't get the file upload to work properly.

Here's the code:

Code: Select all

require 'luacom'

local http = luacom.CreateObject("winhttp.winhttprequest.5.1")

local url = 'https://api.coret.org/publish/create/v1/'
local api_key='***'

http:Open("POST",url,false)
boundary='----------goapipublishcreate'
http:SetRequestHeader("Content-Type", "multipart/form-data; boundary="..boundary)

parameters = ''
parameters = parameters..'--'..boundary..'\r\nContent-Disposition: form-data; name="api_key"\r\n\r\n'..api_key..'\r\n'
parameters = parameters..'--'..boundary..'\r\nContent-Disposition: form-data; name="bestand_gedcom_zip"; filename="test.zip"\r\nContent-Type: application/x-zip-compressed\r\n\r\n'

local fh = assert(io.open('c:\\temp\\test.zip', "rb"))
local contents = assert(fh:read("*a")) 
fh:close()
parameters = parameters..contents..'\r\n'
parameters = parameters..'--'..boundary..'--'

http:SetRequestHeader("Content-Length", string.len(parameters))
http:SetRequestHeader("Connection", "close")

print(string.len(parameters))  

http:Send(parameters)
http:WaitForResponse(30)
I've used a network-sniffer (Wireshark) to see what the plugin sends to server and I see that only a part of the parameters variable is send. It's the part until the first zero byte in the file.

I've printed the length of the parameters variable (which includes the binary contents of test.zip) and from that I can conclude that the parameters variable does contains the complete file. But when call http:Send only the part of the parameters variable upon the (first) zero byte is processed.

Any LUA experts here who can suggest solution?
Any alternative modules available to file uploading in stead of luacom.CreateObject("winhttp.winhttprequest.5.1")?

Regards,
Bob

User avatar
tatewise
Megastar
Posts: 16642
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: LUA: howto upload binary file

Post by tatewise » 03 Nov 2019 20:45

Welcome to the FHUG Bob.

Unfortunately the luacom library only supports http and not the https protocol.

The https protocol needs the LuaSec library that is not currently available for FH.
Use Google search for LuaSec and also LuaSec for Windows to see such as:
https://luarocks.org/modules/brunoos/luasec
http://notebook.kulchenko.com/programmi ... and-luasec

I have a similar problem, and have requested Calico Pie to investigate, but no solution yet.
Since so many online services now require https this is an increasingly significant issue.

Please report your problem to Calico Pie using the Knowledge Base > Problem Reporting method.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
coret
Newbie
Posts: 4
Joined: 03 Nov 2019 18:45
Family Historian: V6
Contact:

Re: LUA: howto upload binary file

Post by coret » 03 Nov 2019 22:29

Hi Mike,
Unfortunately the luacom library only supports http and not the https protocol.
That's not my experience so far?! Calling the Genealogie Online Publish API (without the file-upload part), works just fine. As I see it, this is not the achievement of luacom, but the winhttp.winhttprequest.5.1 component.

A simple working https-with-luacom script:

Code: Select all

require 'luacom'
local http = luacom.CreateObject("winhttp.winhttprequest.5.1")
http:Open("GET","https://www.genealogieonline.nl/en/",false)
http:Send()
http:WaitForResponse(30)
local status = http.StatusText
if status == 'OK' then
	print("ok: data="..http.Responsetext)
else
	print("not ok: eror in http request, status="..status)
end
Bob

User avatar
Jane
Site Admin
Posts: 7751
Joined: 01 Nov 2002 15:00
Family Historian: V6.2
Location: Somerset, England
Contact:

Re: LUA: howto upload binary file

Post by Jane » 04 Nov 2019 08:28

I have used Luacom with https as well with out problems, I think the limitation only affects the lua sockets module.
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."

User avatar
tatewise
Megastar
Posts: 16642
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: LUA: howto upload binary file

Post by tatewise » 04 Nov 2019 12:28

My understanding is that https works with luacom if no login is required.
But once login is needed, then lua SSL sockets require the LuaSec library.
( Note that luacom.CreateObject("winhttp.winhttprequest.5.1") is what handles the POST requests. )

It looks like https://www.genealogieonline.nl/en/ needs a Login so eventually you may need the LuaSec library.
That is where I ran into similar file transfer problems.
The reason your simple example works is that no Login is required to download the HTML script of that URL.

However, returning to your original script, the is a problem with embedding the ZIP file contents in the parameters.
If it contains a null byte then http:Send() will not send it, so it needs converting to the %00 protected URL format.
So use contents = contents:gsub("%z","%%00") which replaces every null byte with the %00 URL code.
BTW: %z represents the null byte and the double %% is needed to escape the % magic character.

That should overcome the http:Send() truncation problem, but other byte values may need URL protecting too.

Having said all that, if the objective is to upload a GEDCOM file to Genealogy Online then why do you need a Plugin, as there is a simple manual upload option (which I have just used)?
I tried to find out more about that web site but the https://www.genealogieonline.nl/en/help/ links are all Dutch and if English is chosen language then all the links are broken!
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
coret
Newbie
Posts: 4
Joined: 03 Nov 2019 18:45
Family Historian: V6
Contact:

Re: LUA: howto upload binary file

Post by coret » 04 Nov 2019 14:44

It looks like https://www.genealogieonline.nl/en/ needs a Login so eventually you may need the LuaSec library.
...
Having said all that, if the objective is to upload a GEDCOM file to Genealogy Online then why do you need a Plugin, as there is a simple manual upload option (which I have just used)?
Yes, you can simply upload your GEDCOM to Genealogy Online (for which you have to register/login). The point of the plugin is that a genealogists doesn't have to export their GEDCOM, goto https://www.genealogieonline.nl/ login and upload their GEDCOM (repeat for every time you want to update your data, as Genealogy Online is publish only, FH is for the editing). Just, select the "Publish on Genealogy Online" option in the Tools menu of FH and you're done :D
(and I saw your 'Test Export Gedcom File' on Genealogy Online ;) )
However, returning to your original script, the is a problem with embedding the ZIP file contents in the parameters.
If it contains a null byte then http:Send() will not send it, so it needs converting to the %00 protected URL format.
So use contents = contents:gsub("%z","%%00") which replaces every null byte with the %00 URL code.
BTW: %z represents the null byte and the double %% is needed to escape the % magic character.

That should overcome the http:Send() truncation problem, but other byte values may need URL protecting too.
Thanks for this suggestion, I'm gonna try it out shortly. Just to be clear: this ZIP file I'm trying to upload via the Genealogie Online API is part of the content of a multipart/form-data in the POST request, not the URL part. So while %00 is good for an URL, I don't think it's suitable for the content part of the request (but I'm gonna try it anyway!!!).

Bob

User avatar
tatewise
Megastar
Posts: 16642
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: LUA: howto upload binary file

Post by tatewise » 04 Nov 2019 16:34

Unfortunately, there is a flaw in your Plugin proposal.
The GEDCOM file in the Project folder uses many custom tags that will NOT be recognised by Genealogy Online.
One significant example is that Media (OBJE) records use a custom _FILE tag to link image file paths.
So those tags are not recognised by Genealogy Online and uploaded images are reported as Uploaded, not linked.

That is why I wrote the Export Gedcom File Plugin to compose GEDCOM dialects to satisfy many popular products.

BTW:
There are also reports of invalid PAGE and DATA and QUAY tags on some perfectly valid SOUR tags.
Those source citations are missing from Genealogy Online.
Those 'invalid' tags and several other problems are reported by GEDCOM Validator and GED-inline but I cannot see what some of the problems are, especially as FH is one of the most compliant programs available.
On Genealogy Online the individual page facts are not usually listed in date order.
It may be worth starting a new thread to discuss these issues.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
coret
Newbie
Posts: 4
Joined: 03 Nov 2019 18:45
Family Historian: V6
Contact:

Re: LUA: howto upload binary file

Post by coret » 04 Nov 2019 20:27

Unfortunally, the contents = contents:gsub("%z","%%00") suggestion didn't work. That is, the entire file was now transmitted, but where there should be a null-byte there was a %00 characters (3 bytes), which made the binary transmitted file invalid...

Another hack I can think of is to apply base64 encoding to the whole file so I circumvent the null-byte problem. I say hack because the size will 4/3 and I have to add functionality to do the base64 encoding to the plugin and to the API to recognize the base64 encoding and decode it, which I don't like.
The GEDCOM file in the Project folder uses many custom tags that will NOT be recognised by Genealogy Online.
One significant example is that Media (OBJE) records use a custom _FILE tag to link image file paths.
Yes, FH does use user-defined GEDCOM tags. For the _FILE tags, I assume this was done because GEDCOM version 5.5 and 5.5.1 only allow 30 characters with the FILE tag (<MULTIMEDIA_FILE_REFERENCE>), GEDCOM version 5.5.5 fixes this shortcoming. By using the _FILE tag the validation won't complain about the longer filenames, but other programs might not understand this user-defined tag of the FH GEDCOM which I think in general is an issue. As I am in control of the Genealogy Onlines GEDCOM reader, I could add functionality to understand this specific user-defined GEDCOM tag.
There are also reports of invalid PAGE and DATA and QUAY tags on some perfectly valid SOUR tags.
You have uploaded the FH example project's GEDCOM, which Genealogy Online automatically validated using the GED-inline component. The warnings about PAGE and QUAY under SOUR seems indeed incorrect, I've submitted an issue, see https://github.com/nigel-parker/gedinline/issues/5.

User avatar
tatewise
Megastar
Posts: 16642
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: LUA: howto upload binary file

Post by tatewise » 04 Nov 2019 21:05

Sorry, but I cannot help much more with the http:Send() with null bytes.
I presume there is something in the luacom http Send script that terminates text on a null byte.
Since 100MB is quite a big GEDCOM why bother with a ZIP file (at least while developing the Plugin).

The reason FH uses _FILE in OBJE records is that the FILE tag is NOT allowed in GEDCOM 5.5 in that context.
It is nothing to do with field size constraints.
Anyway, it is a misnomer to treat {1:30} as a maximum field size, when it is in fact the opposite.
GEDCOM 5.5 & 5.5.1 says:
"The field sizes show the minimum recommended field length within a database that is constrained to fixed length fields."
i.e. In databases with fixed length fields a minimum of 30 characters is recommended.
The same goes for all other field size constraints, and in modern day systems are largely irrelevant.

There are many, many more FH custom tags as explained in Knowledge Base > GEDCOM Extension List.
The most significant are _ATTR (c.f. GEDCOM 5.5.1 FACT), _SHAR & _SHAN, _ASID & _AREA, and _PLAC records.
All the FH custom tags are converted as necessary by the Export Gedcom File Plugin.

GED-inline has several other deficiencies, too numerous to list every one.
e.g.
You've not listed all the errors on GitHub, e.g. the DATA and EVEN tags, and GEDCOM 5.5.1.
A valid SOUR tag with just text (instead of a link to a source record) is reported as invalid.
Several valid AGE tag values involving > or < are reported as invalid.
A review of my Test Export Gedcom File publication reveals some such deficiencies, but maybe not all.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
Jane
Site Admin
Posts: 7751
Joined: 01 Nov 2002 15:00
Family Historian: V6.2
Location: Somerset, England
Contact:

Re: LUA: howto upload binary file

Post by Jane » 05 Nov 2019 16:50

You will probably need to use a POST request with the Form Data Type which I think should sort out the problem as it encapsulates the data in a more secure form.

This AutoIt script is similar, but I have not coded it for FH yet.

https://www.autoitscript.com/forum/topi ... request51/

It needed a content type added before the boundary to work I think.
e.g "Content-type: text/plain"
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."

Post Reply