* New libraries available for FH6 -- please test!

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
Post Reply
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

There are two new Lua libraries available for FH6 (accessible via loadrequire) intended to make it easier for authors to write plugins compatible with FH6 and FH7.

One is compat53 which should be invoked via

Code: Select all

loadrequire("compat53")
and one is the utf8 string functions library supplied with FH7:

Code: Select all

utf8strings=loadrequire["utf8"]
Please try them out. When we're happy they're working, I'll update the knowledge base.
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: New libraries available for FH6 -- please test!

Post by Valkrider »

Can these new libraries be added to @Mike's plugin for the non Windows users please?
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

Colin, let's check that they work as expected first but I agree it ought to be done.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

Helen, have you had much success in getting those libraries working?

Is the intention that a Plugin meant to run in FH V6 and FH V7 would have nearly identical code?

To get loadrequire(...) working needs the Module Require With Load download.
That runs OK in FH V6 but fails in FH V7 due to unsupported loadstring(http.ResponseBody)

So I made the loadrequire conditional:

Code: Select all

if _VERSION ~= "Lua 5.3" then
	loadrequire("compat53")
	utf8strings=loadrequire("utf8")	-- changed from ["utf8"] that did not work?
end
That added a compat53 folder to the Plugins folder, containing init.lua, string.dll, table.dll, utf8.dll

My test code has tried using various functions such as getn, maxn, load, unpack, etc...
But I cannot find anything that works in both FH V6 and FH V7.
After reviewing the keplerproject What's implemented advice, I see that its purpose is rather different.
Of the features I was testing, only table.pack and table.unpack are meant to be supported.
I will have another look at the features it is designed to support, probably over the weekend.

What features does loadrequire("utf8") provide?
It did not appear to add anything to the Plugins folder.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

Mike compat53 isn't required in Lua 5.3 and utf8 is already included with FH7 so, yes, you would make the loadrequire statement conditional.

As the kepler documentation says:
This is a small module that aims to make it easier to write code in a Lua-5.3-style that is compatible with Lua 5.1, Lua 5.2, and Lua 5.3. This does not make Lua 5.2 (or even Lua 5.1) entirely compatible with Lua 5.3, but it brings the API closer to that of Lua 5.3.
So no, code can't be identical unless you avoid some Lua 5.3 features, but it does provide the utf8 functionality from Lua 5.3 among other things...

which is built upon by the utf8 library which
-- Provides UTF-8 aware string functions implemented in pure lua:
-- * utf8len(s)
-- * utf8sub(s, i, j)
-- * utf8reverse(s)
-- * utf8char(unicode)
-- * utf8unicode(s, i, j)
-- * utf8gensub(s, sub_len)
-- * utf8find(str, regex, init, plain)
-- * utf8match(str, regex, init)
-- * utf8gmatch(str, regex, all)
-- * utf8gsub(str, regex, repl, limit)
If your FH6 installation wholly separate from the FH7 one (which already includes utf8)?
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

I've double checked and loadrequire(utf8) downloads it if it isn't already present (i.e. for FH6)
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

I'm somewhat pretending to be a naive author to determine what guidance is needed for FH V6 & V7 interoperability.
I accept that the Lua code cannot be identical especially since there are FH API and GEDCOM differences.
So the guidance needs to identify which Lua 'features' can use the same code and which cannot.
That will supplement Writing Plugins Compatible with Versions 5, 6 & 7.

So one requirement is the loadrequire code snippet and loadrequire calls must be conditional.

I still have FH V6 installed separately from FH V7 beta on the same PC to make updating my Plugins more convenient.

Am I correct that loadrequire["utf8"] was a typo?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

Yes, re typo.

I will work on the necessary documentation when you've confirmed they download correctly and can be invoked.

P.s. Does that mean you expect to have all your plugins upgraded before he end of January?
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

I think all my plugins that were published in the Plugin Store are upgraded as of today, except those I don't intend to upgrade, and about 25 pages of Help for each Export Gedcom File target product that I hope to complete over the next week if nothing too traumatic gets in the way.

Then there are a significant number of unpublished Plugins that are mostly for handling imported GEDCOM from other products that probably should be published, but they are less urgent.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

After much head-scratching and trial & error, I think have got the "compat53" library working but not the "utf8" library in its current form ~ see later.

Below is the code that worked for me.
I had to put the loadrequire for "utf8" before "compat53" otherwise utf8.lua never loaded.
The extend_table(...) function is needed to add the "compat53" components to the global environment.
I plagiarized it from the compat53.module code. Did you need anything like that to get it working?

Code: Select all

if _VERSION ~= "Lua 5.3" then		-- loadrequire for "utf8" and "compat53" libraries

	-- loadrequire code snippet goes here

	-- handle exporting to global scope
	local function extend_table(from, to)
		if from ~= to then
			for k,v in pairs(from) do
				if type(v) == "table" and
					type(to[k]) == "table" and
					v ~= to[k] then
					extend_table(v, to[k])
				else
					to[k] = v
				end
			end
		end
	end

	utf8lib = loadrequire("utf8")	-- Must be before compat53 otherwise does not load due to utf8 conflict?

	loadrequire("compat53")

	local M = require("compat53")

   extend_table( M, _G )		-- Export local M to global _G

end -- if _VERSION ~= "Lua 5.3"
Then the features mentioned in https://github.com/keplerproject/lua-compat-5.3 under What's implemented worked and the Lua 5.3 functions appeared in the utf8, string, math & table libraries.

I tested a variety of Lua 5.3 functions as listed below and they all worked similarly in FH V6 and FH V7.
Is there anything else you particularly want tested?
utf8.charpattern
utf8.len(...)
string.pack(...)
string.unpack(...)
math.mininteger
math.maxinteger
table.unpack(...)
load(...)

However, I cannot get the "utf8" library to work, not even in FH V7.
The only user guide I could find was C:\Program Files (x86)\Family Historian 7 beta\Program\Lua\utf8\README.md
But the utf8 table only includes the Lua 5.3 functions and none of the gmatch, gsub, etc, functions.
So none of the utf8 examples in README.md work and simply say:
An error has occurred - plugin failed to complete
[string "C:\Users\Mike\AppData\Local\Temp\~fhC943.tmp"]:15: attempt to call a nil value (field 'gmatch')

After a lot of digging around and more trial & error, I have found two problems with the utf8.lua downloaded script.

Firstly, in FH V7 Lua 5.3 the unpack function is undefined and needs to be table.unpack
However, for FH V6 Lua 5.1 it needs to remain as unpack
So it needs if _VERSION == "Lua 5.3" then unpack = table.unpack end near line 80.

Secondly, naming the module utf8.lua and using a local utf8 table conflicts with the Lua utf8 table.
So the module should be named utf-8.lua and some other table such as local utfx used.

local utfx = require("utf-8") then works correctly and functions like utfx.gmatch and utfx.gsub are OK.

Attached is my test script including iup.GetParam button demonstration and notes at end about Plugin Store Help page access.
[ EDIT: Attachment deleted as now covered in KB. ]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

1. You're not debugging Calico Pie software. In the case of compat53, you're debugging work I've done which CP has kindly uploaded for use with FH6.

And with compat53 I've made a pretty obvious packaging error (sorry) -- if you delete compat53.lua from C:\ProgramData\Calico Pie\Family Historian\Plugins and replace it with init.lua (renamed to compat53.lua) from C:\ProgramData\Calico Pie\Family Historian\Plugins\compat53 it should work as simple

Code: Select all

loadrequire("compat53")
If you confirm I'll ask CP to modify the packaging.

2. You obviously missed the bit where I said:

Code: Select all

utf8strings=loadrequire("utf8")
i.e. the add-on utf8 library has to be loaded with a different name to the built in utf8 library.

I didn't package this one, and will need to work out which of the forks of the library CP has implemented -- there is one that is supposed to be 5.1 and 5.3 compatible -- I will investigate.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

Sorry, but that does not work.
loadrequire("compat53") now raises an error message saying the compat53.module does not exist.
That is because line 11 of compat53.lua says local M = require("compat53.module")

The problem with utf8strings=loadrequire("utf8") is not the utf8strings variable but the name "utf8".
That conflicts with the Lua utf8 library module once compat53 is working correctly.
Plus when utf8strings=loadrequire("utf8") does work the utf8strings variable is just true and no table of functions exists. To get that working needs such as local utfx = require("utf-8") where utfx is a table of functions.
That applies to FH V7 as well as FH V6.
Unless you can explain in more detail how you get the "utf8" module to work in both FH V7 and V6.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

So, I have downloaded the module.lua file from the keplerproject web site and saved that in the compat53 folder.
Now loadrequire("compat53") works correctly.
Thus the package needs two changes: add module.lua in place of init.lua and rename init.lua to compat53.lua in the Plugins folder.

If in FH V6 I now execute:
loadrequire("compat53")
utf8strings=loadrequire("utf8")

compat53 works OK but utf8 does nothing ~ does not request to download ~ zilch.

Swap them round to:
utf8strings=loadrequire("utf8")
loadrequire("compat53")

Now utf8 requests to be loaded, utf8strings is true but no UTF8 functions work.
utf8strings=require("utf8") simply returns the Lua utf8 standard table of functions created by compat53.

Likewise, in FH V7 utf8strings=require("utf8") simply returns the Lua utf8 standard table of functions.

I am convinced that the "utf8" module name is in conflict with the Lua utf8 library.
If I change the downloaded utf8.lua file to utf-8.lua and use require("utf-8") everything is rosy.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

Using your test code in FH6 (but with

Code: Select all

require ("compat53")
instead of loadrequire, because loadrequire won't work properly until the repackaging is done:
Screenshot 2021-01-13 15.54.26.png
Screenshot 2021-01-13 15.54.26.png (55.71 KiB) Viewed 4542 times
My compat53 directory looks like this:
Screenshot 2021-01-13 160025.png
Screenshot 2021-01-13 160025.png (60.71 KiB) Viewed 4542 times
Re utf8, the problem is NOT the module name it's the packaging! Please leave it to me.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

Yes, as I said in my last reply, the module.lua file was missing from your package, but I fixed that and compat53 works fine.

Can you please explain why require("utf8") does not work in FH V7?

It seems that is where it should work and is not confused in any way by compat53 packaging.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

UTF8 files are missing -- see https://github.com/Stepets/utf8.lua (also read the instructions).
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

Sorry, I must be really dim. What UTF8 files are missing? Read what instructions?

All I could find was C:\Program Files (x86)\Family Historian 7 beta\Program\Lua\utf8\README.md and followed its advice.

And don't forget the unpack problem!
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

ColeValleyGirl wrote: 13 Jan 2021 16:12 UTF8 files are missing -- see https://github.com/Stepets/utf8.lua (also read the instructions).
CP have only distributed one file -- utf8.lua
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

Yes, and it is broken!

The instructions in C:\Program Files (x86)\Family Historian 7 beta\Program\Lua\utf8\README.md don't work and the use of unpack cause the script to abort in FH V7.

But if I manually fix the unpack and rename the file to utf-8.lua everything works fine.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

Mike,

Have you looked at that Github page I have directed you to twice!!!

https://github.com/Stepets/utf8.lua

utf8 as distributed by CP is broken because 95% of it is missing.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: New libraries available for FH6 -- please test!

Post by tatewise »

Helen, today is the first time that link has been mentioned. Anyway, it is so much tecno speak it makes no sense.
If utf8.lua is broken, why ask me to test it?
I am not wasting any more time on this.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

Mike,

I know today is the first time that link has been mentioned (repeatedly), because today is the first time I have understood there is an issue with utf8 -- I had assumed CP had packaged it correctly and just replicated that for FH6 as the documentation said it should work with 5.1 and 5.3 ! I do apologise for not being clairvoyant. However, I did say (in an effort to avoid you wasting effort, because you're clearly too busy to read what I'm posting).
Re utf8, the problem is NOT the module name it's the packaging! Please leave it to me.
On the subject of technospeak, we are talking about a technical subject; it's quite hard to avoid the correct technical terminology, and it's a model of clarity compared to some documentation, if you're experienced with lua libraries.

I will continue to investigate exactly how utf8 should be packaged in the absence of a rockspec.
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: New libraries available for FH6 -- please test!

Post by ColeValleyGirl »

compat53 is now properly packaged and available for loadrequire in FH6.

I have tested the utf8 library at https://github.com/Stepets/utf8.lua and it works for both Lua 5.1 and 5.3 (adding functions into the pre-existing utf 8 library. (In Lua 5.1, compat53 is a prerequisite).

I have asked if this library can be made available for FH6 and FH7.
Post Reply