* Import of RM7 Citations

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
User avatar
fhtess65
Megastar
Posts: 637
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: Import of RM7 Citations

Post by fhtess65 »

Mark1834 wrote: 26 Apr 2021 17:59 @Teresa, the finished product (hopefully by end of this week) will import all your custom RM templates directly, along with re-creating all the source data granularity that is lost on the normal GEDCOM import. Going forward, you can create new templates directly in FH. It does everything that RM does, and some more!
Thanks, Mark - that's great...I'll definitely have a boo when it's ready.

And yes, I have also created templates in FH. I love each program for various reasons, so maintain trees in each. Lots of work, but I also am all about not having all my eggs in one basket.

If I were just starting out, doubtless I would choose FH as my primary program.

Teresa
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com
Researching: Spong, Ferdinando, Taylor, Lawley, Sinkins, Montgomery; Basiński, Hilferding, Ratowski, Paszkiewicz
User avatar
fhtess65
Megastar
Posts: 637
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: Import of RM7 Citations

Post by fhtess65 »

tatewise wrote: 26 Apr 2021 19:15 So that offers little hope of assistance with FM imports of RM Templates, which if FH offered Direct Import from RM then CP would have to consider how to handle RM Citation metafields in the RM Bibliography. The only chink is they say they will investigate and check it out.
I'm a "bad" genealogist, and rarely use that Bibliography field as I don't cite by EE, more a hybrid of Strathclyde and EE, favouring Strathclyde. I'll be curious to see how this all works.
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com
Researching: Spong, Ferdinando, Taylor, Lawley, Sinkins, Montgomery; Basiński, Hilferding, Ratowski, Paszkiewicz
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Teresa, for most scenarios it will work well, especially if you make little use of the Bibliography.
However, the publically published plugins needed to work for as many scenarios as there are users.
So any limitations need to be made clear to users of the Bibliography that include Citation-specific Metafields.
Source record Metafields are perfectly OK.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
fhtess65
Megastar
Posts: 637
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: Import of RM7 Citations

Post by fhtess65 »

tatewise wrote: 28 Apr 2021 09:32 Teresa, for most scenarios it will work well, especially if you make little use of the Bibliography.
However, the publically published plugins needed to work for as many scenarios as there are users.
So any limitations need to be made clear to users of the Bibliography that include Citation-specific Metafields.
Source record Metafields are perfectly OK.
Yes, I understand that. It's always up to the user to decide if the plug-in offers the usability they require and to take appropriate cautions when using it. If I didn't work full-time, I'd be building way more templates in FH than I already do. Just not enough days in the week or hours in the day...plus, my husband does like to see me from time to time when I'm NOT doing genealogy :lol:
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com
Researching: Spong, Ferdinando, Taylor, Lawley, Sinkins, Montgomery; Basiński, Hilferding, Ratowski, Paszkiewicz
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

The plugin is now ready for more general user testing. I have developed it using a tiny database that I constructed in RM, plus the FH Sample Project exported to RM, then imported back again. Everything seems to work as intended, but how well does it cope with real RM projects?

To recap, all sources in RM are based on templates, either one of the 400+ supplied with the program, or those customised by the user. Unfortunately, when FH imports a GEDCOM file originating in RM, everything is converted to what FH7 calls generic sources. The custom template definitions are discarded completely, and while all the source data are imported, the structured version is hidden in Undefined Fields, and only mashups are visible.

This plugin does three things:

1. By reading the GEDCOM file that you used to transfer your database to FH, it extracts all the user-defined template definitions, and re-creates them as FH Source Templates. These templates should be virtually identical to the RM originals, as the basic structure of templated sources is very similar in the two apps. Footnote and bibliography definitions are copied across verbatim, with only minor translation (replacing [] with {}).

2. RM does not export its built-in template definitions, so we have to reconstruct them from the data included in the database. Every field used is included, along with an approximate label. No type definitions are included, so everything defaults to plain text (other RM types are date, place and name), and there are no data prompts. These should be reasonably clear, but you can always tidy them up in FH afterwards if you wish.

3. Finally, all sources are linked to their original templates, and the source data moved from Undefined Fields back into its correct place in the individual source fields. All other project data are left unchanged. This means that it doesn't matter if you have updated your project since your original import. Nothing entered by you is overwritten or deleted. The UDF fields are then deleted, but only after successful copying. Source level fields (yellow background in RM) are added back into their original sources, and citation level fields (green background in RM) are added to the appropriate source citation for the referenced fact. I have chosen not to update source titles, but that can be changed if necessary.

This plugin does not

A. Translate Bibliographies and Footnotes completely from RM to FH format. Hopefully the basic changes made create at least useable versions, but how good are they in the real world? I am not intending to create a "universal translator" (we'll leave that to Dr Who), but if there are any simple changes we can make that would be of use to the general user, I'm happy to add them.

B. Restructure the splitting/lumping of your data. This is intended as a general plugin, not tailored to specific users. If you use a bibliography layout that is not permitted by FH (e.g. including citation fields), you will have choose between modifying your data structure or changing the bibliography format. That is your choice, and the plugin doesn't make it for you.

Feedback welcome - what I'm aiming for is something that does most of the formatting for most of the users, leaving you to fine tune details manually. Realistically, it's a plugin that will be run once only, and only by a limited number of people.

As ever, it is best to try it out on a copy of your project, not the original. You can always select Edit > Undo Plugin Updates from the FH menu if you don't like the results, but better safe than sorry!

(Attachment deleted - replaced by version 2, below)
Last edited by Mark1834 on 29 Apr 2021 19:54, edited 1 time in total.
Mark Draper
User avatar
fhtess65
Megastar
Posts: 637
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: Import of RM7 Citations

Post by fhtess65 »

Thanks, Mark - I'll have a go this weekend and see what happens. Sounds very interesting and I'm curious to see the results :)

Teresa
Mark1834 wrote: 28 Apr 2021 22:15 The plugin is now ready for more general user testing. I have developed it using a tiny database that I constructed in RM, plus the FH Sample Project exported to RM, then imported back again. Everything seems to work as intended, but how well does it cope with real RM projects?

<SNIP>
Feedback welcome - what I'm aiming for is something that does most of the formatting for most of the users, leaving you to fine tune details manually. Realistically, it's a plugin that will be run once only, and only by a limited number of people.

As ever, it is best to try it out on a copy of your project, not the original. You can always select Edit > Undo Plugin Updates from the FH menu if you don't like the results, but better safe than sorry!
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com
Researching: Spong, Ferdinando, Taylor, Lawley, Sinkins, Montgomery; Basiński, Hilferding, Ratowski, Paszkiewicz
User avatar
BakerJL75
Famous
Posts: 201
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 »

Real world data problems. I exported my RM7 database and imported it into FH7. Then I ran the plugin and got:
Screenshot 2021-04-29 085708.png
Screenshot 2021-04-29 085708.png (42.64 KiB) Viewed 5214 times
I chose not to rollback the changes, but as far as I can tell, the plugin did nothing.
I'll see if I have any clue about a nil HINT field.
Thanks,
Jackie
User avatar
BakerJL75
Famous
Posts: 201
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 »

This may help. I use both brief and long hints a lot.
Screenshot 2021-04-29 090312.png
Screenshot 2021-04-29 090312.png (215.24 KiB) Viewed 5205 times
Thanks,
Jackie
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Sorry about that. I'll check it out later when I get back home. The message is telling me where to look.
Mark Draper
User avatar
BakerJL75
Famous
Posts: 201
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 »

Sorry about that. I'll check it out later when I get back home. The message is telling me where to look.
No apology needed. Bugs are part of coding. I'm just grateful you're doing this.
Thanks,
Jackie
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Mark, if this helps it is to do with appending HINT and LONGHINT when there is no HINT.
The fix is I think:

Code: Select all

		if Field.HINT then
			p = fhCreateItem('PROM', pF)
			fhSetValueAsText(p, Field.HINT)
		end

		if Field.LONGHINT then
			if fhGetTag(p) == 'PROM' then
				fhSetValueAsText(p, Field.HINT .. ' ' .. Field.LONGHINT)
			else
				p = fhCreateItem('PROM', pF)
				fhSetValueAsText(p, Field.LONGHINT)
			end
		end
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Sorted. If both HINT and LONGHINT are present, it concatenates them for the FH hint in the form "hint, longhint", which I think looks neater in typical examples than just a space. If only one is present, it uses that one as the FH hint, whichever way around it is.

Updated version attached.
Mark Draper
User avatar
BakerJL75
Famous
Posts: 201
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 »

Progress! I look some more later, but it looks like it brought in all my custom templates. I'll play with them a bit later. However, it didn't do well on the RM Templates. It brought in #139 and #172, but they were incorrect. The fields were brought in, but the templates had data. I'll post the error, and then what one of the templates look like. After those two it gave the error message below.
Screenshot 2021-04-29 155644.png
Screenshot 2021-04-29 155644.png (41.86 KiB) Viewed 5013 times
Screenshot 2021-04-29 155705.png
Screenshot 2021-04-29 155705.png (61.64 KiB) Viewed 5013 times
Thanks,
Jackie
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Thanks Jackie. There are two separate issues here. Firstly, the Bibliography and Footnote definitions for built-in templates are not included in the GEDCOM export, so I have used an associated source to create an example. On reflection, it might be better to omit them altogether. I anticipate that the typical RM user would have used only a small number of the hundreds of examples provided, so while they are copying the original template title across, they might as well copy the Bibliography and Footnotes as well if they want to keep them.

The error you reported comes at the final stage, as the UDF fields are copied back to the sources. I suspect it may be related to blank fields or missing data, so I probably need to build a bit more checking into the code. I'll look at it this evening, and hopefully third time lucky!
Mark Draper
User avatar
BakerJL75
Famous
Posts: 201
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 »

Thanks for looking. As far as me personally, I almost never use the standard templates, so I'm happy with it just bringing in the custom templates. But if you are going to publish it in the store, others may make much more use of the standard templates.
Thanks,
Jackie
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Latest update is attached.

I have not tried to recreate Bibliography and Footnote entries for built-in RM Templates. Any that are included in project Source Records are now left as Undefined Fields, so you can copy them into your new templates and modify as required if you want to keep them.

There is now more extensive checking for error conditions, which should help prevent unexpected crashes. It also detects if it is being run for a second time on the same project, and avoids creating duplicate templates.

There was also a mistake in one of the template selecting routines that meant templates were not always being selected correctly. RM template identifiers are all numbers (<10000 for built-in, >= 10000 for user-defined), but numeric data is generally stored in FH as text. This doesn't usually matter, but when checking equality, both sides must the same variable type. At one point in the script this was not the case, so templates were not being identified correctly, resulting in the error reported above. Hopefully, now fixed.
Last edited by Mark1834 on 01 May 2021 09:31, edited 1 time in total.
Mark Draper
avatar
Ruth
Gold
Posts: 23
Joined: 27 Jun 2018 07:57
Family Historian: V7

Re: Import of RM7 Citations

Post by Ruth »

Thanks Mark. I am a new user of FH7 coming over from RM. This will be a big help to me.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Welcome to FHUG Ruth. I assume from your profile that you looked at FH6 a while ago, but have now made the move. FH7 is a much better fit for RM users than FH6, and you might find the related thread on importing the RM To-Do List useful as well. Neither templated sources nor research notes are available in FH6.
Mark Draper
avatar
Ruth
Gold
Posts: 23
Joined: 27 Jun 2018 07:57
Family Historian: V7

Re: Import of RM7 Citations

Post by Ruth »

Thanks Mark. Yes I had previously reviewed vs 6, but found it a bit daunting. I decided to give vs 7 another look, and really liked what I saw. I am slowly learning the different features, and am really looking forward to using this software.
Ruth
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Ruth, could you please update your Profile to the right of your posting to say Family Historian: V7
Click your username, chose Edit profile, select FH V7, then scroll down to Submit button.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BakerJL75
Famous
Posts: 201
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 »

Tested .3
As before it did fine with all my custom templates, but none of the built in ones came in.
Screenshot 2021-05-01 132649.png
Screenshot 2021-05-01 132649.png (62.53 KiB) Viewed 4813 times
Thanks,
Jackie
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Sorry about this Jackie. I can see what has happened here, but I’m not clear why. The script is reading in your sources, but it can’t find an identifier for the template. The code looks ok, so my assumption about how RM structures its GEDCOM must be incorrect. I don’t have a real database to test on, but I’ll experiment tomorrow.

Which built-in sources do you use please, so I can try to replicate your sources as closely as possible?
Mark Draper
User avatar
BakerJL75
Famous
Posts: 201
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 »

Which built-in sources do you use please, so I can try to replicate your sources as closely as possible?
I don't think this will help much. I switched to entirely custom templates a long time ago, because I always want to change something later. The only built in source I've actually used is Newspaper, Online Archives. And only for source.
Thanks,
Jackie
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Import of RM7 Citations

Post by Mark1834 »

Thanks Jackie. The only way I can reproduce the error is by modifying the GEDCOM file before running the plugin to force an error condition, so can I ask for your help to diagnose what is happening please? I don't want to post multiple test copies of the plugin, but if you could add one additional line of code, it would tell us which source is causing the problem.

Starting from a fresh test database, could you open the plugin window, and instead of running the import plugin, click on More>> at the bottom of the window, and Edit... to open the plugin in the debugger.

Scroll down to the line where the problem occurs, 345, and add the following line of code between lines 344 and 345 (Lua doesn't worry about whitespace, so blank lines and indentations have no significance, they are just there to help readability).

Code: Select all

if not tonumber(TID) then fhMessageBox(fhGetDisplayText(pS) .. ' - ' .. fhGetDisplayText(pT) ) end 
It should then look something like this
Capture.PNG
Capture.PNG (11.87 KiB) Viewed 4733 times
Run the modified plugin by clicking on the Go button at the top of the window (you do not have to save it first).

The error is caused by the template identifier (TID) not being a pure number (either missing or containing non-numeric characters), so the tonumber() conversion returns a nil result. The new line of code will display the name of the errant source and the TID, as in this example where I created a new Newspaper Source and manually deleted the TID prior to running the plugin to force the error. When you click on ok, the script will resume and crash.
Capture1.PNG
Capture1.PNG (2.75 KiB) Viewed 4733 times
What message do you get please? Once we know what is causing the problem, we can move on to why and fix it.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28342
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise »

Just a couple of comments.

You could check if pT:IsNull() that would indicate there is no TID tag at all.

I noticed that there is a lot of management of table values to check for uniqueness or existence in another table.
Have you considered using the value as the table key instead of the table value?
i.e. Instead of table.insert(tblA,Value) use tblA[Value] = true
That ensures there is only one entry for each unique Value and avoids the need for function UniqueValues(tblA) and probably function InTable(val, tblA).
That is the conventional Lua technique for building a dictionary of unique table entries to avoid searching tables for values.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply