* Import of RM7 Citations

Writing and using plugins for Version 5 and above.
avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 24 Apr 2021 09:32

I've had a look at this to see how easy it would be to modify my existing To-Do list importer to read in Template Source definitions instead. The basic logic is very similar, so it was a modification rather than a complete re-write.

I've tested it using an on-line resource posted by an RM user group comprising a database of all of the hundreds of built-in templates in RM7. Most of the fields map very well to FH equivalents, but I'm struggling to make the newly created FH templates useable. See the attached screen grab, where none of the template fields appear in the Citation Window. The window is configured correctly, as it does show all the fields for a built-in FH template (second grab).

Any ideas? If we can get this fully working, it would be worth posting the prototype for user feedback. If it looks potentially useful, I can take it to the next step of capturing all the data hidden in FH UDF fields.
Capture.PNG
Imported template
Capture.PNG (110.64 KiB) Viewed 1121 times
Capture1.PNG
FH template
Capture1.PNG (95.1 KiB) Viewed 1121 times
Mark Draper

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 24 Apr 2021 10:07

Is it that there is no Record Title Format defined, which was a point raised earlier as missing from the RM TMPLT.
I suspect creating a 1 TITL field identical to the 1 SHRT field will fix it.

BTW: All the Format [square brackets] need replacing with {curly brackets} in FH.
Are Qualifiers such as :Reverse and :Abbrev valid in FH?

Also, remember that any Citation-specific Metafield that appears in the Bibliography Format (or the Record Title Format) must become a Source Record Metafield.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 24 Apr 2021 10:19

That was my first thought, but the plugin help says specifically that the title field is optional. I tried adding the title and it makes no difference.
Capture.PNG
Capture.PNG (22.38 KiB) Viewed 1116 times
I'm not intending to get into detailed formatting changes for specific users, but I will make the global change in bracket style, as that applies to everybody.
Mark Draper

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 24 Apr 2021 10:27

If you expand the Field Definition entries by clicking their [+] icon is there any UDF or do they look OK?

Can you edit them via Tools > Source Template Definitions...?

When you do so can you see the Field Definitions?

Do I recall correctly that you have written plugins before that create Source Templates?
If so, then can you compare the techniques that were successful then with those being used now.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 24 Apr 2021 10:54

I’ll experiment further this evening. If I can’t get it to work, I’ll start a new thread on manipulating source templates in code, as it’s a more general issue than just RM importing.
Mark Draper

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 24 Apr 2021 12:48

I think I have sorted it. I created a simple template via the FH interface, then an apparently identical one in code. They looked exactly the same in the Records Window, but I still had this problem with the code version. Comparing the project GEDCOM file, the only difference between them was that the CODE tag was missing from the data fields.

It appears that the CODE is created automatically from the field name in the user interface, but not when the template is created in code. If I set the field CODE explicitly (to the same as the NAME, replacing any spaces with underscores), it seems to work as expected.

I'll do a final proof read of the prototype before I post it later today.
Mark Draper

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 24 Apr 2021 12:59

Well spotted. You need to replace all non-alphanumeric characters with an underscore.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 24 Apr 2021 22:39

First prototype is attached. All this first version does is read the original RM7 GEDCOM export file, and import the user-defined source templates that the normal FH import discards, converting them to new FH Source Template Records.

The basic design of templates is remarkably similar in the two apps (although the detailed syntax is different), so the imported versions are virtually identical to the RM originals. The Footnote, Short Footnote and Bibliography definitions are imported unchanged, apart from converting the RM [] style to the FH {} style. No attempt has been made to translate the detailed syntax or weed out fields that will not work in FH, such as citation fields in the bibliography. This is intended as a general plugin, not tailored to any individual requirements, so it should eventually do most of the converting for most of the users, leaving just fine tuning to be completed in the normal FH user interface as required.

Having the original user-defined templates available means you can create new sources based on those templates straight away.

The next stage will be to repeat this for the standard templates, but it will not be possible to re-create these exactly. RM does not export standard templates, just the ID of the template used. RM understands what the ID means, and provides the corresponding detail, but we are missing that information, so can only reconstruct on the basis of what is imported to FH. All of the data are there, but details such as prompt text or exact field label will be missing. Once that is done, all the source structure that is currently hidden in UDF fields can be re-created by mapping your existing imported sources back into these templates.

That will take a bit of time to finish (particularly with the good UK weather and the easing of lockdown limiting time at the desk :D), but have a play with this first prototype in the meantime. Does it look like I'm on the right track? As with all draft plugins, it is best to try it out on a copy of your project, not the original. It doesn't touch any of your existing data, but it does add new Source Templates.

Plugin coding is deliberately a little on the verbose side, in order to make it easier to follow and potentially adapt in the future. That is the priority here, rather than technical elegance or fastest run time, although as ever, highlighting potential errors is welcome!
Attachments
Import RootsMagic 7 Templated Sources - Prototype 1.0.fh_lua
(5.53 KiB) Downloaded 17 times
Mark Draper

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 25 Apr 2021 14:47

That is an excellent start.
Regarding RM standard templates, are they identical for every installation of RM7?
If so, can the FH equivalent Source Templates be constructed by the Plugin from a preset table of tag values?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 25 Apr 2021 15:43

I'd thought of that route, but RM7 has been around for years so we would have to assume that any new templates added in minor updates use incremental numbering, rather than renumbering existing ones.

If we made that assumption (and I think it is a reasonable one, otherwise GEDCOMs wouldn't import back into different minor version updates correctly) there are still a couple of issues. Firstly, there are over 400 standard templates, so even if we used the same resource as I used for testing, this compilation does not include the reference IDs. It would be a huge amount of work to add these in ourselves. Secondly, I would have reservations about us bulk downloading part of the RM program and engineering it for FH use. I suspect that the templates are covered by RM copyright, but even if not, it feels a little unethical. Put ourselves in CP's shoes if a rival app copied all their FH7 templates in order to entice users into a rival camp. I don't think they would be very happy.

On balance, I'm inclined to stick with what's already in the GEDCOM file - individual RM users probably only use a small subset of templates, so can fill in the missing details by copy/paste from RM7 if they want to.
Mark Draper

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 25 Apr 2021 15:57

I had no idea there would be so many standard templates! I agree we cannot use RM code to synthesise them.
Just so I understand the mechanism...
RM Source records, after importing to FH, have a UDF TID number that identifies their associated Source Template.
Are you saying sometimes that TID number refers to a standard RM Source Template and there is no _STMPLT entry in the GEDCOM file with a matching TID number?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 25 Apr 2021 16:08

I'm going to try it this afternoon on a test database. I use almost all custom templates, so I won't miss the standard ones. I do understand your copyright concerns, but thought I would mention that SqlLiteTools https://sqlitetoolsforrootsmagic.com/so ... it-import/ will copy the built-in to custom. I've used it for that. But that does nothing for the copyright/license issue.
Thanks,
Jackie

User avatar
ColeValleyGirl
Megastar
Posts: 2774
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Import of RM7 Citations

Post by ColeValleyGirl » 25 Apr 2021 16:19

I was just about to post the same link as Jackie...

I guess getting the sqllite ibrary working might help, but I've tried a number of ways to sort it and failed so far,

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 25 Apr 2021 17:10

Correct - user-defined templates have TID values of 10000 and above, and are fully described in the GEDCOM. We can therefore import them perfectly. FH and RM use different descriptors and GEDCOM tags, but exactly the same concepts, meaning everything in the RM definitions has an FH analogue.

Built-in templates have lower TID values, and no _STMPLT tag. RM knows exactly what "2 TID 49" means, for example, and fills in the details, but only NAME/VALUE pairs are exported in the GEDCOM for each FIELD tag, at the appropriate level for source and citation data.

To reconstruct the templates (minus details such as footer/bibliography descriptions, data type, exact labels, etc), I loop through the FH project data and build up sets of fields for each used template and linked sources. It's not difficult, but it is a lot of deeply nested loops in places - I'm getting lots of practice in pointer control and managing tables!

I'd found that link quoted, thanks, and used that dataset of all the templates for testing. Unfortunately, the editable versions don't have the original TID values, so would need editing individually.

I'm happy that this approach will reconstruct all the granularity in the original data, even if some of the cosmetics are missing. It wouldn't take long for the user to tweak the templates in FH, copying descriptors across from RM as necessary.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 25 Apr 2021 17:47

I was going to try this, but I have about 100 custom citations in RM. If I run this will they all get pulled into my Source Definitions, even in a test file? Since I think FH stores them all in the project.
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 25 Apr 2021 19:45

Yes, it won't change any of your data, but it will add source template records. What you could do is start a new dummy project and import them into that. This first prototype doesn’t link the templates to your data, so they can happily sit there in isolation. Do they import ok, and can you create new sources from them?
Mark Draper

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 25 Apr 2021 22:59

I’ll try it and let you know. Thx
Thanks,
Jackie

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 26 Apr 2021 13:04

Sorry to take so long to get back to you. Busy weekend. But I tried it this morning, it created templates and I was able to use them.
Thanks for looking into this.
Thanks,
Jackie

User avatar
fhtess65
Diamond
Posts: 85
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: Import of RM7 Citations

Post by fhtess65 » 26 Apr 2021 16:04

If we do this test with a test database and have the templates created, can they then be added to our current permanent one?

Very intrigued by this as I create templates in RM all the time and have a lot that I could use in FH.

Teresa
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 26 Apr 2021 16:32

I wanted to provide the feedback/response from CP tech support on the ticket I submitted last week.
The ticked raised two issues.
The information in quotes under item 1 is taken directly from CP’s response to me.
The information under item 2 is my summary of communication back and forth with CP and my conclusions.

1) Issues with citations from the direct import via GEDCOM.
Thank you also for drawing our attention to the issue with regard citations in the RootsMagic import via GEDCOM. We have logged that too and will investigate. If the problem is just how RootsMagic creates GEDCOM files, we can't replace information that isn't there. However, we will check it out in any case. Another possibility would be for us (or some third party using a Family Historian plugin) to write an improved direct import from RootsMagic. The existing direct import is not up-to-date
.
2) Requested the ability to use citation level fields in the Bibliography entries to facilitate EE style citation templates.
We compared the FH7 Advances templates where all fields are in Source record - (extreme splitter) with RM7 standard templates with some fields entered at both Source and citation level - (some degree of lumping) but citation level fields used in both the Footnote and Bibliography entries. Also ran a trial to produce a report that included Footnotes, Short Footnotes and Bibliography for two different sources in both FH7 and RM7.
Conclusions: Despite the difference in the RM and FH Evidence Explained templates, to get the Footnote, Short Footnote and Bibliography entries correct, the user must match Source fields to a desired and/or required Bibliography format or format the Bibliography to match the desired Source Fields determined by the degree of lumping needed to match work flow. The EE style formats using layered citations for digital images are only compatible with RM7 and FH7 database programming/management if the user is an extreme splitter. If a user is going to use EE style formats, the degree of lumping for any given record is constrained by the EE style Bibliography entry. That is, any field needed in the Bibliography must be entered at the Source level and any level of lumping will require modifying the EE style Bibliography.

Curt
Curtis Hermann

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 26 Apr 2021 17:53

Second part seems to show that CP came to the same conclusion as our original thread discussion here. EE is not consistent with the normal definition of bibliography, so it would not be appropriate to modify FH to permit this inconsistency.
Mark Draper

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 26 Apr 2021 17:59

@Teresa, the finished product (hopefully by end of this week) will import all your custom RM templates directly, along with re-creating all the source data granularity that is lost on the normal GEDCOM import. Going forward, you can create new templates directly in FH. It does everything that RM does, and some more!
Mark Draper

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 26 Apr 2021 19:15

So that offers little hope of assistance with FM imports of RM Templates, which if FH offered Direct Import from RM then CP would have to consider how to handle RM Citation metafields in the RM Bibliography. The only chink is they say they will investigate and check it out.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 27 Apr 2021 02:27

There were a couple of other threads dealing with EE and the Advanced collections around this same issue. Didn’t know if the info should be posted there as well. Would be happy to supply my RM GEDCOM file for testing, if it would help.
Curtis Hermann

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 27 Apr 2021 19:36

Thanks Curtis. It would be ideal if RM had a Sample Project, but despite requests in their user group, it has never come to pass. I’ll use my tiny test database for now, and users will be able to test the finished product against their RM original once it’s finished.
Mark Draper

Post Reply