* Import of RM7 Citations

Writing and using plugins for Version 5 and above.
User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 14 Apr 2021 19:11

Ok, that is great!
So that 0 _STMPLT structure forms part of the Input Details for the HIPO 1) Create Source Template task.
Part of the design process will involve mapping each RM GEDCOM tag to its equivalent FH Source Template tag.
e.g.
_STMPLT => _SRCT
TID => REFN (i.e. Custom Id)
NAME => NAME
DESC => DESC
CAT => CATG ?
FOOTNOTE => FOOT
and so on...
Notice that there is no RM equivalent to the FH TITL Title Format template nor a COLL Collection!
Then as the plugin reads each line of the GEDCOM file it will create a tag in the FH Source Template and assign a value.
Sometimes the value is a copy of the GEDCOM value, but for such as FOOTNOTE and BIBLIO the [brackets] must be {changed}.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 14 Apr 2021 19:42

I do not see any of this portion of the GEDCOM in the UDF file. Does FH7 just ignore?
Curtis Hermann

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 14 Apr 2021 21:27

Yes, as Mark confirmed earlier. I suspect it also says so in the Window > Log Files...

So the HIPO 1) task must read the RM GEDCOM file to obtain the details needed to create the FH Source Templates.
However, the subsequent tasks can all operate on the UDF that are imported into the Project.
That is crucial because those UDF identify the records that need modification by the plugin.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 15 Apr 2021 11:42

by tatewise » 14 Apr 2021 15:11
Sometimes the value is a copy of the GEDCOM value, but for such as FOOTNOTE and BIBLIO the [brackets] must be {changed}.

Does this mean the plugin would need to change the [ ] to { } or I would need to do this manually after the FH template was created?
Curtis Hermann

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 15 Apr 2021 13:37

The plugin can make those character changes.
There is an Lua function that handles that kind of global substitution, see string.gsub
e.g.
FHfootnote = RMfootnote : gsub( "%[", "{" ) : gsub( "]", "}" )

The first :gsub( "%[", "{" ) replaces every [ with {
It needs the % escape before the [ to make it a literal character, because [ is a 'magic character'.

The second :gsub( "]", "}" ) replaces every ] with }

So if RMfootnote was "[Jurisdiction], [Series] [Vol]" then FHfootnote would be "{Jurisdiction}, {Series} {Vol}"
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 15 Apr 2021 14:57

That’s great, now I have to figure out how I want to handle the issue that Citation-specific Fields are not allowed in the FH Bibliography Format. One option is to abandon Evidence Explained methodology. Another would be to add another field to Source Level.
Either one will require a rework of the templates.
Curtis Hermann

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 15 Apr 2021 15:21

If you have a rational EE argument for Citation-specific metafields in the Bibliography then ask CP whether they have considered that case and if there is a solution. Include the need to import them from RootsMagic. It seems illogical to me that they are valid in all the other formats but not the Bibliography. This topic continues in Bibliography Citation-specific Metafields (19297).

If no FH solution is forthcoming then the plugin may have to omit such fields from the Bibliography but produce a report of those cases for the user to consider.

Are such Bibliography fields a common occurrence?

I don't see how adding a Source record metafield will help, because every Citation has a different value for that field.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 16 Apr 2021 03:57

The plug-in needs to bring all fields over. I can’t think of any citation level meta fields that would not be needed for the Footnotes.
The issue in my case is my decision to follow EE methodology, see import of RM7 Citations (19266) and only involves the templates for citing digital images. This may not apply to other users of the plug-in.
My “digital image” templates include a citation level field: access date. The Bibliography needs just the year of the access dates. RM has the flexibility to include the access date field in the bibliography and use the :Year qualifier. FH has the same qualifier, but won’t allow a citation level field in the bibliography to be utilized. The RM flexibility reduces the number of fields that have to be entered and helps “automate” the publication year in the Bibliography entires. The issue only involves a portion of the templates, so the time required to revise them to match FH program restrictions is doable. Hopefully CP, will provide similar flexibility down the road.
Curtis Hermann

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 16 Apr 2021 09:50

There is no problem with Citation-specific fields in the Footnote or Short Footnote.

It is only the Title and Bibliography that disallow Citation-specific fields.
Hoping CP will do something is unlikely to work.
Either contact CP via http://www.calico-pie.com/osticket/open.php or post in the New Wish List Requests Forum.

Thank you for joining the Bibliography Citation-specific Metafields (19297) discussion.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 17 Apr 2021 00:10

Poor choice of words
Hoping as in "they will (or are able) to provide flexibility in response to ticket",
not Hoping as in "wishing something magic happens"
Curtis Hermann

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 18 Apr 2021 13:13

The issue with the importation of the Source record for EE style citations of online digital images led me to review my RM Master Source list to see if there were other instances of citation level metafields in the Bibliography formats. Short answer is yes. See my post in FH7 : Using Evidence Explained Style.
It appears the template to lump citations using the EE style model for layered citations of digital images is a unique situation that FH7 does not handle well, but the other instances are a result of lumping citations to fit the data entry task at the time and could easily be revised to fit within FH7's structure.
Given my limited knowledge of GEDCOM files and even more limited (AKA none) knowledge regarding plugins, is it better to modify the Source Records in RM or wait until imported to FH7? Have not looked at it in detail, but I believe if the RM Sources were modified, some could be merged and certainly the GEDCOM file would be more consistent.

I assume it is better to let the plugin create the FH template rather than create templates and have the plugin identify a template to paste to?
Curtis Hermann

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 18 Apr 2021 13:41

Not knowing how easy it is to modify the RM Master Sources to eliminate Citation-specific fields it is difficult to say.
However, I suspect there is probably little difference between RM and FH.
I have edited a Source Template and moved a Citation-specific field into the Source record and resynced.
It was quite straightforward but beforehand the Source records would need to be split so there is one for each Citation with different field values. Otherwise, where will all those Citation field values end up?
The RM techniques would probably be very similar. It is the creation of all those Source records that might not be easy and perhaps needs a Plugin in FH. In that case, FH may be the better bet unless RM has a similar Plugin feature or some automated way to create the Source records.

Yes, it is much better that the Plugin creates the Source Templates.
Trying to use manually create Source Templates would be a nightmare, because the Plugin would have to associate each metafield in the Source Template with the metafields in the RM GEDCOM file, which relies on the user creating fields with matching names & types and is therefore error-prone.

The Plugin Help & Advice needs to advise users they must give up Citation-specific fields in Bibliographies in one of two ways.
1) Use the Plugin and the Citation-specific fields will be removed from Bibliographies making them 'lumpier'.
2) The user must reorganise their RM Master Sources to have more fields as you are proposing before using the Plugin.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 18 Apr 2021 17:37

Sorry, I fooled myself with that experiment of moving the Citation-specific field into the Source record in a Template.
It had no Source Citations attached so it was allowed to go ahead.
I have repeated the experiment with Source Citations using the Source Template and it sensibly disallows the change.

So what is needed is a new similarly named Source record metafield in the Source Template which is OK.
That new metafield can be used in the Bibliography Format.

That change could even be partially carried out in RM which has benefits in the way the Plugin would work.
The import Plugin could recognise that there is a Citation-specific metafield being used in the RM Bibliography.
It could then ask which Source record metafield to use as a substitute.
Thereafter the whole process of importing Source Templates, and Source Citation metafields would also create duplicate Source records where needed and move the Citation-specific metafield values to the Source record substitute metafields.
The whole import and reformatting would happen in one run of the Plugin.

On second or is it third thoughts, the Plugin can perform all the corrections automatically with no manual intervention.
When it sees a Citation-specific metafield being used in the RM Bibliography it will create the FH Source Template with a Source record substitute metafield with the same name & type instead of the Citation-specific metafield.
When the Source records with the matching TID are processed they will gain the Source record substitute metafield.
When the Citations linked to those Source records are processed they will create Source record clones as necessary and move the metafield value from Citation to Source record. The end result will have multiple Source records holding the values that were originally in the Citations.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 18 Apr 2021 20:21

Potential bigger problem - I was going through each of my RM Source Records to get a feel of how many FH Source Records would be needed based on the lumping limitation of the FH file structure (as if I was starting from scratch in FH and forgetting about the whole digital image format issue for the moment)
I think, US Federal Census are going to present the biggest challenge. I chose to lump the US census population schedules at the census year level so I have one source for the 1870 US Census, one source for 1880 US Census, one source for 1890 US Census, etc. I have confirmed the RM GEDCOM lines for the federal census contains only one NAME field:
0 @SXXX@ SOUR
1_TMPLT
2 TID XXXXX
2 FIELD
3 NAME YYYY
3 VALUE 1880

The Jurisdiction is included in the Bibliography, requiring a Source record for each County in each State. A quick look through citations for the 1880 census identified 10 states and an lets say 2 counties per state. (Some had only one, but a couple had 5 counties), so if I was creating the Source record(s) from scratch in FH7, instead of 1 Source record for the 1880 US Census, I would have 20 Source Records, one for each County, State combination. If I understand correctly, if I don't do some work in RM, every time the plugin came across a US Census Source record, it would need to search through all the citations to find those linked to the SOUR and then created new FH Source records for each Jurisdiction it found - and if the Jurisdiction is used in multiple citations, keep track of those already created to avoid duplication.

To complicate things further, I do not use shared facts so if there are 6 family members, I will have a census fact for each person and depending on the census and what is recorded, the same citation could be used multiple times. I identified one citation that was used in 23 different people/fact combinations.

I am still trying to get my head around how this might work, if I was doing it manually, let alone get a plugin to do it.
.
Curtis Hermann

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 18 Apr 2021 22:29

cwhermann, just wanted to offer some encouragement. I've been "playing with" going from RM7 to FH7 for several months. I like FH7 so very much more. But cleaning up sources and citations manually is overwhelming, and I'm debating sticking to RM7 for that reason. If you were able to write this plugin, it would be wonderful. I am also a lumper, especially census records. Except I do use shared facts. In my newly created FH7 source templates, I've chosen to ignore the EE bibliography fields that are citation fields and just leave them out. I do try to follow EE, but decided it would not be the end of the world to do that. Anyway, thank you for trying. I'll follow along this thread.
Thanks,
Jackie

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 19 Apr 2021 10:17

Curtis, what you describe is exactly what I described in my last reply.
I realise it sounds daunting but it is perfectly feasible for a Plugin to sort all that out.

One thing to recognise is that many FH users would use Method 1 'splitter' Sources that results in one Source record for every household and year. I have dozens. So large quantities of Source records is not a problem for FH to handle.

20 Census Source records is tiny by comparison and it will only reach 20 if your family has lived in every State.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 19 Apr 2021 14:47

Remember that plugins don't get bored or distracted - if they need to check tens of thousands of citations to identify what’s used where, they will do it willingly and speedily! Your approach is the right one though - be absolutely clear in your mind what you want the plugin to do before getting into the weeds of how to do it.
Mark Draper

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 19 Apr 2021 16:54

Agree - the main thing I have learned through this transition, is that the lumping-splitting decision is driven as much, if not more, on the database file structure than it is on any desired methodology and that I will need to employ a much higher level of splitting going forward. I have been playing with the FH7 built in templates, attempting to determine what my Source/Citation structure and methodology would look like if I were starting from scratch in FH7. In fact one of the options I was considering - a plugin to delete all Source records and citations and then re-enter all the sources from scratch.

Assuming I don't want to "start over", as I see it, there are three general scenarios I could pursue for the plugin design :
  • Continue with the strategy we have been looking at -having the plugin determine the level of splitting via the citation meteafield-RM Bibliography comparison and proceed accordingly, I will end up with a methodology that is similar to what I was using in RM. Would still have the Bibliographies, but will need to revamp the Bibliography for each template.
  • Abandon the use of the Bibliography. Have the plugin just import the citation metafields, Footenote and Short Footnote. Much simpler plugin and I would be able to continue with the same lumping/citation methodology employed in RM. Right now, I can't envision publishing anything that would involve the use of a Bibliography so not sure loosing them would be a big loss.
  • Commit to extreme splitting and have the plugin move all citation specific metafields to the Source record. Major change in methodology and I would need to develop a Source record titling system, but would end up with the most FH7 / GEDCOM friendly project.

No matter which option, I have identified another distinction between the RM and FH the plugin will need to handle. In RM7 the repository and repository location fields in the citation templates are text style metafields (at either the citation or source level) and are independent of the Repository records (@RXX@ REPO) in the GEDCOM. So I assume the pligin would need to crate a Repository record for these in FH7.
by tatewise » 12 Apr 2021 05:35
So the top-level HIPO tasks now look like this:
1) GEDCOM RM Source Template _STMPLT ~ Synthesize FH Source Template ~ FH Source Template & TID cross-ref
2) RM Source RM _TMPLT.FIELD & TID cross-ref ~ Synthesize Source Metafields ~ Updated FH Source Records & RID cross-ref
3) RM Citation RM _TMPLT.FIELD & RID cross-ref ~ Synthesize Citation Metafields ~ Updated FH Citations
Clarification on 3) HIPO and/or impact of which option I head down. Will the "Updated FH Citation"s be new citations for each event or an update to the existing citation created during the original GEDCOM import? What is potential impact on media links?
Curtis Hermann

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 19 Apr 2021 17:15

Many of those decisions hinge on whether CP updates FH to allow Citation-specific metafields in Bibliography and Title Formats.
If they do, then 1st bullet and 2nd bullet become unnecessary options.

In your 1st bullet, the Plugin would create the Bibliography Formats so no manual revamping required.
The RM Bibliography would migrate directly to the FH Bibliography unchanged.
That works because the necessary RM Citation-specific metafields become FH Source record metafields with the same name.

Your 2nd bullet is a personal position, so the Plugin might be of limited use to others.
A factor in developing public published plugins involves considering what other users might need.

You might end up developing a Plugin that offers all three options!

There are options for Repositories. Try and match the metafield names to the Repository records. Automatically create new Repository records derived from the metafields. Leave the metafields as they are as plain text.

The HIPO 3) Citation updates would modify the existing imported entries whichever bullet option is chosen.
However, for your 3rd bullet the Citation Media should be able to be moved to the Source record along with the metafields.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 19 Apr 2021 17:21

I was re-reading this thread, and I’d like to suggest something that might make the process easier. There seems to be two distinct steps involved.

The first of these is converting all the raw data into FH7 templated sources. There are probably four steps to this:

1. Read the GEDCOM file for the core source definitions and source-level fields.
2. Read all the UDF fields in your project data to add the citation-level fields.
3. Create exactly analogous FH templated sources.
4. Convert all your project sources to these new templates.

That part of the process is exactly the same for all RM users, is relatively mechanical to do, and doesn’t need any design decisions. A plugin that does that for everyone without needing complex customisation is a good prize.

Once this is done, you then have all your data visible in FH, so only at that stage worry about the presentation of the data. Depending on the complexity of your sources, it may be appropriate to think about how the plugin might to tweaked to generate footnotes, bibliography entries, etc, or it may just be simpler to add these to the templates manually.

Whichever way you do it, the presentation aspect (and any conversion between lumping and splitting) is likely to be an individual requirement. Somebody else may want the data sliced and diced differently. You will probably want to be clear in your mind that stage 2 can be done, but don’t make things too complex by doing everything at once. Get the sources converted first, then fix the presentation.
Mark Draper

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 19 Apr 2021 18:28

Yes, that is very similar to the three HIPO tasks I posted a week ago:
i.e. Input data structure specification ~ Processing task to convert the input to output ~ Output data structure specification
1) GEDCOM RM Source Template _STMPLT ~ Synthesise FH Source Template ~ FH Source Template & TID cross-ref
2) RM Source RM _TMPLT.FIELD & TID cross-ref ~ Synthesise Source Metafields ~ Updated FH Source Records & RID cross-ref
3) RM Citation RM _TMPLT.FIELD & RID cross-ref ~ Synthesise Citation Metafields ~ Updated FH Citations
Those cross-ref details tabulate the Source Record Id (RID) against the Source Template (TID) and the Metafields.

It is only necessary to read the GEDCOM file for the _STMPLT codes in order to synthesis the FH Source Templates.
Everything else is already imported to FH and just needs the two task of updating Sources and then Citations.

I agree that converting to full splitter Source Citations would be better as a separate plugin.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 707
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 19 Apr 2021 19:15

Agree - there have been many discussions on lumped/split converters over the years, and they always come to the same conclusion- technically possible to do, but highly specific to individual setups and virtually impossible to generalise.
Mark Draper

User avatar
cwhermann
Diamond
Posts: 77
Joined: 20 Mar 2021 22:04
Family Historian: V7
Location: Amherst, New Hampshire, US

Re: Import of RM7 Citations

Post by cwhermann » 22 Apr 2021 11:25

I have decided the transition to FH7 is a much larger task/effort than I have the time to commit to at this point.
After way too many hours spent in frustration and failing to successfully create even simple plugins, (or simple queries for that matter), and coming to grips with the fact that this was just the tip of what was becoming a very large iceberg, I decided the limited hours I have to work on genealogy would be more fulfilling if spent on doing genealogical research.

The FHUG is an awesome resource and I would like to thank everyone who has provided guidance during the process and I look forward to the opportunity to tap into all the knowledge found here when I have more time to commit to a transition.

The ticket I submitted to CP regarding the use of citation level metafields in the creation of Bibliographies is still open. I have exchanged emails with tech support and provided them with some additional clarification and specific examples where this would be useful. I will continue to monitor the ticket and post their response.

I am grateful for all the guidance, perspective and encouragement.
Curt
Curtis Hermann

User avatar
BakerJL75
Famous
Posts: 118
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 22 Apr 2021 12:42

I came to a similar conclusion. I've decided to stick with FH, but use the citations as imported. Perhaps tweak a few as needed, but like you, I have too much research to do to try to convert them all to templated. I likely will use templated sources as I find new information. Eventually I may start converting a few at a time. Especially if there are any that imported particularly badly. Thanks for trying. I used to code, but it would take me a lot longer to learn Lua than I want to spend. I am getting better at query's.
Thanks,
Jackie

User avatar
tatewise
Megastar
Posts: 21534
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 22 Apr 2021 12:59

I would be tempted to offer to write the Plugin but do not have the paid-for version of RM7 that exports the GEDCOM template tags & UDF needed to test the Plugin. I am happy to offer snippets of Lua script and advice but without a comprehensive & representative GEDCOM testing is a challenge.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

Post Reply