* Import of RM7 Citations

Writing and using plugins for Version 5 and above.
avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 02 May 2021 10:46

Yes, that makes sense on tables, thanks. I’ll tidy it up in the next version (and then iterate with pairs rather than ipairs - I’ve used that technique before, so not sure why I didn’t apply it here!).

I didn’t build in a check for blank or non-numeric TID values as I assumed (probably incorrectly) that could not happen. Let’s see what Jackie reports back, but I might have to add that so the plugin at least fails gracefully and identifies the errant source rather than crashing.

As an aside if we do go down that route - is it possible/easy to force an undo of plugin changes so far, so we don’t end up with a half-converted project?
Mark Draper

User avatar
tatewise
Megastar
Posts: 21551
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 02 May 2021 13:05

No, I don't think it is possible to force a Plugin to Undo Updates.
The only options are:
1) Raise an error(...) which gives the user the choice to Undo Updates or not.
2) Identify the problem to the user and continue to convert as much as possible, but perhaps advise that Edit > Undo Plugin Updates is a good idea.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 02 May 2021 13:19

Thanks - #2 is what I do if it finds a problem when converting UDF fields, so I’ll carry on with that approach.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 02 May 2021 14:14

What message do you get please? Once we know what is causing the problem, we can move on to why and fix it.
Hope this helps. Keep reading after screenshot.
screenshot 2.png
screenshot 2.png (30.4 KiB) Viewed 409 times
The editor also shows this:
screenshot 3.png
screenshot 3.png (53.07 KiB) Viewed 409 times
Don't know if it helps, but I poked around in my RM database, and I think the only two built-in source templates I'm using are Book image copy (online) and Newspaper, Online Archives.
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 02 May 2021 14:31

Thanks Jackie - it all helps build up a picture. I'm going to modify the plugin so it makes fewer assumptions about what I think the RM database should look like, and as far as possible identify problem records and data fields rather than just crashing.

Another option might be for you to do a very selective GEDCOM export from your RM database, selecting just a few (non-living) individuals who have facts supported by these built in sources. Just select "Sources" and "Extra details (RM Specific)" from the export options to give a minimalist file but one that should contain enough information to help diagnose the problem. Would that be possible please? If you are prepared to share that but don't want to post it here, send me a PM and I'll give you my e-mail address so you can send it directly.

Thanks.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 02 May 2021 14:48

do a very selective GEDCOM export from your RM database
Done and attached. These people are long gone. I did replace my personal info in the header with fake info. I ran it with the plugin and it does the same thing. Hope it helps.

newtest.ged
(41.88 KiB) Downloaded 7 times
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 02 May 2021 14:50

Great, thanks. It will probably be tomorrow before I get back to you, as I want to make the plugin more resilient before I post another update.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 02 May 2021 14:55

Actually, it's not going to help. I thought I had the ran the plugin against the new gedcom, but I didn't. I did it again, with the gedcom I sent, and it runs the whole way through. I should have some time this afternoon to make a Gedcom with no living and see if that one errors out.
Thanks,
Jackie

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 02 May 2021 23:37

do a very selective GEDCOM export from your RM database
I narrowed it down to one person who will generate the error. I played around a bit to see if I could isolate the cause, Looked at all his sources in RM, ran all the database tools, etc. So here is his gedcom. Hope it helps.
RR 6 pm.ged
(95.43 KiB) Downloaded 12 times
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 03 May 2021 09:45

Found it! I'll blame an "unexpected characteristic" in the RM export - I won't call it a bug, there may be good reason to do it that way ;). Templated data are stored as a NAME/VALUE pair for each data field, both in RM and FH. Sometimes, empty fields are exported just as a NAME with no associated value. I'm note sure why, as it does not appear to be consistent.

If that field was also missing from the template definition, we ended up with a UDF data field that the plugin didn't know how to process, and crashed. It's not clear to me how RM decides what to export. My test dataset from the web of 400+ sources appeared to come across ok, even though most of them were unused in my RM file. Certainly, in your test file there was a field that was included in individual's data when it shouldn't have been, and missing from the template definitions when it should have been included!

Yesterday's plugin update managed to process the GEDCOM without problems, as it does much more checking, and discards just bare NAMES with no VALUE before reassigning the data to the correct place. I'm optimistic that will work ok on the full dataset. I need to do just a little more checking and tidying up, but I'll post it later today.

Thanks for sticking with me - it would have been much harder to find the problem without being able to focus on your small test dataset.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 03 May 2021 11:30

Thanks for sticking with me -
I'm very grateful you are doing this. It will save me (and many others) so much time.
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 04 May 2021 09:59

I've put so much error checking in the plugin that I think it is finding an inconsistency in the GEDCOM file that Jackie posted.

Lines 2410-2415 of the GEDCOM file define the field CreditLine as a Source Level field (ISDETAIL = N) for User-Defined Template 10032 (TID tag on line 2346).

Line 1197 (2 TID 10032) records that Source Record @S373@ is based on this Template.

However, when this source is cited for the birth of Raymond Russel McDonald, the CreditLine is stored at Citation level, not Source level (lines 401-404).

FH imports faithfully what is in the GEDCOM. The plugin will flag this as a problem when moving UDF data back into their proper place, and leave this field untouched. UDF fields are only deleted after moving is confirmed as successful.

Can I have a second opinion please? Is it a GEDCOM error? I've no way of knowing if this is a one-off possibly caused by a very selected GEDCOM export, or whether such errors will be common. We'll find out soon enough!
Mark Draper

User avatar
tatewise
Megastar
Posts: 21551
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 04 May 2021 10:34

Your analysis looks correct.
Did you notice that Source record @S373@ also has a CreditLine field but with no value.
So that field appears at both the Source and Citation level in association with TID 10032.

Maybe Jackie could review template 10032 !Ancestry Basic Images Template
How does it define the CreditLine field?

Also perhaps she could review that Source record and the Citation.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 04 May 2021 12:28

It's Source level in the standard template that Jackie presumably based hers on...
Attachments
Capture.PNG
Capture.PNG (31.86 KiB) Viewed 291 times
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 12:37

Getting my morning started but saw this. I'll take a look in about an hour and post.
Thanks,
Jackie

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 14:15

Wanted to give you what I have so far, but I'm continuing to look at a copy of my RM database. I've played around with SQL Lite and am going to look at the tables and see if I see anything odd. I've also tried greatly simplifying that template and so far haven't found anything that helped. I also tried putting something in the CreditLine and that didn't help.

Meanwhile, this is the template:
Template.png
Template.png (67.27 KiB) Viewed 282 times
And this is how it looks in the citation:
Citation.png
Citation.png (79.83 KiB) Viewed 282 times
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 04 May 2021 14:40

Curious - the GEDCOM says the field is occupied, but the original is blank. Is the RM export corrupting records as well...? :?

4 FIELD
5 NAME CreditLine
5 VALUE citing Pennsylvania Historical and Museum Commission, Harrisburg, Pennsylvania, Pennsylvania County Marriages, 1852-1973; County: Fulton; Year Range: 1947 - 1956; Roll Number: 549832

Meanwhile, have a try with the latest plugin update. Does it complete without crashing? Are there any data errors reported?

(Attachment deleted).
Last edited by Mark1834 on 04 May 2021 16:49, edited 1 time in total.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 15:52

It still crashes, with the error below. I may have noticed something else though. Let me make sure it has something to do with this. It involves 2 sources made from the !Ancestry Basic Images Template, name very similarly, and one source has the credit line filled, the other does not. It shouldn't matter but I think possibly RM is confusing the two on export. But I need to look at the full Gedcom some more. Meanwhile here is the error:
Error.png
Error.png (37.08 KiB) Viewed 261 times
Thanks,
Jackie

User avatar
tatewise
Megastar
Posts: 21551
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import of RM7 Citations

Post by tatewise » 04 May 2021 15:55

Jackie, that is not a crash. It is one of Mark's new diagnostic messages.
Crashes say Plugin Error with a red cross. These messages say Family Historian and have a warning triangle.

Mark, the blank CreditLine in the upper yellow Source record pane exactly matches the GEDCOM for @S373@ that also has a blank CreditLine entry:
2 FIELD
3 NAME CreditLine

It is the lower green Source Details pane that omits the CreditLine Citation for the birth of Raymond Russell McDonald included in the exported GEDCOM:
4 FIELD
5 NAME CreditLine
5 VALUE citing Pennsylvania Historical and Museum Commission, Harrisburg, Pennsylvania, Pennsylvania County Marriages, 1852-1973; County: Fulton; Year Range: 1947 - 1956; Roll Number: 549832

So the GEDCOM is including data not shown in the RM7 GUI. I wonder if that is something left over from some earlier variant of the Source Template. It is odd that the Source record [CreditLine] definition sits amongst Citation definitions, as if it was itself once a Citation definition.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 04 May 2021 15:58

That should be just flagging the inconsistency discussed above. What happens when you click on OK to dismiss the message? With your one person GEDCOM, it goes through to completion and displays the following message:
Capture.PNG
Capture.PNG (7.13 KiB) Viewed 255 times
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 16:23

After I captured the message I totally forgot to keep running it. The good news is it finished with the Gedcom with only Raymond. However, when I ran it on the full gedcom I got the following:
1.png
1.png (67.6 KiB) Viewed 251 times
2.png
2.png (66.87 KiB) Viewed 251 times
3.png
3.png (27.82 KiB) Viewed 251 times
See next reply for 4th attachment
Thanks,
Jackie

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 16:24

4th attachment
4.png
4.png (41.26 KiB) Viewed 251 times
Thanks,
Jackie

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 16:28

I seem to be quite forgetful today. I also got messages that it didn't like the NewspaperPlace and PubPlace for
Template Record: 6
Source Record 328
Thanks,
Jackie

avatar
Mark1834
Megastar
Posts: 708
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire

Re: Import of RM7 Citations

Post by Mark1834 » 04 May 2021 17:20

OK, good progress, thanks. If RM is going to throw these wobbles in its GEDCOM, it might be better to save a list of suspect fields as a table, and present it in the Records Window when the plugin has completed. It's not a difficult change. I'll see if I can add more details, such as the fact that the citation is against, to make it easier to manually check the GEDCOM file.

The Meta-field failure means that the plugin could not create a shortcut for adding ItemID data to a source based on Template #72. This aborts updating the Citation data and displays the second critical error message warning the user to undo the updates. You didn't say that the plugin crashed after this, so I will assume it stopped cleanly.

If you took the advice to undo changes, could you run it again please, and review the structure of Source Template #72 (this is the Record Id, not the RM template number). Are the data fields and their definitions the same in RM and FH? If there is a discrepancy, the first prototype would not have spotted it as it doesn't do anything with the templates. It's only once we start using them that any mismatch in structure comes to light.

Ditto for Source Template #6 if that was also the same Meta-field error, please.
Mark Draper

User avatar
BakerJL75
Famous
Posts: 128
Joined: 14 Dec 2020 11:29
Family Historian: V7

Re: Import of RM7 Citations

Post by BakerJL75 » 04 May 2021 17:56

I have an appointment in a bit. When I get back would it help if I make a bigger gedcom and strip out the living? And then email it to you in case I miss something.

For both #6 and #72 the fields are defined the same way.

#72
In FH:
<{Subject}, ><{Title}, ><{CreationDate}, |undated clipping, ><from unidentified newspaper;>< {Collection}><, {Provenance}>; <privately held {YearOwned:Year}> by <{Relationship}, ><{LastKnownOwner}, ><!{StreetAddress}|{<sc>address for private use</sc>}><, {CityAddress}><. {Annotation}>.

In RM:
<[Subject], ><[Title], ><[CreationDate], |undated clipping, ><from unidentified newspaper;>< [Collection]><, [Provenance]>; <privately held [YearOwned:Year]> by <[Relationship], ><[LastKnownOwner], ><![StreetAddress]|[<sc>address for private use</sc>]><, [CityAddress]><. [Annotation]>.
#6
In FH:
<{Author}, ><"{Article}," ><{Type}, ><i><{NewspaperPlace}|{Newspaper}></i><, {IssueDate}><, {Pages}>; database and images, <i>Ancestry.com</i> (http:///www.ancestry.com< : viewed {AccessDate}>)<, {Collection}><; {CreditLine}><. {Annotation}>.
In RM:
<[Author], ><"[Article]," ><[Type], ><i><[NewspaperPlace]|[Newspaper]></i><, [IssueDate]><, [Pages]>; database and images, <i>Ancestry.com</i> (http:///www.ancestry.com< : viewed [AccessDate]>)<, [Collection]><; [CreditLine]><. [Annotation]>.
Thanks,
Jackie

Post Reply