* FH -> RootsMagic -> Ancestry (Work-in-Progress)

[shoshk]

This is a continuation of a discussion started in this thread: Making the Move from Generic to Templated Sources (18549).

This is a work-in-process. A lot is based on the work and experiences of Mark Draper.

I want to upload my database to Ancestry in order to facilitate research, both through hints and directed searches. FH will store the master database; all changes will be made there (and only there). Periodically, the RootsMagic database will be updated with changes from FH, and from there, uploaded to Ancestry via the TreeShare feature.

I am currently implementing the FH -> Rootsmagic leg of the journey. This is what I've got so far.

1. Treatment in FH:

(a) Plugin (custom) -- Each individual in FH is assigned a custom id (REFN) with the format FH-<nnnnn> where <nnnnn> is the record id for the individual, left-padded with zeroes. This enables me to uniquely identify them both in RootsMagic and Ancestry.

(b) Manually -- make sure that all individuals have a name and that all families have at least one parent. RootsMagic doesn't like individuals without a name and families without a parent.

(c) Plugin (custom) -- Generate a gedcom which contains only INDI and FAM records with a subset of tags and only DATE and PLACE details for facts.

2. Treatment in RootsMagic - first import

(a) Import the gedcom generated in step 1a, above.

(b) Export a gedcom, including only 'Extra details: (RM specific)' under Data to export. The purpose of this is to update the FH database with RootsMagic's unique identifiers for each individual (_UID). This will enable using the automatic merge feature (ShareMerge) after future imports from FH.

3. Treatment in FH

(a) Open the gedcom generated in step 2a, above. Use the Split Tree Helper to remove all facts. Use 'Delete all facts except list below'. This should leave only NAME, SEX, REFN and _UID tags as well as relationships. TBD: I really only need the individual records. It might be a good idea to write a plugin to strip the gedcom. This will reduce the size of the gedcom and make step c, below, run faster

(b) Close the gedcom and open the project.

(c) Plugin (custom) [TODO] -- update/create _UID tags, matching individuals using REFN.

4. Treatment in RootsMagic - subsequent imports

(a) Import the gedcom generated in step 1a, above.

(b) Use Tools -> Automatic Merges (Share Merge) to merge the resulting duplicates. There will be many.

(c) If you have added new individuals in FH, follow steps 2a and 3a-c, above to create _UIDs for them.

Things to Think About

(a) This procedure will not handle deletions from FH. I believe that this will need to be handled manually. [TBD] Can we devise a way to capture deletions using an automated process? What about Compare files in RootsMagic?

(b) Others?

[Mark1834]

If we can make the linking less clunky that will be a great outcome, so thanks for this.

I was just about to put my spare FH7 beta test NUC back in the cupboard and reinstate the Linux Mint box in the second space under my desk, but I can see it has one more job to do . I'll do a complete run through on a test database using my existing export script but completely separate from my live installation and Ancestry tree and report back...

Added in edit: I was installing RM7 on my test PC (their licence appears to allow unlimited installs, but only one used at a time), and received notification that the promised community beta of RM8 is now available. It can sit alongside RM7, so I will test on both. That may delay me slightly - issued previews indicate that the RM7->RM8 change is bigger than FH6->FH7, so I've got some playing to do!

[shoshk]

I've been beta testing RM8 for a little and just upgraded to the community preview version. As you say, there's a lot to get used to. I do enjoy the new look and feel.

That being said, I don't see myself moving to the new version any time soon. I'll let others shake out the bugs. For my purposes (bridge from FH to Ancestry), v7 seems to be sufficient.

[tatewise]

Guys, I have moved this thread to Importing and Exporting that is more appropriate. I hope you don't mind?

It also sounds like a good topic for the Knowledge Base...

[Mark1834]

I don’t think I’ll spend too much time on RM8. Visually, it looks really nice, but its file comparison routine is still broken (hundreds of absolutely identical records being reported as different) and I’ve managed to crash it completely twice in my first half hour of playing with it!

[mjashby]

Users on this Forum should be very careful about making comments/sharing opinions on the RootsMagic 8 Beta software. Testing is ongoing and is still covered by a Confidentiality Agreement which applies to anyone who has gained access to the Beta releases; and we should fully respect that in the same way that the FH7 Beta Testing was.

Mervyn

[Mark1834]

Mervyn - agree that closed beta testing is confidential. FH7 testers have a continuing obligation not to disclose bugs that were uncovered in testing, what still persists in the release version, and what changes were made (although CP have removed access to the detailed agreement, so we cannot check exactly what the terms are ).

However, the community preview is public and open to anybody with no confidentiality agreement. The only stipulation is not to contact their tech support regarding problems, but use a dedicated feedback process. I fully agree that it would be grossly unfair to RM to reward this openness by detailed discussion on a rival product's user forum, but it is fair comment to make a brief reference to a key feature that we are evaluating for use with FH to plug a gap in this app's functionality. All I am saying is that it is not ready for this use yet. If this is corrected in the final release, it could become a very useful tool for us.

[shoshk]

I'm with Mark.

Mike, I agree that the appropriate forum for this discussion is Importing and Exporting.

[shoshk]

Mark,

I was thinking about your problem with losing relationships when importing new people using your technique.

I believe that if you are entering a new person who has relationships with other new people, when you enter his record, you are entering references to people which have not yet been defined in the database. And so, RM doesn't enter the references.

A potential solution to this problem, which would continue to use your technique, would be to first generate a gedcom with bare bones (NAME, SEX, REFN) data for individuals; that would enable you to enter them before trying to enter the relationships. Then you would generate your 'full' export and add facts and relationships. Basically, you would be doing two passes. More work, but I believe that it might solve your problem.

Also, I've been thinking about UIDs. In the middle of the night (seems to be a common time for thinking about stuff), I remembered that somewhere on my computer is the information regarding how RM generates UIDs; I believe it is not a proprietary algorithm. It may be possible to generate the UIDs on the FH side and save ourselves the RM -> FM step. I will check it out and get back to you.

[Mark1834]

Shosh, I've been experimenting. There's quite a bit to report here, but I'll take it one step at a time...

1. My database already has unique custom IDs, so can RM merge records based just on this parameter? No. If I import the same FH-derived GEDCOM twice into the same database (so it has exactly two copies of each individual) and then merge them, there are a lot of duplicates created.

2. I read in an early blog post that RM could use an existing _UID tag rather than assigning new ones, so I converted all the 1 REFN values to 1 _UID, imported into RM, then exported to a new GEDCOM that I could inspect. RM had blown away my old values and replaced them all with new ones, so that option did not work.

3. I imported my FH-derived GEDCOM into a new RM database, exported a GEDCOM, then re-imported that, adding the records to the existing database. This is similar to 1. above, except that this time the RM-assigned UID tags are included. An automatic merge on this file worked perfectly, and restored the database exactly to its original state. UID-based merging therefore works, but REFN-based does not, even when REFN is present for all records and unique.

4. I imported my FH-derived GEDCOM into a new RM database, and made an exact copy of the file with Windows File Manager. I then made selective modifications to the copy, adding one individual, deleting another, adding one fact to a third, deleting one fact from a fourth, and modifying a fact for a fifth. Finally, I exported a GEDCOM from this modified copy. This simulates an updated file coming from FH after further work on your tree. On importing this modified copy into my master unchanged database and merging the records, all the additions were present, but so were the deletions and original version of the modified fact. I rationalise that by assuming that there is no concept of which is the master version of two matching records, and it simply combines all the information present in both of them.

This suggests that this process will work very well if all the database modifications are just additions, but changes and deletions look harder to handle. It would be ideal if RM had an option to identify which duplicate records were non-identical and present them for manual review, or even better to automatically update file A to mirror file B, but neither option seems to be there at the moment.

I’ve also looked at your suggestion for not copying across family relationships when comparing between files, and unfortunately I don’t think it will work. I tried simply adding a new child to an existing family, but it still did not bring the family relationship when copying between files.

Where do we go from here? Merging records in the same database works well for additions, but not changes or deletions. Comparing databases copes with all these scenarios, but is slower as families have to be rebuilt in RM and individuals and facts updated from a crib sheet to avoid missing some.

[shoshk]

Mark, I'm going to continue this discussion in several posts, which I hope will keep things a little easier to follow.

First, UIDs...

I had hoped that we could generate UID's in FH and thus save the return trip from RM, but it turns out that, while technically possible, it's not a trivial task. Without going into details, I think it best to leave the "heavy lifting" to RM for the moment.

I have, written 2 plugins to facilitate this process.

The following procedure should be used:

In RM

1. Export a gedcom, including only 'Extra details: (RM specific)' under Data to export.

In FH

2. Open the gedcom generated in step 1, above. Note that this is done under the Project Window dialog under MoreTasks -> Gedcom File Tasks -> Open Gedcom File.

3. Run the RM Import - Preprocess plugin. This will create a csv file with the values for RecordId, REFN (not really needed, but I put it in for debugging purposes), and _UID.

4. Close the gedcom and open the project.

5. Run the RM Import - Update UID plugin. This will add/update the _UID tag for all individuals specified in the csv file.

Let me know what you think.

[shoshk]

In response to your last post ...

You are correct, using Custom Ids (REFN) for matching does not work in RM. I believe that they are ignored entirely. The only reliable method is to use _UID. UIDs must be in a very specific format. Tamura Jones has a good discussion of the _UID tag in the following post: The _UID tag. It's old but still, for the most part, relevant.

Regarding what I would call 'Update Imports,' I think we have some (actually probably quite a bit of) work to do to get this right. Using the automatic merge feature is quick and easy, but definitely has its problems since it doesn't take care of changes and deletions.

My plan is to work on testing various scenarios during the coming week and, hopefully, finding solutions to problems.

I'll keep you informed of my progress.

[shoshk]

Warning...

RM Import - Update UID does not work!

I made some changes before posting it and it seems that I broke something.

I'll figure it out and repost.

Sorry

[shoshk]

Ok, it does work. But I've observed a rather strange behavior.

It adds the _UID tags, but they don't show up in the All tab until I close the project and re-open it. Strange...

Mike, do you have any ideas why this might be happening?

[Mark1834]

That goes some way to explaining why UID is a 36 digit hexadecimal. 22 million million million million million million million is just a touch excessive to ensure a unique value for each person .

[shoshk]

Did a little playing around this evening and feel that I can state conclusively that Compare files is buggy and cannot be relied upon.

I created a new tree.

I created another tree which is an exact copy of the first using the RM copy function.

I used Compare files to compare them.

It found a bunch of people with differences. The files are identical!

In this particular case, it seems that RM is having a problem with Names (it's duplicating them) in the compared file. When I click on Edit person to see what's going on, the names show up as they should.

It doesn't duplicate names for everybody. Why is anybody's guess.

Anyway, since it doesn't work when comparing files which are absolutely identical, I see no reason to experiment with seeing how it handles differences. I don't think we can trust it to give reliable results.

So... Back to playing.

[Mark1834]

Agree, its problems are widely discussed on RM’s version of FHUG. It works with a crib sheet of which records have changed, but not without. Particular issues appear to be handling aka names, and child order in large families.

[shoshk]

If we make the decision to stick with RM7 for now, SQLite Tools for RootsMagic may provide solutions for some of our problems.

Unfortunately, according to the FH Plugin Help, LuaSQL (which would enable us to access the RM database from a plugin) is not currently available in FH7.

Can we install it manually and use it as a module?

[Mark1834]

It is already in the public domain that RM8 uses a new incompatible file format, so that might be an issue. While it will presumably retain the ability to read and write the old format, it’s another potential complexity if RM7 is no longer available.

[shoshk]

RM8 still uses a sqllite database.

But the database structure has changed. understanding and manipulating the new structure is not an insurmountable problem.

Also, I don't see a problem to stick with RM7 for the time being. I think it will be a while before they shake all the bugs out.

[shoshk]

checked Compare files again on a database without AKA names. Observed the problem with child order. There were also problems with events.

Useless feature in my opinion.

[Mark1834]

That’s a bit harsh. It suffers from too many false positives, but has zero false negatives (updated records that it says are identical) and does enable an Ancestry tree to be updated without losing your hint history. At the moment, it’s the only working method we have...

[shoshk]

I guess I'm feeling a bit frustrated.

92 false positives on an identical db compare. That's after I excluded AKA names. It randomly duplicates events in the 'compare file'.

I just get this feeling the process will be very prone to error.

[Mark1834]

Indeed, too much of it is outside our control. If you haven’t done so already, it might be worth you having a look at the RM-Ancestry interface as well. I can’t see any option for that other than having to use it as it is, so it needs to fit into the end-to-end process.

[shoshk]

Mark, good morning! After a bit of sleep, breakfast, shower and brushing my teeth, I find myself in a better mood. Still frustrated, but better able to deal with it.

In designing this process, one of the considerations for me is that my husband needs to be able to execute it with a reasonable chance of success. If there's too much room for human error, his chances of success over time are pretty low. He's not at all stupid (he has a PhD), but he's not a power user.

I agree with you that Compare files is our best option at the moment as it provides the most options for comparing and updating data. If it wasn't broken, I'm pretty sure that I wouldn't be considering any other options.

I'm going to keep playing.

BTW, in one of the posts on the RM forum, I read that, on the request of what is probably a power user, RM added using _UID to match individuals if present in both files. Nice to know ...