* FH-RM-Ancestry Sync to exploit hints (update)

[Mark1834]

This is a direct continuation of the earlier discussion of syncing FH with Ancestry via RM to exploit hints, but I've started a new thread to avoid having to scroll through three pages of now largely irrelevant material. It does, however, build on the approach suggested by Shosh Kalson in the previous thread.

The existing process is not very elegant, and requires many more manual steps than is desirable. Now that FH7 has added support for the UniqueID tag and the ability to connect to RM databases directly, there is an opportunity to improve this considerably.

I have a draft plugin that I am testing, but before posting it here, it is worth outlining the new workflow to get feedback from potential users. The existing plugin has had nearly a hundred downloads, but there has been relatively little user feedback. I don't know if that is because it is working perfectly, or users have given up on it as too complex.

There are two steps to the process - updating RM to reflect changes in FH, and syncing those changes with Ancestry. The first step was the weaker one, due to limitations in the RM file comparison tool, but that has now been largely automated by reading RM data directly from FH. The subsequent sync with Ancestry is still manual, but we have no access to the Ancestry API to modify that. However, it is now much better guided.

These screen grabs illustrate how it is intended to work, where the core feature of the new plugin is an audit report of all the differences between FH and RM, saved as a Research Note.

1. Export a GEDCOM from FH into RM using the plugin one-click approach to generating a sanitised GEDCOM extract. It doesn’t matter whether you use RM7 or the new RM8 - it works identically in both versions. The plugin demonstrates that the two files are equivalent, using the Sample Project as an example.

Original.PNG (26.57 KiB) Viewed 7888 times

2. Continue with your research until you are ready to update Ancestry. I have married off Judy Susan Munro, adding a husband and three children, updated a range of facts for other individuals, and detached Amanda Morag Munro from her parents, as further research has indicated that she is not their daughter.

3. Run the plugin to generate an accurate and complete list of all the differences between FH and RM.

Pre-update.PNG (39.77 KiB) Viewed 7888 times

4. Generate a new export GEDCOM (one-click via the plugin), and import into the existing RM file. Use Share Merge in RM (which is designed for exactly this scenario, updating the database from an external GEDCOM) to automatically add new individuals and data and create family links. The existing process using RM File Compare requires all new family links to be created manually, which is tedious and error-prone, as well as generating many false positives in its assessment of which records have changed.

5. Run the plugin a second time to list any remaining differences that need to be tidied up manually, as FH is limited in its ability to create, link and delete RM records.

Post-update.PNG (28.99 KiB) Viewed 7888 times

6. Finally, update Ancestry from RM, using the audit report generated in step 3 as a complete and accurate guide of which records need syncing.

I want to give the plugin a good test before posting it using real rather than just test data, but comments from potential users are invited. Any interest?

To keep it focused, I suggest we don't revisit the “hints are useful/waste of time, do it this way instead” discussion. We've had that already (several times... ).

[David Potter]

Hi Mark

This sounds very promising - I need to read the post a few times over before any comments. But certainly I am an interested party.
I made some progress with the tools you recommended yesterday: firstly creating the UID's in FH7 and then running your formatting Plug-In choosing option 3. Then using the FH7 Gedcom Export function to create a GEDCOM with the Default option selected for the target App. Imported this GEDCOM into RM8 and managed to confirm using SQLite that the UID matched across FH7 and RM8. I then ran the RM8 import once again and imported the exact same GEDCOM to see if RM8 through up any messages regards the duplicate UID's but it did not. Lastly I ran the Auto Merge to see if the UID might now play a part in that process and as you have pointed out before it does not. So I ended up with a few hundred additions where the Auto Merge probably couldn't distinguish enough to suggest a Merge. IE, insufficient facts to work with for these Individuals, I'm not sure if that is the true reason though as to why it did not Merge. (I'm still going through a major Do-Over now FH7 exists) so some individuals are without Facts.

Next steps
I'll read up on that Share Merge feature to get familiar with it and test it out.
I was considering approaching RM8 development to ask if the UID could become part of the Merge processing as an optional choice - what do you think?

Cheers - David

[fhtess65]

Just so I'm clear, all this happens within your plug-in? If so, that's very cool. Will be busy for the next couple of weeks but would love to play with this plugin as overall I find FH to be the superior program, but also want to keep my Ancestry tree current without retyping.

Also, is there a way in the plugin to move info from RM to FH? Or is it just one way, FH to RM? I admit to having a fear of keeping all my eggs in one basket.

Thanks,

Teresa

[fhtess65]

Further question - I assume it's safer to use the plugin in a copy of my project, not the master one?

Teresa

[Mark1834]

All in the plugin, Teresa . But to be clear on scope - it is not designed to keep a full project replicated across FH, RM, and Ancestry, but to manage an extract (key standard facts only for non-living/private individuals, with no media, sources, or other frills), such as you would use for hint generation, cousin-bait, or DNA matching. The source data always comes from FH.

Added in edit - if all you want is a copy of your tree on Ancestry, any GEDCOM export will do, either the full fat Export Gedcom plugin (most options), standard FH export, or my plugin (simple sanitisation). It’s only worth going via RM if you want to preserve hint history.

[ColeValleyGirl]

I'm looking forward to using this if/when I ever get round to updating my research.

[tatewise]

It’s only worth going via RM if you want to preserve hint history.

You would need to go to RM if Media are important because Ancestry does not bulk upload Media.

[fhtess65]

All understood - I will play with it after my vacation (will be visiting someone, so little time for genealogy) and see how it goes.

Thanks for the information

Teresa

All in the plugin, Teresa . But to be clear on scope - it is not designed to keep a full project replicated across FH, RM, and Ancestry, but to manage an extract (key standard facts only for non-living/private individuals, with no media, sources, or other frills), such as you would use for hint generation, cousin-bait, or DNA matching. The source data always comes from FH.

Added in edit - if all you want is a copy of your tree on Ancestry, any GEDCOM export will do, either the full fat Export Gedcom plugin (most options), standard FH export, or my plugin (simple sanitisation). It’s only worth going via RM if you want to preserve hint history.

[David Potter]

Hi Mark
I have read this over and don't have any further comments apart from when can we test the new Plug-In?

Thank you.

[Mark1834]

Thanks David. I’ve been exploring potential additional options, such as allowing living people to be copied, synthesising missing birth and death data, etc, but decided in the end that they add too much complexity for a first prototype. The whole point of this plugin is that it produces a consistent output without having to remember multiple settings.

The list of events to be included will be easily modified (and can include custom events) by a simple edit of the plugin script.

As it stands, the plugin isn’t a drop-in replacement of the earlier KB version for existing users, as the existing RM UniqueID values will need to be used to update the FH project to maintain the link between apps. For my main project, that wasn’t an issue, as it uses a unique Custom ID for each individual that I made sure I exported right from the beginning for just this eventuality. It was a simple additional script to match individuals on Custom ID and update the UniqueID accordingly. I have another mid-size project (~ 1000 people) that doesn’t use Custom IDs, so I will need a more generic solution for that.

My idea at the moment is to match individuals on full name, lifespan dates and spouse full name. That should take care of most people.

I’ve got a busy week coming up, so it might be next weekend before it’s ready for posting.

[David Potter]

Thank you Mark. That sounds good and gives me time to finish my last few Re-Key's in my tree under FH7.

Best wishes

David

[Mark1834]

Prototype plugin attached (Now removed as it has been superseded). Use as per earlier postings. Additional points to note:

• The plugin takes care of formatting UniqueID values, so there is no need to convert to an RM-compatible format prior to export.
• It works equally in RM7 or RM8, as the relevant tables are unchanged.
• It does not write anything to your project apart from the Research Note audit report, but as ever, it will be safer to use on a copy, not the original. Also best not to risk your live Ancestry sync if you already use the original version, so generate a new RM database for testing.
• The help button brings up the relevant page on the KB, but this has not yet been fully updated for the new plugin.
• The list of Facts exported and compared can be changed by editing the plugin script at lines 112 to 119. Instructions are included in the plugin.
• There is no enhanced memory management included in the plugin (which is often needed in FH7 for large datasets), but it is not anticipated that very large projects will be synced to Ancestry (if you do, you’ll be spending a LOT of time reviewing hints!).

Remember, this is NOT a general FH-RM comparison tool, and should be used only in the context of the plugin. Over to potential users to take for a test drive…

[David Potter]

Thank you Mark. I'll begin testing with this Plug-in tomorrow.

[Mark1834]

I have made some minor changes to the script as a result of my own observations in use. The posted version does not identify differences if the Fact is simply deleted in FH rather than updated. In addition, if you change the set of Facts exported, it does not flag what are now redundant events in RM for deletion. I have also separated out a "Compare" function, which compares databases with no changes, and an "Update" function, which updates RM with no further confirmation. This is a better workflow than asking for update confirmation one event type at a time.

They are all "nice to have" refinements rather than key changes, so I will wait for more substantive changes before reposting to avoid too many versions on the forum.

[fhtess65]

I'll be watching for it...Still haven't had time to try it. Am on vacation again in two weeks and hope to devote some time to my genealogy work (it's my birthday week, so I'm going to be selfish)...

Am really curious to see how this works.

Teresa

[David Potter]

Hi Mark
I have had a run through of your plug-in and hit an issue when using the compare feature. The compare identified a Fact in RM8 that did not exist in FH7. This was deliberate. The plug-in presented me with an option to Delete that RM data. I chose YES and then received a String Error of some kind. I didn't grab a screen shot of this sorry. I'll resume further testing and try to repeat this issue and capture the details.

I look foreward to the new version when released...

[Mark1834]

Thanks David. Did you create the extra fact in RM directly, or was it exported from FH? Plugins have only a limited ability to write or delete RM data, so my first thought is that it might contain a field that FH cannot amend.

[David Potter]

Hi Mark it was created in RM8 using treeshare and downloaded from Ancestry. It was a birth index EG GRO flavour.

[Mark1834]

Without trying to duplicate exactly what you did, I suspect that was out of the plugin scope. The assumption is that FH is the master version of all the data, and an extract of this is sent to RM and on to Ancestry. This extract is deliberately kept very simple, so it can be managed in detail - just who, what, where, and when, with no sources, media, notes, witnesses, etc.

If data are created in either Ancestry or RM directly, it does not feed back to FH - it is strictly one-way updating, so the plugin will attempt to delete it. If the plugin created the data, it understands how it is structured, and can delete it completely. If RM or Ancestry created the data, there is a risk that it is more complex and has other elements that the plugin is not familiar with or capable of deleting.

I don't like error messages, as they often mean that the plugin author has not anticipated a potential error condition. I'll give some thought to how best to deal with the scenario of trying to sync an existing RM and/or Ancestry tree. Perhaps the plugin should actively check for things in the RM file that might cause problems.

[David Potter]

Hi Mark, yes I agree I was literally messing around and trying to better understand the behaviour of the Plug-In in that situation. Also, I'm not yet fully conversant with RM8 and Treeshare capabilities so apologies for time wasting.
I'll read again the scope the and process flow and keep within the design.

Thanks for the response.

[StockportGerbil]

Sorry if this is not the right place to post this...

I want to synchronise Family Historian with Ancestry for the following purposes

1) Use and retain across updates Ancestry hints
2) I want to retain the tree that DNA tests are attached to across updates and thereby supporting Thrulines. (Note that the DNA tests are carried out against living people.)
3) I want to retain the tree such that I can add limited media (for which I have the rights) that is retained across updates.

Because of point 2, I can’t use either “Family Historian – RootsMagic – Ancestry Sync (0.1)” or “RootsMagic Gedcom Export for Ancestry” to export the initial gedcom as they do not export living people.

I’ve tried the built in export into a gedcom file and the “export gedcom file” plugin (both excluding sources and multimedia) and then use the compare function in “Family Historian – RootsMagic – Ancestry Sync (0.1)” to check the import into rootsmagic works as hoped for. Unfortunately compare fails with an error “[string ‘’C:\ProgramData\Calico Pie\Family Historian\Pl…]:533: attempt to concatenate a nil value (field ‘?’). No changes have been made to data records.”

Is what I am asking for reasonable / realistic? Is it possible that the ancestry sync could include living / private people but retain the flags during the export process? Is there any other way I can do this?

Thanks

[Mark1834]

This is definitely the right place, so let's see what is possible...

As it happens, I have a significant update to the plugin ready for posting, but I'll hold off for the moment in case there is anything from here that can be incorporated.

Your error message arises as you are comparing the FH project with a RM file that it did not create, so there will be mismatches in the structure. The update checks for this before making the comparison, and warns the user that what they are trying to do is out of scope.

There is deliberately no menu option to include Living and/or Private Individuals, as I want to keep the operation completely automatic. However, the plugin is designed such that the user can make a simple change to the script to permanently change this selection (for example, include Living but still exclude Private). I can describe how to do that if it is of interest.

Adding media to the tree is a little bit more complex. I'm reluctant to include media links in the plugin as that significantly complicates the interaction with RM and the number of links that need to be considered. However, that does not stop you adding the media directly to the Ancestry tree. It could be tedious if you have a lot of items, but they should be retained during the update process.

It's not clear whether you have your tree on Ancestry already and want to build the link to FH, or are thinking about how to add it. RootsMagic TreeShare cannot link an existing database to an existing tree. When it is set up, you have to either upload a new tree from an existing database, or download an existing tree to a new database. Establishing a link between an existing Ancestry tree and your FH project is therefore more complicated. In principle, it should be possible, but it involves downloading the tree to a new RM database and updating the FH Unique ID values (which are used for record matching) from this.

If you can give a bit more background about exactly what you have now in terms of FH project, Ancestry tree and any linked RM database please, I can give more specific guidance.

[StockportGerbil]

Hi!
Many thanks for the prompt and full reply, it is appreciated.

I didn't realise the plugin was potentially readable (and editable?) in notepad and would be interested in how it can be updated to allow living people to be transferred.

I'm not looking for the plugin to share media, the vast majority of my media is downloaded from ancestry anyway, but I do have some family albums with photos going back to the 1800s which I think should be shared and I am happy to do that once manually, but don't fancy doing it every time I do an update to my tree.

At present my way of working with ancestry has consisted of uploading a new version of the gedcom - moving the DNA to reference the new tree and start from scratch with the hints and don't bother with the photos.

I accept I will need to add a new tree on ancestry and am happy to do that / start from scratch, but am looking for the ability to update that tree only with Family historian being the master copy from then on. To put it another way, I think it would be a nightmare trying to reconcile what's currently on Ancestry with what's in Family Historian.

Thanks again

Kevin

[Mark1834]

Thanks Kevin, that is clear. My recommendation is to wait for the plugin update later this week and modify that. I will need to check that Living flags are read into RM and synced across to Ancestry correctly, but that's easy enough to do.

By way of background, plugin scripts are written in a programming language called Lua. The scripts are plain text files that can be created and modified in any text editor. FH has a simple editor included but it is hidden by default as most people won't use it. In the Plugins Window, click on the More>> button to give a column of additional options, including Edit...

[Mark1834]

A couple of minor constraints that I have noted in testing-

• RM/Ancestry do not have an equivalent of the FH ‘Private’ flag that applies to the whole record, so if Private Individuals are exported they become visible to anybody who can view the tree.
• Syncing ‘Living’ flags between FH and RM is straightforward, and they are uploaded to Ancestry correctly for new Individuals or Trees. However, if they are changed in FH, TreeShare does not recognise that RM and Ancestry now have different values. However, it’s easy enough to update Ancestry manually from the TreeShare window, and the plugin will prompt you to do it. It won’t know whether it is done or not, so it is the user’s responsibility to ensure that Ancestry is set up correctly.