* Parallel Family Historian and Ancestry trees

Importing from another genealogy program? This is the place to ask. Questions about Exporting should go in the Exporting sub-forum of the General Usage forum.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Parallel Family Historian and Ancestry trees

Post by Mark1834 »

I have uploaded a simplified copy of my FH tree to Ancestry to exploit their automatic hints (which despite the obvious limitations, do occasionally come up with genuinely new information as new record sets are released). I'm not clear about the best way of sustaining this in the future as my FH database develops and new hints become available, as the two programs don't talk to each other very well. My impression at the moment is that I will have to make manual changes to both in parallel if I want to keep the data in sync while also preserving the reviewed status of old hints, as a fresh upload seems to blow away all the old hints and starts again. There does not appear to be any facility in Ancestry for a bulk update to a tree maintained outside their program (and with my commercial hat on, I can see the logic of that from their narrow viewpoint.... ).

However, I'm happy to be corrected if there is a better way that I have missed!
Mark Draper
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by ColeValleyGirl »

Yes, as far as I can tell, that's the only way... manual changes in parallel.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

Thanks Helen - I was afraid that would be the answer... :(

A better technique might be to load a series of sub-trees, so the focus can be on individual branches rather than the scatter-gun of all records. I'm thinking perhaps one tree for each grandparent plus recent non-living family, so five in total.

What's the best technique for splitting out a sub-tree to contain everybody upstream of a given record? My general rule is to record to first cousin level (so direct ancestor siblings and their immediate families), so I would want everybody included, not just direct ancestors and siblings. I have a long-standing system of personalised record identifiers that indicate exactly where in the tree somebody is using letter codes for generations and a series of binary flags for individual branches, so it would be relatively straight-forward to populate the sub-trees by querying on this identifier, but is there a simple technique for doing it within standard FH tools and queries?
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by tatewise »

Yes, Helen is essentially correct, but there are a couple of other ideas.

If you are mainly interested in BMD & Census records that are missing from your Project, then consider running the Lookup Missing BMD Records and Lookup Missing Census Facts Plugins, that not only offer searches of Ancestry but also other online databases. Unlike 'hints', they are not automatic, and only search BMD & Census records, but do focus on what is missing from your Project for whichever Individuals you choose to research.

Another option involves the RootsMagic product, which offers TreeShare for Ancestry to synch RootsMagic with Ancestry.
That is available even in the RootsMagic Essentials free version (although I have not investigated the feature fully).
It allows all RootsMagic data including Media to be shared with an Ancestry tree in both directions.

So you could experiment with sharing with Ancestry and use RootsMagic to selectively Export data to GEDCOM for Import into FH and File Merge with your master Project.

The reverse route is also possible to Export GEDCOM from FH for Import and Merge in RootsMagic for sharing with Ancestry, but the Merge command in RootsMagic is only available in a bought version (about £20).
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

Thanks Mike. I find the main benefit of hints to be the more obscure references it can produce, so they are worth persevering with for that reason alone, not just the "obvious" gaps in census and BMD data that can be readily searched on other sites.

I'll experiment with maintaining the simplified trees or sub-trees (just key events, without sources, media, notes, etc) in RootsMagic if that can be kept in sync with FH more readily than Ancestry itself. Even when I used earlier editions of FTM as my main desktop program, I still made all updates manually after checking hint validity rather than blindly copying Ancestry records, so would continue that way of working. A monthly update would probably be sufficient for my purposes, rather than real-time sync.

I guess the same principle also applies to the current edition of FTM, but £20 is a much more reasonable price for a helper program than the £80 demanded for FTM now!

Checking the RootsMagic website, it appears that the US version is available for USD 20, but it's probably worth paying the little bit extra for the UK-specific one. I'll play with the free one first, though!
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by tatewise »

RootsMagic is easier to interoperate with FH as its GEDCOM implementation is far better than Ancestry/FTM.
It even supports Fact Witnesses similar to FH.
See plugins:help:export_gedcom_file:rmt_roots_magic_tree|> Export Gedcom File ~ (RMT) Roots Magic Tree for details of what is different in RootsMagic.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

First impressions playing with RootsMagic are quite positive. Its Ancestry integration works well, and it is actually easier to review Ancestry hints within RootsMagic rather than on the Ancestry site or via their iPad app as more relevant information is displayed on-screen at once. I reject all hints, even if relevant, as I want to keep the Ancestry tree clean and not populated with their own records and sources. The program gives you the option to push updates in either direction record by record, rather than the blanket syncing that FTM uses. I used the standard US version of the program (less than £16 as a download), as it is only the support package that differs for the UK one, not the core program.

Having data entered in three different programs is a recipe for disaster given the inconsistencies in GEDCOM implementation, so my approach is to enter all data strictly via FH, so that is my master copy by definition. It's not worth doing all this just for one or two updates, but after large changes (e.g. when major new record sets such as the 1939 Register or your particular local parish records become available) it certainly saves a lot of tedious data re-keying and the associated risk of errors.

It will be necessary to produce exactly the same type of export from FH each time so records merge and update cleanly within RootsMagic, so I will look at getting back to writing plug-ins to ensure consistency. If living or private flags are set, do not export. For all other individuals and families, export all facts, but only the associated date and place. Delete all source, place, multimedia, note, etc records. Doesn't sound too difficult........
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by tatewise »

You have investigated that RootsMagic capability far more than I have time or motive to, and it seems to offer a useful way of keeping FH and Ancestry trees in synchronism under your terms and with little financial cost. It sounds like a technique that can be recommended for others to use.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

I’m happy this approach works in principle, so I’ll start using it for real and report back later how I get on.
Mark Draper
User avatar
mjashby
Megastar
Posts: 719
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Parallel Family Historian and Ancestry trees

Post by mjashby »

Mark1834,

If you want the UK Version of RootsMagic, it can be downloaded via this link: https://rootsmagic.co.uk/demo/ - The the RootsMagic Essentials (Free) Version is identical to the full version, but If you already have a full licence purchased direct from the RootsMagic Site, simply enter the Licence details to unlock the paid for features and you're ready to go. There is no separate arrangement for UK/International licences, i.e. all licences are supplied by RootsMagic Inc. (via e-mail for downloads purchased from other suppliers.) Explanations of 'what's different' about the UK Version, apart from the UK Badging, are difficult to find but seem to be limited to a UK focussed Spellchecker and some UK specific settings that are pre-installed by S&N Genealogy Supplies as an aid to new users and, no doubt a small selling point for S&N in the UK.

I also use RM as a secondary and relatively straightforward method of checking Ancestry Hints and downloading relevant source documents prior to entering any Ancestry sourced data into Family Historian. Worth mentioning the RM also works similarly with FamilySearch Family Trees, 'though that's not a feature I use.

Mervyn
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

One key difference between FH and RootsMagic is that the latter does not permit unnamed individuals, whereas FH does. There appears to be a bug within RootsMagic whereby a GEDCOM import containing such entries succeeds without raising any error flags, but it is then virtually impossible to edit the unnamed person. Critically for this Ancestry linking application, the unnamed person causes the RootsMagic database comparison to crash rather inelegantly.

See my discussion of this on the RootsMagic forum at http://forums.rootsmagic.com/index.php? ... databases/.

Before going down this route, it will be necessary to modify my FH database to add dummy placeholder names where the actual name is not known (for example, an unknown earlier spouse for somebody described on marriage as a widow(er)). Not difficult, but I've seen the earlier discussion on these forums that tell me there are multiple ways to do this, and no right or wrong method!!
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by tatewise »

Depending on whether there is a quick response from RootsMagic, the Export Gedcom File Plugin could detect unnamed Individuals and give them at least a surname such as /?/ just for RootsMagic.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

Worth updating with my experiences so far -

If you are exporting a subset of your data, it is vital that you are consistent in what you save, to avoid any new or missing fields being interpreted as new information. I have very specific requirements for my Ancestry copy, just Individuals and Families at the top level, so no media, sources, etc, and only basic data on key events (only DATE and PLAC for NAME, SEX, BIRT, BAPM, CENS, etc, about a dozen in all). Also no information at all on living or private individuals.

It was getting a bit complicated running the various selection and export routines in FH and plug-ins consistently, so I considered customising the published plug-ins to do exactly what I wanted. However, they looked a bit complex given my current rudimentary experience with plug-ins. A better technique for me was a separate Python script that read the FH GEDCOM data file directly and had all the required selection, processing and filenames hard-coded in (including converting CENS to RESI and inserting dummy names where necessary to appease Ancestry and RootsMagic, respectively!). That works well, and the entire script prints on one sheet of paper. It can also be run directly from the Windows desktop at any time with just one click.

Merging the updated database into RootsMagic and subsequently up to Ancestry is relatively straight-forward, but neither routine is completely reliable so some care is required. Within RootsMagic, it is well-recognised that there is a bug that means identical individuals can be shown as slightly different. If you are making a small change to somebody who already has lots of data available, you could miss it among the large number of identical records that are shown as only 98-99% equivalent. Similarly, Ancestry can miss small changes. I found the best technique was to use the FH list of individuals sorted in order of last updated as a crib sheet, to ensure all recent changes have been processed. On the positive side, all reviewed hints in Ancestry (which I always reject and never append to my data) have stayed reviewed, so in time I will clear the huge number of hints that inevitably come from a new upload!

My merging is also strictly one way only. New data are always entered directly into FH, so any corruption on moving between the three different applications will never contaminate my master data.

It sounds a bit convoluted, but it is definitely easier than remembering to continually update Ancestry in parallel, and enables me to have separate uploading sessions as and when I think there are sufficient changes to make it worthwhile. Crucially, no data are ever re-keyed, so while the odd data-point might be missing, it is always consistent with FH.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by tatewise »

Mark, that sounds extremely useful.
If I understand correctly, Python (https://www.python.org/) is free to use.
So might it be possible to publish your Python script for others to use?
Could it be tailored so users can easily customise it for their own needs?
I could add it to the how_to:exporting_gedcom_with_multimedia#export_to_website_without_sensitive_data|> Export to Website Without Sensitive Data techniques.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

Indeed - it is a free download and easy to install. If you take the option to add the Python folder to the Windows path, it can be run from the command line with just the name of the script file as an argument. In this case,

py D:\OneDrive\Documents\Python\FH2Ancestry.py

Put this line in a batch file and create a desktop shortcut to the batch file, and it runs just on double-clicking the icon, giving the following output:
1.PNG
1.PNG (6.9 KiB) Viewed 13607 times
The script file is just a plain text file, so easily edited. I have the following lines early on, so they can readily be customised with user's own filenames and choice of tags.

input_file = 'D:\OneDrive\Documents\Family Historian Projects\Draper\Draper.fh_data\Draper.ged'
output_file = 'D:\OneDrive\Documents\RootsMagic\Draper Ancestry Update.ged'

individual_tags = ('ADOP', 'BAPM', 'BIRT', 'BURI', 'CENS', 'CREM', 'DEAT', 'EMIG', 'NAME', 'NATU', 'PROB', 'RETI', 'OCCU', 'REFN', 'RESI', 'SEX', 'FAMC', 'FAMS')
family_tags = ('CHIL', 'HUSB', 'MARR', 'DIV', 'WIFE', 'REFN')
detail_tags = ('DATE', 'PLAC')

The file names are just string variables, and the tag lists are what Python calls tuples, which are basically just lists of values that remain fixed throughout the program.

It probably needs just a few more explanatory comments to be used by others, but happy to do that later this week and post back to this thread.
Mark Draper
User avatar
David2416
Superstar
Posts: 398
Joined: 12 Nov 2017 16:37
Family Historian: V7
Location: Suffolk UK

Re: Parallel Family Historian and Ancestry trees

Post by David2416 »

Hi, I have been following this conversation with great interest as I run several parallel trees. My main tree is in FH having migrated from FTM a while back. Once satisfied myself everything had migrated successfully I generated Gedcoms for creating new trees in Ancestry, FMP and MyHeritage with limited data for deceased ancestors (bmd and census/resi living were people were excluded).

This allows me to view the hints provided by each of those websites and accept/decline as appropriate and allow these to update their online tree. A tree is maintained on each of those websites which only includes information from those sites. The Ancestry tree is synchronised to a FTM tree which I allow to create it's own citations and download media. The MyHeritage tree synchronises with the MyHeritage Family Tree builder which creates source citations but does not download images (so far as I can see). Te FMP tree has now downloaded equivalent.

I manually enter any new data thus discovered into FH along with citations and linked media.

What has caught my interest is the discussion of using a Python script to populate a Roots Magic tree to sync and merge with Ancestry. I have downloaded and run the 64 bit Python installer (ignoring the offered fix to the 260 char path limit - don't know which program that might break).

David
User avatar
David2416
Superstar
Posts: 398
Joined: 12 Nov 2017 16:37
Family Historian: V7
Location: Suffolk UK

Re: Parallel Family Historian and Ancestry trees

Post by David2416 »

Mark, I am very interested in looking at your Python script. Would you be prepared to let me have a copy please?

Regards

David
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by tatewise »

Mark, says he will post his script later this week with just a few more explanatory comments.

I then intend to incorporate it into the [kb]|[/kb] for general use by anyone.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

Python script attached. Modifying it to use your own filenames and/or list of tags is simply a matter of changing the lines indicated within the file. In my experience, Python is relatively easy to read, so if you have any previous experience of other languages it should be possible to work out what the script is doing once it gets into the detail.

The forum software doesn't seem to allow .py as a file extension, so I have changed the filename to FH2Ancestry.txt. Change it back to .py (and whatever filename you like, of course) prior to running.
Attachments
FH2Ancestry.txt
(4.57 KiB) Downloaded 204 times
Mark Draper
User avatar
David2416
Superstar
Posts: 398
Joined: 12 Nov 2017 16:37
Family Historian: V7
Location: Suffolk UK

Re: Parallel Family Historian and Ancestry trees

Post by David2416 »

Thank you Mark, will look at later.
User avatar
David2416
Superstar
Posts: 398
Joined: 12 Nov 2017 16:37
Family Historian: V7
Location: Suffolk UK

Re: Parallel Family Historian and Ancestry trees

Post by David2416 »

Hi, have tried amending and running as instructed however am getting this error:

Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> C:\Users\David\Documents\Family Historian Projects\David\David.fh_data\FH2Ancestry.py
File "<stdin>", line 1
C:\Users\David\Documents\Family Historian Projects\David\David.fh_data\FH2Ancestry.py
^
SyntaxError: unexpected character after line continuation character
>>> "C:\Users\David\Desktop\Run anc exp py.bat"
File "<stdin>", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
>>>
Can anyone help with this. My amended file is attached.

TIA David
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Parallel Family Historian and Ancestry trees

Post by tatewise »

This is a bit of a guess, but is your Project GEDCOM file saved in UTF-16 or UTF-8 format?
The Python script expects UTF-16.
i.e. Have you set Tools > Preferences > File Load/Save > Save in UTF-8 file format?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
David2416
Superstar
Posts: 398
Joined: 12 Nov 2017 16:37
Family Historian: V7
Location: Suffolk UK

Re: Parallel Family Historian and Ancestry trees

Post by David2416 »

Thank, Um, no I haven't, file is just a copy the FH file.
1 CHAR UNICODE
think that's the default
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Parallel Family Historian and Ancestry trees

Post by Mark1834 »

David - two things spring to mind:

1. Your print statements are not correct. In modern versions of Python (i.e. 3.x), print() is a function with the variables or literals to be printed in parentheses - see other examples in the script. If you want to confirm the file names, simply put what you want to be displayed in parentheses.

2. This wouldn't produce the error you described, but I was getting some strange similar errors when running your modified script. How did you edit the original? It must be with a simple text editor (e.g. good old fashioned Notepad on a Windows system), not a program that could introduce spurious control codes or formatting commands, as these would corrupt the script.
Mark Draper
User avatar
David2416
Superstar
Posts: 398
Joined: 12 Nov 2017 16:37
Family Historian: V7
Location: Suffolk UK

Re: Parallel Family Historian and Ancestry trees

Post by David2416 »

Hi Mark, thanks for your reply. I used Notepad++, guess maybe that's the cause.
Will start from scratch using notepad.

(I added the print statements to get some output as the batch file opened the command box which closed straight away)
Post Reply