* Import from TMG with Sort Date in GEDCOM

Importing from another genealogy program? This is the place to ask. Questions about Exporting should go in the Exporting sub-forum of the General Usage forum.
avatar
E Wilcock
Megastar
Posts: 1181
Joined: 11 Oct 2014 07:59
Family Historian: V7
Location: London
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by E Wilcock »

tatewise wrote: my router died in a thunderstorm last night and I am using someone elses PC. Also OneDrive where the Plugin should be saved is misbehaving. So cannot communicate very often until next week.
Commiserations Mike. As a devoted computer user, I feel for you. But also hope you will enjoy a true holiday over the New Year. And not feel there is any hurry to complete the plug in.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

My new router has arrived, so I am back in operation again!

Try new attachment Order Facts by Sort Date Plugin Version 0.9 Date 02 Jan 2018 in later posting.

This version has several significant enhancements and some minor ones :-
  • The Result Set for GEDCOM Exceptions now lists FH Facts where the GEDCOM fact is missing
    ( as opposed to the GEDCOM fact not used by Project or the GEDCOM Sort Date missing ).
    This will most probably be because the TMG Fact is not Standard GEDCOM and is not using the custom EVEN tag.
    In the Result Set click on the Record Name or the Fact Summary to open the Property Box to see the relevant Fact.
  • The Result Set for Project Exceptions now only reports Different from Fact Date if the Sort Date is inconsistent with the actual Fact Date.
    Typically, it needs to be within a few days of the earliest and latest actual Date possible.
    It now takes account of Date Period and Date Range values including Quarter Dates.
    It allows a Sort Date to be upto 400 days earlier than a Before Date and 400 days later than an After Date.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by jbtapscott »

Mike - certainly finding that the changes to the Result Set are helpful in respect of the Different from Fact Date - fewer records reported (on my project) makes it easier to identify differences that really need investigation.

On the other hand, the change in respect of GEDCOM Fact is Missing means that I now have something in the region of 10,000+ extra records appearing - I am assuming this is because I have no TMG exported GEDCOM file but am applying the Plugin purely against my FH project.

Leaving the above aside, after running the latest Plugin I am seeing LMO Sort Dates already set in FH remaining unchanged (as expected) and, so far, LMO Sort Dates input by the Plugin in line with the various Rules you have previously published.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

Sorry Brent, forgot to check that GEDCOM Fact is Missing only applies when there is a GEDCOM File chosen.

Try attachment Order Facts by Sort Date Plugin Version 0.9 Date 02 Jan 2018.
[ Attachment deleted as superseded by Version 1.0 in Plugin Store ]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by jbtapscott »

Thanks Mike - that looks better!.

I have been through the Result Set after running the updated Plugin and all Individuals have LMO Sort Dates in line with what I expected based on your published rules. As I had already input manually some LMO Sort Dates (e.g. for undated Baptisms with dated Births, etc), I will run some queries during today to check that, for example, the LMO Sort Dates on all Birth records is before that on Baptisms, etc.

An area though where I do have queries is Family Records, and in particular the Synthetic Date calculations - some examples / queries shown below:

Birth Date of the individual is after the calculated LMO Sort Date for the Marriage:
Bryan Family record.PNG
Bryan Family record.PNG (7.02 KiB) Viewed 13679 times
As for the previous attachment, calculated LMO Sort Date is before the Birth dates of the two individuals:
Carmichale to Arnett Family record.PNG
Carmichale to Arnett Family record.PNG (9.05 KiB) Viewed 13679 times
A variation on the first two - neither Individuals has a Birth Date, so unsure how December 1901 is calculated (I have several Family records like this):
Briggs to Button Family record.PNG
Briggs to Button Family record.PNG (11.67 KiB) Viewed 13679 times
Other Family records looking good in terms of calculated LMO Sort Dates - e.g. Dated Marriage and undated Divorce have the LMO Sort Dates added that put the two events in the correct order.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

Thanks for that feedback Brent. It looks like the Synthetic Date calculation may need some refinement.

Currently it works through first 3 Children using the EstimatedBirthDate(%INDI%,"EARLIEST",9) function.
Then through both the Husband and Wife using the EstimatedDeathDate(%INDI%,"EARLIEST",9) function.
The Synthetic Date is set to the earliest of those estimates.

I know there are some issues with the FH estimates when certain date formats are used, and I have written corrections for them where used in other Plugins, so may have to do the same here. I had hoped that they would be accurate enough for the Sort Dates, but clearly not in some circumstances.

The estimation functions start by looking at the persons own key fact dates, but if none, then FH looks at direct relatives, and in the cases above includes up to 9 generations. FH also uses the Tools > Preferences > Estimates. So that is how FH gets the 1901 date. See the FH Help for those functions for more details.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by jbtapscott »

Thanks Mike - I'll look at the Help and work through the estimation process to see how these dates came about.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by jbtapscott »

I've worked through one of my 1901 records!.

In essence, and as per the attachment above, the couple have no Birth or Death events, and nor do their three children. I have no ancestors on the husbands side (Ern Briggs) while on the wife's side (Doris M Button) I have her parents dates as 1895-1953 (Father) and 1890-1930 (Mother). In turn, again, no details for the Fathers parents, but on the Mothers side 1863-1930 (Grandfather) and 1862-1951 (Grandmother). These grandparents had four other children (who I have Birth / Deaths events for) who all lived well in to the mid 1900's.

I don't know the algorithms FH uses within the functions, but I would have thought that, at a base level, if the parents of Doris M Button were born in 1895 and 1890, she would not have got married in 1901 !! Looking at the results of the Relations query, I also have multiple cousins, etc., so they may have come in to the FH calculations.

Whatever, I don't propose spending more time on this as all those I highlighted are not directly linked to me so I will probably go through the dozen or so records and manually add the LMO Sort Dates just to ensure some realistic sorting (just in case I find other data for these people at a later date).

I will, of course, though continue working though other records checking calculated LMO Sort Dates and let you know any issues I may find.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

Remember, the calculation is NOT currently estimating the Marriage Date.
It is setting a default Sort Date that should position the Marriage (a Family fact) sensibly amongst the spouse's Individual facts, ideally between Birth and Death, particularly when migrated to other products such as GedSite.
Currently, I accept that ideal is not always being achieved.

Taking the case of Doris M Button, it is not just Birth and Death event Dates that are considered, but also Baptism, Christening, Burial & Cremation. However, I suspect her immediate family have none of those either.

The exact algorithm that FH uses is not published, but this is my guess of how 31 Dec 1901 arises.
Since the immediate family have no dates, then one generation earlier will be investigated.
Her mother was Born in 1890, so earliest date is 1 Jan 1890.
If your Tools > Preferences > Estimates have Installation Settings then Mother's age at childbirth - Minimum is 12.
So earliest Birth and thus earliest Death for Doris M Button is 1 Jan 1890 + 12 = 1 Jan 1902.
The Plugin adjusts that earliest date to one day earlier, thus 31 Dec 1901 QED.

However, in this case, assuming that Doris M Button and Ern Briggs have no Individual facts recorded at all, the Marriage Sort Date is probably irrelevant as it is their only fact, so its sort position will be first, middle & last. An exception would be if that Marriage fact were to appear amongst the facts of other relatives in the GedSite displays.

So to recap, the Marriage Sort Date is only significant if that Marriage fact is shown sorted amongst other facts.
The synthetic Marriage Sort Date is calculated from the earliest Birth of Children or earliest Death of Spouses.
The objective is to sort the Marriage between the Birth and Death of the Spouses.
That objective is not currently being achieved in some scenarios such as the Birth After dates you posted.

So maybe the Plugin needs to take spouse Birth Dates into account too?
However, the bigger the calculation the slower the Plugin runs, as it is currently performed for every Family record.
Although it may be possible to postpone the calculation until needed for an undated Marriage or other Family fact.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Linda Reinfeld
Famous
Posts: 124
Joined: 04 Nov 2014 17:34
Family Historian: V7

Re: Import from TMG with Sort Date in GEDCOM

Post by Linda Reinfeld »

Hi Mike

I was finally able to run the latest version of your plug-in, and I was able to complete the process - previously it had aborted due to errors. So that's good news.

I have 40,000+ people. Unfortunately it takes me about 3 hours to run the Gedcom export (in order to match the dates). And I think the process does a 'hiccup' every once in a while because I got 46 messages Gedcom fact is missing. Not really a problem because the data was already in FH. I got 42 Gedcom fact not used by project, some of which were birth & death facts, which were in the project. I think the problem was with the old-style dates, such as Jan 1705/06 and 3 Feb 1707/08. I don't have a problem with this either because the data was in FH - I mention it only in that you might want to look at the plug-in.

I am OK with the plug-in now, I am able to run it to completion.

Thank you -

Linda Reinfeld
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

I don't think those old-style 'double-dates' are a problem, as they are perfectly valid.

Is it possible that any of the 46 GEDCOM fact is missing pair with the 42 GEDCOM fact not used by Project?
If those facts pair up, then an analysis may explain why they don't match exactly.

Remember that clicking on any Result Set column heading will sort in that order.
e.g. Click on Rec Id column to bring messages for the same record together.

Please post a screenshot of some Result Set examples.

BTW: You don't need to export the TMG GEDCOM every time you want to run the Plugin.
Just keep a clean copy of the original TMG directly imported Project with a GEDCOM Only Backup if necessary.
Don't overlook using Edit > Undo Plugin Updates to gte back to pre-Plugin state.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

Brent, I have had a brief opportunity to look at the Plugin again.
It now only calculates a Synthetic Date for Family records if it needs to because no other Sort Date is available.
This allows the calculation to be more extensive without significantly affecting Plugin run time.

So now it determines the earliest of any estimated Child Birth or Spouse Death date, and then determines the latest of that date or any Spouse Birth date + 16 years to come of age.
It would be possible to use earliest, middle, or latest estimated dates in those, if they proved to be better, and the current 9 generation search to form the estimates could be reduced if that was more reliable.

Try attachment Order Facts by Sort Date Plugin Version 0.9 Date 27 Jan 2018.
[ Attachment deleted as superseded by Version 1.0 in Plugin Store ]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by jbtapscott »

Thanks Mike - I'm away visiting children / grandchildren at the moment but will have a look as soon as I get home next week.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

Is this Plugin ready to be published in the Plugin Store?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
E Wilcock
Megastar
Posts: 1181
Joined: 11 Oct 2014 07:59
Family Historian: V7
Location: London
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by E Wilcock »

Coincidence - as I came here this morning to ask something similar.
A missing marriage (and missing gt gt gt grandmother) which defeated my poor father, myself and the professional genealogists we both employed - has suddenly turned up on line.

That Project is still in TMG. I can use it to export a GEDCOM.

I was going to ask you Mike whether you thought the plug in was now so good that I should bring this project into fh, even tho I had sworn not to bring in any more until Calico Pie allowed import of the sort dates?

I know you have probably explained this before but please dont be cross. I am confused about when and how to use this plug in and so far have not used it on my real Projects (only on copies).

As I will be importing this Project from TMG, the first step is to run it using a TMG exported Gedcom. That will create (I believe) a fh file that has the extra Tag of sort dates.
But I wont need those sort dates until I want to export to Gedsite. Which may not be for a year or two.
What happens to data entered in fh during that interval?
Does one run the plug in again, just before the export? But this time not referring to the TMG Gedcom?

And as a supplementary to this - should I now run the plug in on the top copies of the 4 family trees that I had already irrevocably imported?
Or should I copy them yet again and carry out the same checks - which I will have to look up, to remind myself.
I am deep in other work at the moment and this isnt a good time to need to think about all this importing again. Which is why i did not import this project yesterday.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

The process for migrating a Project from TMG to FH with Sort Dates is as follows:
  1. Direct import the TMG Project into FH via File > Project Window > New Project > Import from other family tree file as you usually have done before.
  2. In TMG export a GEDCOM file including its Sort Dates.
  3. In FH run the Order Facts by Sort Date Plugin and choose that GEDCOM file to assign its Sort Dates.
Thereafter, you can run the Order Facts by Sort Date Plugin whenever you like without choosing a GEDCOM file.
That will allocate Sort Dates to any new Facts that have been added, and arrange them all into Sort Date order.

As for the four Projects you have already irrevocably imported, you have options.
  1. Use steps 2. & 3. above on existing Projects to assign Sort Dates and review the results, but if unsatisfactory use B. below.
  2. Migrate the Projects again as above, and apply same checks as when first imported - which may be difficult to remember.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
E Wilcock
Megastar
Posts: 1181
Joined: 11 Oct 2014 07:59
Family Historian: V7
Location: London
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by E Wilcock »

First, I much appreciate the clear instructions now in the UKB on importing from TMG. I think someone has edited them and simplified. Thank you so much.

But one question arises.
My first imports from TMG were done after using John Cardinal's TMG utilities to alter some Event names in TMG.

Later, I was told about the fh Alter any fact plug in and was advised to leave any alterations till after import. This seemed sensible as if things went wrong, Mike could help me..

However for the sort date plug in it becomes necessary to look closely at the TMG export of any custom Tags.
And the Plug in is working from a gedcom exported from TMG.

I am thinking it may be better to revert to my original method and do the tidying or standardising of custom Events and tags before opening the file in fh. And before exporting the Gedcom - so it all matches up.

Any comments would be welcome before I revise my instructions to myself.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

The only significant changes made to the how_to:import_from_tmg|> Import from The Master Genealogist (TMG) page in the last year are my addition of the Import Fact Sort Dates section last December. (I know that from the KB Revision history.)

Yes, the original advice was to wait until imported into FH before tidying up.
BUT the correlation of Fact Tags is critical to assigning Sort Dates correctly, so if that feature is required, then some adjustment to Tags in TMG may be necessary, as described in the Import Fact Sort Dates section.

As we discovered with your trial imports, the TMG Tag adjustment, direct import Project into FH, and transfer GEDCOM using Order Facts by Sort Date Plugin may need repeating a few times until all the Tags correlate correctly.

Apart from that, I think all other tidying can be postponed until after migrating to FH.

See also the plugins:help:order_facts_by_sort_date:order_facts_by_sort_date|> Order Facts by Sort Date Plugin that provides general advice for the Plugin itself, and will be updated at the time of publishing the Plugin.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
E Wilcock
Megastar
Posts: 1181
Joined: 11 Oct 2014 07:59
Family Historian: V7
Location: London
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by E Wilcock »

tatewise wrote:The only significant changes made to the how_to:import_from_tmg|> Import from The Master Genealogist (TMG) page in the last year are my addition of the Import Fact Sort Dates section last December. (I know that from the KB Revision history.)
If that is the case Mike there may be a lesson here.
That as a newcomer I was totally confused by three different sources of information - turned up when I searched - that provided by Calico Pie, the User Knowledge and long threads on this forum.

Then there is a possibility that I didnt understand the fh terminology at that point. Not that I always do now. But I can better understand what I am reading. And I have a rough idea how to use fh.

I also know (in case any new user is reading this) that it is not a total disaster if you just click on a TMG tree and open it in fh - as that is what I did with several trees in the early days. Because at that time there was no warning about flags and sort dates and witnesses.
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by jbtapscott »

Sorry Mike - funerals, family illnesses and preparations for a drive to Greece (completed today!) have rather got in the way of things.

Yes, the last check I did I found that the Sort Dates applied using the latest plugin did give a better result (on the examples I raised here). Not perfect, but given how little date information was available on the families I believe it is as near as I'm going to get - the main thing is that the calculated Sort Dates now do not get out of kilter in relation to the few "real" dates on the various individuals in the families.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

The Order Facts by Sort Date Plugin Version 1.0 Date 14 Mar 2018 is now published in the Plugin Store and its plugins:help:order_facts_by_sort_date:order_facts_by_sort_date|> Order Facts by Sort Date Plugin Help & Advice page has been updated.

The prototypes attached to this thread will be deleted soon.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
FH_gather
Newbie
Posts: 2
Joined: 26 Nov 2016 16:49
Family Historian: V6.2

Re: Import from TMG with Sort Date in GEDCOM

Post by FH_gather »

Attempted today to use "Order Facts by Sort Date" plugin version 1.0 dated 14 March 2018.
Using FH 6.2.6 in Windows 7 64-bit. FH project is imported directly from TMG and GEDCOM file as exported from same TMG database.
Plugin produces error ".failed to complete [string "C:ProgramDataCalico PieFamilyHistorian..."]:3308:attempt to index global 'DicGed' (a nil value)."
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Import from TMG with Sort Date in GEDCOM

Post by tatewise »

Welcome to your first posting on FHUG John.

You will find a corrected version to try out in Order Facts by Sort Date Plugin Problem (15805).
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply