* Import from TMG with Sort Date in GEDCOM

[tatewise]

This discussion proposal is a continuation of these other threads:
Import from TMG Summary (15482)
Importing from TMG - no sort dates in FH (15127)

I have found TMG™ GEDCOM Export Issues that says:

Sort dates
Sort dates are an advanced TMG feature. While they always exist within TMG, they can only be explicitly entered/modified when in TMG’s Advanced (versus Beginner) edit mode. This enhanced export option will export these sort date values as a custom GEDCOM tag type following any tag’s standard DATE tag or any custom _DATE tag added by this enhanced export option.
2 DATE date
2 _SDATE sort date
or
2 _DATE date
2 _SDATE sort date

So after a direct import from TMG into a new FH Project, and before doing anything else, an Import Gedcom Sort Date Plugin could read through an exported TMG GEDCOM file and add each _SDATE sort date into each Fact within the new FH Project. Thereafter, an Order Facts by Sort Date Plugin could auto-sort facts into Sort Date order.

What format should this Sort Date use within the FH Project?
There are a number of candidates:

1. _SDATE tag is preferable, BUT is an invalid FH tag, so a Plugin cannot add that without extreme difficulty.
   (That difficulty arises again whenever a new Custom Fact is defined by the user and needs an _SDATE tag.)
2. A labelled Note such as [[ Sort Date: August 1900 ]] has been introduced by Export Gedcom File Plugin, but is a bit untidy, has possible issues with Reports, and the Date formats are not checked when manually entered.
3. A new idea is to use a Local Multimedia Object (LMO) with only a Title of Sort Date and its Date field.
   Right-click on fact and choose Add Multimedia Object > Add LMO to this Record, enter Title and right-click to add Date, then optionally delete the Format & File fields.
   This has the following advantages:
   • Fully GEDCOM compliant, and enforces all Date formats.
   • Easily added on All tab, and looks neatly like a Sort Date.
   • Does NOT appear in any Reports (except Record Detail Reports).
   • LMO are rarely used, but the Sort Date instance is easily differentiated from any real LMO.
   • Easily managed by Plugins such as Export Gedcom File and proposed Import Gedcom Sort Date and Order Facts by Sort Date, and can be transformed into _SDATE tags if Calico Pie implements them.
     These Plugins could 'tidy' malformed LMO where the Title was misspelt or missing, and remove empty Format & File fields.
     

     

     LMO Sort Date.png (21.06 KiB) Viewed 15720 times

Feedback on these concepts would be very welcome.

[ColeValleyGirl]

Mike, as you say, a tag would be ideal but I think the LMO approach is the best alternative, working as you describe it. (Shame I spent a lot of time yesterday typing in [[Sort date: ...]] notes but perhaps there'll be a way to transform those...)

[davidm_uk]

Just a thought ( as I don't have any TMG files) but how might this sort date method interact with FH's automatic sort functions, or are they mutually exclusive?

[tatewise]

To get the _SDATE tag to work requires one dummy Individual record and one dummy Family record with EVERY Fact and its _SDATE tag included. They need to be updated every time a new Custom Fact is created. They can only be created by directly editing the GEDCOM file either manually or with a Plugin. That is why it is so difficult, UNLESS Calico Pie were to update FH to at least allow _SDATE tags with a Date value, even if FH provided no other features the Plugins could do the rest.

Don't worry about creating [[Sort date: ...]] notes, because the proposed Plugins would offer to convert them to the LMO format.

The Plugins and FH commands such as Tools > Re-order Out-of-Sequence Data and Edit > Re-order Events by Date would be generally mutually exclusive. BUT it would usually only impact Facts with no Date whose Normal Time Frame setting does not sort the same as the Sort Date, or adjacent Facts with same Date whose order is set by the Sort Date. So for example, where a Family of Individuals has all uniquely dated Facts, the Edit > Re-order Events by Date command or the Property Box cog Menu > Sort Family and Events in Date Order will behave the same as the Plugins, because the Sort Date is the same as the Fact Date field. However, it would be just as quick to invoke Tools > Order Facts by Sort Date Plugin as Tools > Re-order Out-of-Sequence Data.

What is needed now is details of how the Record Id in the TMG GEDCOM relate to the Record Id in the directly imported new FH Project.

[ColeValleyGirl]

I agree, _SDATE is too cumbersome (for users) to pursue without support from Calico Pie.

[E Wilcock]

Mike - I am sorry. I have lost you.
You asked whether a TMG Gedcom included sort dates.

I went to TMG and exported a Gedcom and it didn't. I have just checked and done a text search for _DATE - and there is nothing.

It is true that the TMG Export Wizard might offer a chance to configure the Export - but I don't know how to do that.

[tatewise]

Evelyn, please read the quote I posted earlier that talks about "TMG’s Advanced (versus Beginner) edit mode" and says that "enhanced export option will export these sort date values". Are you using that enhanced mode?

BTW: I presume you were searching for _SDATE and not _DATE as you posted ?

[E Wilcock]

Mike _ I always used TMG in advanced mode.
The other is very basic. But as far as I can see the advanced Mode option in TMG applies only to data entry.

And the File export Wizard gives no directions.
That is why I asked whether one should not go to a TMG Forum and ask one of the TMG experts there. Which I am not.
I rush to bring a tree into fh - when someone asks me about it - and intend to tidy it but am always just too busy.

And no - sorry- I did the wrong text search, but have just searched again and it isnt there.

Now it could well be that I have done something else wrong - I blunder about with software and cannot remember anything.

Consulting the TMG experts:
Terry Reigel's invaluable "A Primer for The Master Genealogist" 2008 has a section on exporting GEDCOM which warns about the limitations of GEDCOM export mentioning the things that crop up on this forum - events with two unrelated principals and witnesses etc.

The more detailed book by Lee Hoffman et al "Getting the Most out of TMG" 2003, has a chapter on GEDCOM contributed by Jim Byram who is still an active helper on the forums. He takes you through the many options offered by a custom (rather than default) GEDCOM export.
Under Tagtypes he says you can select None, All or Selected but he says that the default is All.

However this book may apply to an earlier version of TMG because when I went into the wizard to set up a custom export I saw there was an option to select enhanced GEDCOM TAG export and when I checked this I got a Gedcom with sort dates. (Success)

I was for a time totally unable to bring up this option screen again and the GEDcom default for export does indeed (as Jim stated) show all events as the Default.

Confusing as it is I think one needs in the Export wizard to opt to Save as a new configuration for GEDCOM and then one will get the Gedcom with sort dates. And once I save it as Export for fh - one might never set eyes on those options ever again.

I apologise that I told you wrongly the sort dates were missing - and to Jim Byram who was my guide thro TMG - but I really dont know what I am talking about.
None the less, I hope this may be useful to you. Since the advice on both fh and RM is to import TMG Projects direct (not via Gedcom) Gedcom options are not something of which I have any recent experience.

[tatewise]

Thank you Evelyn, that suggests we need some Knowledge Base advice to ensure _SDATE are included in TMG GEDCOM.

Can you give any indication of how Individual & Family GEDCOM Record Id relate to the FH Project Record Id.
The GEDCOM Record Id have the format 0 @I1234@ and 0 @F1234@ where 1234 is the Id.
The FH Project Record Id are listed in the Records Window in Individuals and Families tabs.

[E Wilcock]

I am not sure that I understand what you are asking Mike.

Are you asking me to look at person numbers in the GEDCOM in Notepad and then open the same GEDCom in fh and tell you the numbers there?

There is already a slight complication in the Gedcom as in TMG that major family tree included two other data sets within the same project. My direct import to fh excluded those. I shall need to set them up as separate fh projects at some point.

0 @I1@ INDI

but if I sort the list of individuals in fh by record number they look the same. I mean the same lady is record ID 1. And her husband record 2.

However, I did get a massive list of errors on opening it in fh - many non standard Events saved in an accompanying document. Again the rejections could be due to my user error or failure to make the right choices for the custom export. But the sort dates are non standard and I assume that they have all been rejected by fh on those grounds.
I am just not expert enough to steer you through this.

My bringing my TMG projects into fh depends on the direct import.
It has been my view that it is up to Calico Pie to tweak the direct import.

Jim Byram himself recorded how he moved to RM in the early days of switch over by adjusting and using the GEDCOM method. The average genealogist is probably not as computer literate. I myself did not feel up to doing this editing with multiple projects.
One of the attractions of fh is that one doesnt have to struggle with a relational database. So I think keep it all as simple as possible.

[tatewise]

Evelyn, no you have misunderstood, and my explanation was a bit too brief.
It relates to my earlier proposal that has the following steps.

1. Directly import a TMG Project into a new FH Project.
2. Export a TMG GEDCOM for the same TMG Project.
3. A Plugin runs in the new FH Project and reads Sort Dates in from the TMG GEDCOM file.

For that to work the Record Id in the new FH Project and the TMG GEDCOM file need to be the same.
So for your recently imported Project (that has no Sort Dates) I was asking you to perform that comparison.
The FH Project Record Id are listed in the Records Window in the Individuals and Families tabs.
The TMG GEDCOM Record Id have the format 0 @I1234@ INDI and 0 @F1234@ FAM where 1234 is the Id.
So the same people & families should have the same Record Id in both - I hope.

[DonF]

TMG WILL export Sort Dates in GEDCOM.
However, this function was added in TMG v9.04, so if you are using an earlier version you will not have access to the required option in the Export function.

Go to TMG's File -> Export
In the Export Wizard, Step 2, choose GEDCOM
Work through the other steps until you get to Step 5.
There you will find a check box for 'Enhanced GEDCOM tag export (see Help)'
If you tick this box (and have selected GEDCOM 5.5 format in the drop-down box above), then you will get Sort Dates exported in the format Mike showed earlier. And a whole lot of other stuff, as explained in the Help.

This has nothing to do with TMG's Basic/Advanced mode.

Don

[E Wilcock]

Thank you Don.
I am using the latest version of TMG (and of fh too).
You confirm what I found. And my assumption that the GEDCOM choices had been introduced subsequent to the writing of the two guide books I dutifully consulted.

And by the way, I did not get the choices until I opted to create a new customised Export.
And it does indeed say see Help but I found nothing when I clicked that! My problems finding the relevant help seem to span all software.

Mike - No need to apologise. I will clear my mind and follow these instructions. May be this evening.

[E Wilcock]

Mike I have now checked the numbers by the people.
I reported before that individuals nos 1 and 2 in the TMG generated gedcom had the same numbers as in the list of individuals in fh.
I have now checked a number chosen at random Indiv no 80 and he too has the same number in fh.

I havent yet checked the family numbers - I think I know how to check the family numbers in a gedcom (lower down) but dont know how to check family numbers in fh.

I dont want to be negative but I should mention again that indiv numbers in a GEDCOM exported by TMG may be (and in this case are) preceded by a number indicating the data set from which each individual comes.

TMG allowed one to hold subsiduary projects or datasets within an oversall Project. My mother's family tree project still has two of these. One is a locked tree sent me by another researcher, relating to a forebear so long ago that I did not want to include it in my own tree. The other I created myself when I was studying my ancestors' education as doctors in the 19th - It lists the background and careers of their fellow medical students and medical colleagues, many of whom are represented in the old family photo albums.

When one exports a gedcom from TMG one selects just one data set - but, although only one project is exported, the indiv ref numbers are none the less preceded by a number I. Indiv 80 in my fh Project is Indiv I80 on the TMG Gedcom.
Individuals on the list I was sent by another researcher are prefixed 3 and the medical list by 4. This shows that at some point I had a further list, number 2 that has been deleted.
Looking at a more recent TMG Project started about 20 years ago, I see to my surprise that it too includes several data sets.

Data sets in TMG allowed one to collect data on e.g. a common surname, or researched by other people and hold that data compartmentalised so that it was not yet amalgamated with one's main tree. In fh I have used a named list to throw in lists of e.g. possible naturalisations.
But the existence of all these data sets is easilly overlooked when moving from TMG to other software. I keep TMG for back reference. There is no way I would every copy everything across.

[tatewise]

You check Family Record Id in the FH Records Window in the Families tab.
If that tab is not shown then use View > Record Lists > Families.

Don't worry too much about the leading dataset number as the Plugin can be designed to sort that out.

I presume there cannot be more than 9 datasets in one TMG Project.
Otherwise, there could be a conflict with the numbers.
e.g.
1234 could mean dataset = 1 and Id = 234, or dataset = 12 and Id = 34, or dataset = 123 and Id = 4

[E Wilcock]

Mike - I appreciate your patience.

I have found the Family record List in fh, as guided by you.
I fear that the answer is not what we were hoping for -

My Individual no. 80 appears in the marriage record ID16 with his wife Indiv 14 And his son Indiv 81.

But when I look at the Gedcom from TMG (using Notepad)
The F links below his death are
1 FAMC @F70@
1 FAMS @F13@

His wife (who married twice) has family records
1 FAMC @F10@
1 FAMS @F12@
1 FAMS @F13@

In fh, the family (marriage) record ID 13 is for a different couple entirely.
Her marriage to Indiv 80 is fh marriage record 16.

This could still be down to my user error.
I may well be confusing marriage records with GEDCOM FAM family records?
TMG did not use family records (events) which I met for the first time in fh.

So if you give me some further instructions I will look again. Or paste you in a snipped image.

[tatewise]

Your analysis appears to be valid, and it is not ideal.
However, if the Individual Record Id are correct the Plugin should still be able to match up the Family records.
Just needs a bit more cunning coding that associates via the Husband and Wife pairings.

[DonF]

Just to close off some items above and add some more clarity.

The TMG Help explains what extra things you get under the Enhanced GEDCOM option in the Help page labelled 'GEDCOM Export' - just scroll down until you get the bit about Enhanced GEDCOM.
This is certainly not discussed in Lee Hoffman's reference book as that was written at TMG 5.05 time (2003) and this feature (as I said) came into being in v9.04 (2014).

I don't think there is any limit on the number of datasets you can have within a single TMG project. It's not a widely used feature though. (Correction: there is a 2GB limit on any TABLE in the TMG/Visual Foxpro database. This would be the eventual limit. Usually the Name table grows the fastest and we have evidence it reaches the 2GB limit at about 1 million names.)

People IDs when there are multiple datasets within a project show as (d:nnn) where d is the Dataset ID and nnn the person ID, so there is no possible numbering confusion. When exported to GEDCOM the dataset ID is dropped as otherwise the ID would not be GEDCOM-compatible.
Evelyn is miss-reading the GEDCOM individual id as 180, thinking it is dataset 1, individual 80, when it is I80 (capital 'I' for individual).

Don

[tatewise]

Thank you Don.
I was coming to the same conclusion about mistaking prefix letter I (for Individual) and thinking it was a one 1.
That despite me clearly giving the format @I1234@ and @F1234@ where every Id has a prefix letter I for Individual, F for Family, S for Source, etc.

[tatewise]

I am quite advanced in developing an Order Facts by Sort Date Plugin.
Your feedback on its features will be welcome.
It combines the repeatable sorting of Facts into LMO Sort Date order with the initial creation of LMO Sort Dates.

It starts by checking how many Facts have existing LMO Sort Dates.
If there are fewer than 9 then it offers to create those LMO Sort Dates from various sources.
Otherwise, it goes ahead and reorganises the Facts into LMO Sort Date order.
If while doing that it finds a missing LMO Sort Date it will create one from the sources below except 2.

When it offers to create LMO Sort Dates one option is to choose a GEDCOM File such as from TMG.
Then it will create each LMO Sort Date according to the following precedence rules:

1. If an LMO Sort Date already exists then that will be retained.
2. If a GEDCOM File was chosen, then a _SDATE Sort Date will be used.
3. If a Note labelled "Sort Date:" exists then that will be used and "Sort Date:" deleted.
4. If a fact Date exists then that will be used, unless same Date as previous fact.
5. Undated facts, or same Date as previous fact, use the previous fact Sort Date + 1 day.
6. Initial undated facts will use 1 Jan 1000 or 1 Jan 3000 for Individual or Family facts respectively.

I have discovered that any Date with a missing Month or Day when sorted by FH uses December or last day on month, so a Sort Date created from such a partial Date uses the same philosophy.

Remember that the true Fact order can only be seen in the All tab.
The Facts tab will override that order where Fact Dates &/or Normal Time Frame rules demand it.

[E Wilcock]

Thank you Don.
I was coming to the same conclusion about mistaking prefix letter I (for Individual) and thinking it was a one 1.
That despite me clearly giving the format @I1234@ and @F1234@ where every Id has a prefix letter I for Individual, F for Family, S for Source, etc.

It is completely true that I did not understand the significance of what Mike posted. I dont understand all these conventions and abbreviations and thought it was just an instruction to look for @ - in the gedcom another mysterious symbol used in a variety of ways.

Many of your genealogy customers are elderly. And we may find time for genealogy when there are other more pressing matters occupying our minds. I was at home and checking the family archive - only because of family illness. So let us be plain. I have not been at my brightest. I do not know what LMO means.
I do not know why the GEDCOM I exported from TMG has so many sort dates - possibly for every event - when I have no memory of entering them.

I use all genealogy software as my working notebook. I throw information into it and trust that it is findable. I have only once in a long lifetime of family History been asked to produce a full narrative website record (TMG and Second Site) for a particular family reunion. And the sentences and paragraph breaks for that carefully constructed narrative, did not import to fh.

But if you want to test import and sort dates from TMG, Mike, you need to find a genealogist with a tidier brain than mine. And more exact data entry.
There is a TMG sample project for instance and I have just checked for undated facts ie those that might have sort dates. I also see that even dated facts have their sort dates too. At some point someone might try to open that in fh? But I myself have returned to the archive.

[tatewise]

Evelyn, despite what you may think your feedback is helpful.

LMO was explained in point 3. of my first posting as a Local Multimedia Object (LMO).
See that first posting for a more detailed explanation.

You are correct that every TMG fact has a _SDATE Sort Date that defaults to the actual fact Date.
That will be replicated in FH but using an LMO Sort Date.

I would appreciate your help in testing the import of Sort Dates from TMG into FH.
Your possibly less than perfect data entry will be a better test than a perfectly manicured Project.

[tatewise]

Attached is prototype Order Facts by Sort Date Plugin Version 0.3 Date 8 Dec 2017.
[ Attachment deleted as superseded by many later versions. ]

It implements the features I posted yesterday.

As usual it is advisable to work with a Copy of your Project whilst testing this Plugin.

Otherwise, ensure you have at least a File > Backup/Restore > Small Backup although Snapshots should be available.

Remember you can use Edit > Undo Plugin Updates to reverse its changes, providing that is done BEFORE closing FH.

BTW:
This version as yet has no help page associated with its Help & Advice button.
There are no progress bars, so large Projects may take a while with nothing apparently happening.
Perhaps a Result Set of newly created LMO Sort Dates and their derivation would be useful (except in the initial creation mode when their would be far too many). So when new Facts are entered without their LMO Fact Date then the Plugin would list how their LMO Fact Date was derived and the user could adjust it if necessary.

[jbtapscott]

I have run the plugin against a copy of my main project - 4000+ records with new "sort dates" inserted in the blink of an eye!. I moved over from TMG too long ago to have the option to use a TMG Gedcom export so your Rule # 2 was not utilised in the processing.

The subsequent output though does highlight some issues (for me):

#1 I have some Birth facts with a Date set to a "before" year - eg "BEF 1670". The sort date generated is "31 December 1670" whereas I think it should be referencing 1669. I appreciate you commented in an earlier post about the way FH sorts year only facts but as the Plugin is actually generating data I would question whether following FH in this instance is correct. If I subsequently added a fact for, say, Baptism with a full date of "31 March 1670" I would need to (remember to) change the Sort Date on the Birth fact to a date before 31 March if I wanted output sorted correctly.
#2 I wonder if there is a need to include an option that gives the user the ability to decide if a newly generated sort date takes the 31st or the 1st if there is no full date on the fact. By way of example, I have a Baptism fact with a date of "18 September 1734" and a Birth fact with a date of "1734". The latter is given a Sort Date of "31 December 1734" which, in sorted output would put it after the Baptism fact. Now perhaps this highlights "poor" input methodology on my part by not setting the Birth date to "BEF 18 September 1734", but I originally took the view that in narrative style reporting (which is what I use almost exclusively), it is pretty obvious to the reader that a child was born before his baptism and so therefore I don't need the "BEF" prefix. A similar sort of issue could arise with Death and Burial facts where the death is recorded as "March 1834" and the burial as "21 March 1834" - the Sort Date for the former is set to "31 March 1834"

As I say, I suspect some of these "issues" could be down to poor methodology on my part so it perhaps highlights the need to carefully review data before running the plugin on anything but a copy of my main projects.

I'll carry on reviewing data during the next couple of days to check that output is in line with the "Rules" in your earlier posts - so far, all is looking good.

[tatewise]

Thank you for looking at the Plugin concepts Brent.

#1 regarding BEFore Dates is a good point and I will look into that.
I need to consider Before <year>, and Before <month> <year>, and Before <day> <month> <year>.

#2 is more complex.
I suspect a global option for 1st of month & January versus last of month & December may not satisfy all cases.
One idea could be to use the Fact Normal Time Frame to influence the Sort Date with respect to other Facts.
So a Birth Sort Date always precedes Baptism Sort Date and Death Sort Date always precedes Burial Sort Date in much the same way as FH sorts Facts.
Unfortunately, Plugins cannot currently determine each Fact Normal Time Frame very easily.

Whatever the Plugin does to set LMO Sort Dates, whenever the user spots an anomaly, they can edit the LMO Sort Date.