* Import from TMG with Sort Date in GEDCOM

[tatewise]

Evelyn, thank you sending sample Projects and Gedcom by Email.
They have provided lots of useful examples that led to several Plugin amendments.
That includes fixing the Family Rec Id matching problems, due to CHILdren in a different order in TMG than FH.

One problem has arisen with Place name matching that is extremely difficult to fix in the Plugin.
With direct import from TMG into FH all Place fields are included: Addressee, Detail, City, County, State, Country.
But with Export Gedcom from TMG the Place fields omit Addressee and Detail.
When the Plugin tries to match Facts that use those two fields it fails and the _SDATE is not imported.

In the Export Gedcom extended Step 7 options under Exclusions there seem to be Place settings.
Evelyn, can those be changed to include Addressee and Detail along with all the other Place fields?
If that works I will update the KB advice.

Try attached Order Facts by Sort Date Plugin Version 0.9 Date 23 Dec 2017.
[ Attachment deleted as superseded by many later versions. ]

With the Sample project it successfully imports 196 of 206 Sort Dates from the GEDCOM File.
With the Sample UK project it successfully imports 568 of 575 Sort Dates from GEDCOM File and if that Place change on Export Gedcom is successful then it should import another 3.
I will investigate the few remaining cases later.

[E Wilcock]

Mike -

That is not the case. it is true of the gedcoms I have used and have sent you because those options were not ticked.
Your advice to me was to leave what seemed to be the default export for enhanced tags (i.e. sort dates) and not alter anything else in the export. And this is not the only screen offering choices.

TMGExport.JPG (43.79 KiB) Viewed 14108 times

[E Wilcock]

TMG Screen 1 choices to export gedcom
(Screen 2 has all options checked to use ALL so I am not posting it)

TMGExportscreen1.JPG (37.83 KiB) Viewed 14109 times

[E Wilcock]

I will not test the latest version till we have this Export sorted. And you have told me what you would like and I have sent you fresh Gedcoms, following your directions.

If your eventual Plug in is to help TMG people and using a TMG Gedcom, you will in any case need to post detailed instructions for how that Gedcom is to be created.

[tatewise]

I have analysed the few outstanding issues.
You were correct to keep default Enhanced Export Gedcom settings, but it seems they do need changing to be consistent with the TMG direct import to FH, and yes I will provide detailed instructions in the KB.
Evelyn, please send Gedcom for both Sample & Sample UK after making the Step 5 & Step 7 changes below, but only when you have time.

Step 5: Option screen 1
Ensure Suppress details for living people is NOT ticked, i.e include everyone living or dead.

That should fix a problem with several Facts missing from Gedcom file for a few people.

Step 7: Option screen 3 ~ Places
Ensure Place Levels are ticked for Addressee and Detail plus the others down to Country.

That should fix the problem with Place names not matching.

Other missing data
Two Facts in the Sample project have no _SDATE recorded, so are not imported via Gedcom file.
i.e.
Sarah Martha BUCHANAN [56] Residence in Hickory, Pennsylvania
Samuel Lee ALEXANDER [3] & Sally JONES [64] Marriage
Nothing can be done about that, other than assigning _SDATE values.
But perhaps these should be listed in the Result Set so the user can deal with them.

In the Sample UK project the Misc fact is not exported in the Gedcom file.
i.e.
Thomas LOWE [40]
Misc 10 Oct 1845 in Vancouver County, Oregon
Misc 05 Apr 1849 in Clackamas County, Oregon
Misc 26 Sep 1850
Does any TMG expert know why they are not exported? Is it something to do with the Tag definition?

[E Wilcock]

Two Facts in the Sample project have no _SDATE recorded, so are not imported via Gedcom file.
i.e.
Sarah Martha BUCHANAN [56] Residence in Hickory, Pennsylvania

I am reading the project in TMG. correct there is no date and no sort date. The tags are in order Residence, Married name, (similarly undated) Marriage and death,

Samuel Lee ALEXANDER [3] & Sally JONES [64] Marriage

Again you are correct there is no date for marriage or aquisition of married name. And as we have found re the sort date question, these undated fields head the list. So his marriage (undated) comes before his birth. death and burial.

Nothing can be done about that, other than assigning _SDATE values.

But perhaps these should be listed in the Result Set so the user can deal with them.

In the Sample UK project the Misc fact is not exported in the Gedcom file.
i.e.
Thomas LOWE [40]
Misc 10 Oct 1845 in Vancouver County, Oregon
Misc 05 Apr 1849 in Clackamas County, Oregon
Misc 26 Sep 1850

Does any TMG expert know why they are not exported? Is it something to do with the Tag definition?

I have looked at the Tag definition Misc and it is set to export with all the others. I cannot give a reason but only confirm that each time I export a gedcom it is missing. We could ask the author, but she will be celebrating Christmas and I wont disturb her today. It is best left till after the holiday.

[tatewise]

After some further testing with sample projects a revised version of the Plugin is available.

Try attachment Order Facts by Sort Date Plugin Version 0.9 Date 24 Dec 2017.
[ Attachment deleted as superseded by Version 1.0 in Plugin Store ]

It caters for various nuances of TMG direct import versus TMG Gedcom file.
The TMG Export Gedcom settings discussed earlier are also important.
See how_to:import_from_tmg#import_gedcom_file|> TMG > Import GEDCOM File for necessary export Wizard settings.

[DonF]

Evelyn, Mike,
The reason the TMG UK Sample 'Misc' tags do not export to GEDCOM is because these tags are assigned a GEDCOM tag-type of 'MISC', which does not exist in GEDCOM 5.5.
If the MISC TMG tag is changed to an EVEN (Event) tag-type it would export, provided it met the GEDCOM restrictions, namely it must have a single principal or two principals who are married. Having had a look at these 3, they do conform, so would export.

Evelyn, to fix this issue, go to the Master Tag list in TMG, select the MISC tag, click Edit, go to the 'Other' tab and in the GEDCOM export area, choose the '1 EVEN, 2 TYPE misc' button (default is 'TAG MISC').

Don

[DonF]

And I should have added, before anyone complains about TMG conformance to GEDCOM, the MISC tag was part of GEDCOM up to v4, hence the MISC tag would have been built into the TMG tag list from its first release.

[E Wilcock]

Once again, Thank you Don.
I will do that editing and send Mike fresh copies. For test purposes, tomorrow.
I started using TMG in 2008. And have in the meantime provided Mike with some of my own projects to test - I personally use a custom "Event Misc" Tag.

I will now be out of action for the rest of the day. Our family are coming and I wish you all a happy Christmas too.

[Linda Reinfeld]

Hi Mike -

Back to THG/FH/sort dates after a nice holiday.....
I have been checking my test program (2,000 people) and everything looks OK with the sort dates.
So I tried the latest plugin .9 with my whole project - over 40,000 people. Unfortunately there were a couple of problems.

Error msg:
Invalid Date Day Number. Should never get here.
This occurred for 4 individual people (I have their ID numbers) and one Family record.
Immediately after receiving the family record number the following message:
An error has occurred - plugin failed to complete.
[string "c\ProgramData\Calico Pie\Family Historian\..."] attempt to compare nil with number.

This happened twice - I reran the conversion (with a new TMG-import FH file) and got the same errors with the same ID numbers.

I looked at the people and didn't see anything wrong. What should I be looking for?

Thanks - Linda

[tatewise]

Try new Order Facts by Sort Date Plugin Version 0.9 Date 27 Dec 2017.
[ Attachment deleted as superseded by Version 1.0 in Plugin Store ]

There is revised advice about settings in TMG to improve the exported GEDCOM file.
See how_to:import_from_tmg#import_fact_sort_dates|> Import Fact Sort Dates.
Several facts need adjusting in the Tools > Master Tag Type List such as Address, Misc, Note, Research Notes & Telephone.
In the export wizard Step 7: Option screen 3 ensure Place Levels has every item ticked.

@Linda ~ This now includes the problem date in the message Invalid Date Day Number. Should never get here.
Use the progress bar message to identify the Individual or Family record, but you may need to inspect the subsequent record too, as the progress bar display can lag behind.
Look in the record for the reported date, not only in FH but also in the TMG Project.

[Linda Reinfeld]

Mike -

I have 4 individual records that are a problem - and they are all dates between 3 Sep 1752 and 13 Sep 1752.
I didn't even know about the missing 10 days of Sep 1752. I shall fix those dates.

The plug-in again posted a fail to complete message. This time the number after the colon was 2078. The message was: Attempt to call method 'GetDisplayText (a nil value). I have no idea how to fix or even find that problem.

It takes almost 4 hours to complete a GEDCOM for all my data. One of the great things about a direct transfer is the time - not so long. Now that you have found a method of storing sort dates and presumably using them in reports and transferring to GedSite - I shall be testing that soon - I would fervently hope that Calico Pie, or whoever writes the software, would be able to directly transfer the sort dates.

One question about your instructions on creating the GEDCOM from TMG - you said Step 7 all place levels should be ticked. Does that include the last box to be ticked, which says 'Commas when missing'?

Linda

[tatewise]

I wondered if those 1752 dates might be the problem.

The GetDisplayText(...) error is something I have to fix, but may take a day or two.

No, do not tick 'Commas when missing'.

[DonF]

Issues with dates around 1752 immediately suggest an issue with Old Style dates, but if Linda's dates are written as she has stated, I don't see the problem - it may depend on what is actually in the GEDCOM date field.

As we all know, the Gregorian calendar was introduced by Pope Gregory XIII in 1582 to correct an error in the Julian calendar, as it's year did not match the length of the Earth's orbit round the Sun (at this stage the difference was 10 days).
The Gregorian calendar was adopted in 1582 by France, Italy, Spain, Portugal, and Luxembourg and within a few years by most of the German Catholic countries. In other countries, it varies considerably.
Great Britain and its colonies (including America) did not adopt it until 1752, by which time the difference in the two calendars was 11 days. In addition, the legal start of the calendar year was moved from March 25th (Lady Day) to January 1st by the Calendar (New Style) Act of 1750.
Scotland actually adopted the 1 Jan as the year start in 1600 (as had some other countries), but not the full Gregorian calendar until 1752
The Gregorian system spread later to other countries - Turkey was one of the last to convert in 1926.
The Russian Orthodox church and various Middle Eastern Christian churches still retain the Julian calendar.

In TMG, under File -> Preferences - General Options, one can set the period for which Old Style dates are to be handled (by default 1582-1752)
TMG recognises the following date forms:
24 Feb 1691 OS
24 Feb 1690/1
24 Feb 1690-1
24 Feb 1690 .. 1

In the above examples, if you enter the date in any of these formats (which is how you commonly see them printed in source documents), TMG will use 24 Feb 1690/91.

[Linda Reinfeld]

Yes, the dates are a problem. FH will not let you enter a date between 3 sep 1752 and 13 sep 1752.
I removed the day, and it is fine.

Perhaps a statement should be made in the Knowledge Base - do not tick 'commas when missing'

Linda

[E Wilcock]

Linda, when Mike has finished, I could repost the two TMG screen shots - showing how they are set.

I need to ask possibly stupid questions. I am very confused.
Mike is adding TMG sort dates to Projects imported from TMG. So we dont lose the sort order we once had.

That is brilliant.
But what happens to the new data added in fh?

What fh facilities or plug ins should we use or avoid using in order to retain the sort order for export to GEDSITE?
And would one run the plug in a second time (this time not using the TMG gedcom) before exporting to Gedsite or another genealogy software.

Once we have Projects where the ALL tab shows a sort date field, and are working with those projects, can we open that field and put in sort dates?
One would have to open that specially as it doesnt seem possible to add new events on the All tab but again I dont know why.

[jbtapscott]

Evelyn - as Mike has already indicated, the plugin can be run as often as you like within FH - it will not modify existing LMO/Sort Dates and will add any missing ones. I left TMG some time ago (basically, shortly after its demise was announced) and therefore do not have a TMG export to run the plugin against, so have been running it purely against a copy of my main FH project. If I add a couple of Events to this project and then run the plugin again, it reports that X already have LMO/Sort Dates and then summarises the "new" ones that have been added.

I can confirm that, from all my tests so far, the "Export Gedcom" plugin utilises the LMO / Sort Dates on the FH records and puts them in the exported Gedcom file so that they can be used by GedSite. It means that some of the techniques I was using in GedSite to overcome the issue of undated Events can now be removed as the processing can rely purely on the Sort Dates on the incoming records

[E Wilcock]

Thank you for that reassurance.

But dont underestimate the problem for people like me. I am a historian. Mike explains things in terms that I dont understand.
Like LMO. I dont know what that means when I read it.

After reading your post, I just did a search of this topic to remind me. LMO is Local Multimedia Object
Explained as (LMO) with only a Title of Sort Date and its Date field.
But I am a historian. I dont understand half the things I read in fh discussions.
I now realise that LMO means the extra sort date field Mike has invented. But beyond that a massive mystery.

And I think that is going to be so for a lot of people.

Plus we make mistakes. In TMG we used a custom Tag New paragraph to introduce a carriage return to split up long narrative reports and make them more readable. Mike told me I had one such tag with a Memo (note field) with text in it which should never happen. But I can see that it did because in an alphabetical list of tag types Newpara precedes Note and it is easy to click on the wrong one.

One of the most annoying things about fh is that you cant alter the Tag type once you have got it wrong, or if you change your mind. But using a data base in TMG, one could. Off topic but there you go.

[ColeValleyGirl]

Evelyn, The Change Any Fact Tag (available in the Plugin Store) will let change fix tag problems or changes of mind.

[tatewise]

Evelyn, it IS possible to add facts via the All tab.
Actually the All tab lets you add every valid FH/GEDCOM tag, many of which cannot be added via Facts tab.

To add a fact right-click the record name (1st row under Record header) and choose Add Attribute or Add Event.

BTW: Have you read the plugins:help:order_facts_by_sort_date:order_facts_by_sort_date|> Order Facts by Sort Date Plugin Help & Advice?

[E Wilcock]

The adding via the all screen looks very good. I didnt know one could do that. Thank you.
By changing tag type I meant changing one instance of a Tag.
As an example - one might enter an Emigration event with the source and citation and very full notes about other passengers on that ship and then decide that it ought to be Deportation.
In TMG one could click on the Event name and click another event name to change it.
I think in fh one has to begin all over again?

[tatewise]

No, as has been said, you can use the Change Any Fact Tag Plugin.
You can not only select the type of Tag to change but also Pick the Record(s) to apply the change to.
This thread is in danger of going in too many directions, so please post anew if you need more detailed advice.

[Linda Reinfeld]

Hi Mike -

Looking closely at a sort date: I am using a GedCom file. I believe Rule #2 should be applied - the _SDATE from the file should be used.
For person #12 tag NotePersonal
TMG has no date, sort date of 1867. Gedcom has no DATE, _SDATE of 1867. FH has no date, sort date of 8 February 1864. This sort date is 1 day after she was born. There should actually be a witness tag between the birth date and the NotePersonal date.
But, shouldn't I be seeing 1867 on the sort date in FH? Don't you just use the sort date from the Gedcom? Am I misunderstanding something?

Linda

0 @I12@ INDI
1 REFN 12
1 NAME Alice Bessie /Forbes/
1 NAME Bessie /Sargent/
1 SEX F
1 CHAN
2 DATE 17 NOV 2017
1 BIRT
2 DATE 07 FEB 1864
2 _SDATE 07 FEB 1864
2 PLAC Suisun City, California
2 SOUR @S204@
1 EVEN
2 TYPE NotePersonal
2 _SDATE 1867
2 NOTE She lived with her Aunt Susan (Susan Forbes Silliman - her father Charles
3 CONC ' sister who was married to Benjamin Silliman). She was known as Bessie

[tatewise]

Linda, please read the Help & Advice where it explains the rules.
If the Plugin cannot match the GEDCOM facts with the FH Project facts then it cannot use the _SDATE and will use the Fact Date or if undated then 1 day after previous fact.

The Result Set should help identify why the facts do not match, but the current version may not be comprehensive enough.

In this case the facts appear to have different names:-
NotePersonal in GEDCOM, but Note-Personal in Project (i.e. with a hyphen).
Check the TMG Tools > Master Tag Type List and View that Tag's definition to see why that difference may exist.

BTW: Witness tags only appear in the GEDCOM against the Principal person, and similarly in the All tab of FH.

The next Plugin version gives much better clues in the Result Set about what matches and what does not, but I cannot post it because unfortunately my router died in a thunderstorm last night and I am using someone elses PC. Also OneDrive where the Plugin should be saved is misbehaving. So cannot communicate very often until next week.