* Import from TMG with Sort Date in GEDCOM

[E Wilcock]

Mike, I have now run your Fix dates Plug in against my Wil tree copy.
I would like to run it against the real project.
I asked it to list (not to correct)
There were about 9 non standard dates. It seems fh does not permit either or. Often from age at census I would enter either or for birth year.
But the other errors are all census dates put in quotes. With a zero before the day - that could have arisen from the UK dated census fields. I dont know how that came about. as in.
"07 April 1861"

I am reporting back on this in case other people have the same.

[E Wilcock]

The problem with this fh problem solving is that it occupies one's mind to the detriment of other things. I have a work deadline and it is like kids with smart phones. Addictive.
The question about tags and facts in TMG can probably be answered in detail by Don.
But this is my view. And I felt I had to give you this info Mike, even tho I am supposed to researching.
In TMG one could set up different languages - not in the sense of foreign languages but each language could hold its own set of tags and sentences.
This specially designed output might be required for various purposes but I used it for Second site - My SS language allowed me to design an outline website giving b m d plus a generic occupation and place of residence.
And not to give a complete run down of a career or many changes of address which would appear in the output from my default language.
In UK village genealogy there are duplicate names and birth dates so it helped to show John Smith was a baker in Little Bookham.

Any irregular Address and Occupation tags in my Projects may well arise from this.
But they would be peculiar to me. Each person will have used TMG in their own way.

These undated tags of mine have a bearing on fh - In fh one cant pick a primary instance of a Tag. So I need the undated occupation and place to come at the top of the list - so that they are the ones that will show using Gedsite. However there is a new version of Gedsite just out which I havent yet bought or examined. So I cannot tell you the precise requirements. Very often something one uses in a workround with computers becomes obsolete.

[tatewise]

@Linda ~ Thank you that confirmation about the date format option. I have updated the Plugin to cope with either option.

@Evelyn ~ When migrating from TMG to FH it is important to use the Order Facts by Sort Date Plugin BEFORE the Fix Date Fields Plugin, or any other corrective measures such as Change Any Fact Tag or Give Witnesses Their Own Facts.
The reason is that the Order Facts by Sort Date Plugin is trying to match TMG Facts to FH Facts using their Tag Name, Date & Place fields. If those are changed, the Facts won't match, and the Sort Date will NOT be imported, and many unused GEDCOM Tags will be listed in the Result Set.

What I am struggling to identify are the automatic changes made to Facts by FH during direct import from TMG.
1) Does each 1 _ADDR tag become a 1 EVEN 2 TYPE Address custom event?
2) Does each EITHER date1 OR date2 become just date1?
3) Does each SAY date become an EST date?

1) must be true if for your Sadler tree you "had done nothing to tidy before or after import. i.e. I hadnt sorted out or renamed Tags and facts."

2) is in doubt because you have found EITHER date1 OR date2 in your FH Project.

I don't think a leading zero on the day number would cause a date to be invalid and become a "Date Phrase".
Are you sure there is no other explanation for why those Census dates are invalid?

[E Wilcock]

I think I should be able to answer these questions Mike - given a bit of time. To think out how.
As I have copies of my Projects in TMG and can re import them into fh as often as I want - we have plenty of trees we can look at.

But the census dates which cropped up in my husband's WIL tree may well have been entered in my first DOS genealogy programme with year only dates. I never bothered with day and month - before censuses went on line. And so old census entries may have acquired day and month in some way when I was using John Cardinal's TMG Utilities to give day and month dates to all my census entries, in the process of shedding the dated UK census fields from TMG.

I also understand that you dont want me to alter any dates before running the sort date plug-in. Tho I had imagined that would make life easier for the plug in. As it was faultering only at my poor data entry.

I didnt ask your dates plug in to do any correcting - It found the dates and I did the alterations myself by hand. And only on a copy of my WIL project.

So a supplementary question - if one uses the plug in only to scrutinise a project, to discover the date phrases etc, and makes no changes, does that still cause a problem?

[E Wilcock]

Mike _ I have done what I think you wished me to do.

I made a new copy of my problematic sadler tree.
I downloaded the new plug in.
I ran it and got a long list of records -haven't examined them.

SAD21.JPG (29.41 KiB) Viewed 13057 times

Then looked at the events in Barnsley and London which still seems out of order.

SAD21b.JPG (49.55 KiB) Viewed 13057 times

The gedcom was the same as before. In my Post of 19 Nov. But I need to post this before examining it. I suspect the error arises from my dating, because from the computer point of view it is arbitrary which comes first?

[E Wilcock]

Mike - I saved the massive list of unused GEDCOM records as a pdf. But I havent a clue what it means.
I dont really understand what I am doing and I feel that my confusion shows I was right to decide to leave that Sadler tree for the time being.
There is so much room for errors made by myself that it is hardly a reliable test of your plug in.

[tatewise]

Running the Fix Date Fields Plugin in List mode is OK as nothing gets changed.
However, if you tell it to Mend dates, then the Date fields on many Facts in FH will be altered.
Those Date fields will now be different from the DATE values in the TMG GEDCOM file.
Thus the Order Facts by Sort Dates Plugin can't match those FH Facts to the TMG Facts because their Dates don't match.

Regarding your Sadler tree, the list of unused Gedcom Tags is all the TMG Facts in the TMG Gedcom file that the Plugin cannot match against FH Facts in the Sadler Project. That may be due to deficiencies in the Plugin not coping with automatic changes made by FH when it imported that Sadler Project from TMG, or it may be due to changes that you have made since then. The only way we will find out is if you direct import that project again with a new name, and run the Plugin with the TMG Gedcom on that new project and see whether it copes with more Sort Dates and fewer unused Gedcom Tags.

In the meantime here is a revised Order Facts by Sort Dates Plugin Version 0.8 Date 21 Dec 2017.
[ Attachment deleted as superseded by many later versions. ]
It fixes the Date Format option problem experienced by Linda.
It copes better with EITHER date OR date format and SAY date format.
BUT the main thing is the user dialogue has changed:-
It now asks what Mode for dates missing a day or month :

• Before other dates will use 1st and Jan like TMG
• After other dates will use last day and Dec like FH

That Mode and the Gedcom file settings are now 'sticky' and are saved from run to run.

[E Wilcock]

Mike - I wish you very well in your work for those of us who arrived from TMG. But I wish you would believe me when I say that just at present I cannot give you the test you want. I have Christmas, family and a work deadline to meet. And due to worry about this, I slept very badly last night, convinced that the errors were my own. That I had possibly used the wrong GEDCOM which is easy to do since TMG will always use the same name and overwrite unless one intervenes.

So this morning I got up and exported a new Sadler TMG Gedcom. I re imported the Sadler tree into fh and ran the plug in version 0.7 20 Dec. The results looked much the same as yesterday.

sadler22.JPG (29.85 KiB) Viewed 13037 times

The fact that the events come in the wrong order is hardly surprising. Both have TMG sort dates of 1823. The first one receives a sort date on import of 31 Dec 1823 and by logic the next one in the Gedcom gets 1 Jan 1824. There is no way a computer programme can discover the order they were in in a Project in TMG. That is why (even with import of sort dates) RM also has them in the wrong order.

After running the plug in I noticed you had put up a new version 0.8 21 dec. So I reimported the Project but used the same GEDCOM and got an error message.

error.JPG (18.68 KiB) Viewed 13037 times

As I said. I will pull myself together and work with you please on the three trees which I have irrevocably in fh. But I dont have the time or energy to worry about this tree at the moment and certainly not to lose sleep over it. And neither should you.

[tatewise]

I am a little surprised that the summary statistics are exactly the same as before.
That suggests you had NOT made any changes to your earlier import of the Sadler tree after all.
As far as you can recall is that true?

If that is NOT true, are you sure you ran the Plugin on the newly imported Sadler tree?

BTW: After running the Plugin you can get back to the original imported tree by using Edit > Undo Plugin Updates before closing FH. That saves going through the TMG direct import process repeatedly.

An error has occurred - plugin failed to complete suggests a significant mismatch between the TMG Gedcom File and the FH Project. If that error persists then revert to the Version 0.7 Plugin attached to my Wed Dec 20, 2017 posting.

Can you attach the massive Result Set PDF you saved yesterday, or is it too big?
If it is too big, can you post a screenshot of the beginning of the Result Set.

[E Wilcock]

I ran the latest version on my SAN tree and no problems.
This is just an observation about my way of dating historical periods. If I know someone was in the forces during WW1, I enter between 1914 and 1918.
For WW2 war work, I enter between 1939 and 1945. The sorting moves war time work in ammunitions to 1 Jan 1939 so it comes before the summer of 1939 and the 1939 register entries.
It is my fault I should have entered months for these wars. Between Sept 1939 or Aug 1914. The same with Holocaust deaths when people vanished 1942-1945 or 1938-1945 with no date. But these death dates do sort to the end.
I dont know how other people deal with historical periods?

[jbtapscott]

I have events with BET dates of just years (eg 1939-1945) and some with Month Years - it all depends on what source data I have.

That said, don't forget - you can add or change a LMO Sort Date in FH. Although I don't have the TMG exports any more (I moved away from TMG for my main processing too long ago), I have been testing the new plugin on copies of my main project. I have already gone through a process on my main project of adding LMO Sort Dates to Events where I thought the plugin rules might not give me results I wanted - eg Birth and Baptism Events where the former is just Month / Year and the latter is a full date in the same month, and similar for Death / Burial events.

[jbtapscott]

Mike - just to let you know, "no news is good news"!. I am continuing to test each version of the plugin against newly created copies of my main project, but to date have not found any issues / queries over and above those already mentioned here. As before, I don't have a TMG export so this is purely running the plugin against my FH project copy and then exporting via your Export plugin to create a website in GedSite.

[E Wilcock]

I am a little surprised that the summary statistics are exactly the same as before.
That suggests you had NOT made any changes to your earlier import of the Sadler tree after all.
As far as you can recall is that true?.

Mike - It is not surprising. because I did exactly the same a second time and got the same result.
You asked me to reimport it and not to use the a copy of the Sadler project I had worked on in fh.
I repeated the exercise because both you and I suspected that I could have made an error.

But I have also repeatedly asked you to ignore the Sadler tree for the time being.

And volunteered to test instead the three projects already in fh.

In the mean time, I have asked for a check list of what I should be looking to check in these three projects.

[tatewise]

Sorry, I have rechecked, and the summary statistics are NOT exactly the same as before.
But they are very similar, with only 21 extra GEDCOM File _SDATE facts matched with FH Project facts.
However, lets put that Sadler Project aside and concentrate on the others.

The ideal objective is for summary statistics to say all LMO Sort Dates were derived from rule 2) GEDCOM File _SDATE.
However, there may be a few derived from rule 5) Previous Sort Date +1 day where successive _SDATE are the same.

If any other rules are used, then some GEDCOM File _SDATE facts are NOT matching with FH Project facts.
So the Result Set must be interrogated to discover why the facts are not matching.
The Plugin matches by comparing Fact Tag Name & Value, Date, and Place.
Since the Project has only just been directly imported from TMG, the explanation will be due to an error in the Plugin.
The reason will most likely be some transformation by FH during direct import that the Plugin does not cater for.
So you need to look for recurring patterns of Fact Tag or Fact Date formats that don't match the Project facts.

The only other thing to check is that Facts are sorted as desired for the actual Sort Dates.
But this is only worthwhile after the _SDATE values are being correctly imported.

[E Wilcock]

Thank you for the explanation. Now I understand what we are looking for I will try to produce some examples.

I have a bad memory and easilly get confused. I have just deleted many copy files and versions of the plug in. To make a fresh start.

I assume you need a TMG file just newly opened in fh (direct import) run against the latest version of the plug in.
I think it best to find some TMG Project that has never been worked on in TMG.

[E Wilcock]

Test 1

I opened the TMG Sample project in fh and ran the latest version of the plug in Version 8, 21 Dec (freshly downloaded to avoid confusion) and using a gecom exported for the purpose. The plug in failed to run.

I did not snip the error message but I believe it to be the same as with my own project.

I will delete that version of the plug in and my next trial will be to do the same with the previous version.

[E Wilcock]

Test 2

As above TMG Sample project opened in fh and run with the same gedcom file as above but using plug in version 0.7

This looks much better Mike.

sample7.JPG (28.93 KiB) Viewed 12970 times

I will double check now by doing the same to the UK sample file.

[E Wilcock]

Test 2

I ran the TMG UK sample file with version 0.7 of the plug in and a gedcom exported from tmg for the purpose.

Sampleuk7.JPG (26.83 KiB) Viewed 12970 times

I will leave this with you Mike. So you can have a look at that version 0.8. I also have the list of names generated - but forgive me I dont know or cant remember what they are for? But I have saved both lists, if you need them.

If you are happy with these figures I will progress to one of my own projects.

[E Wilcock]

Test 3 -
I thought it might be interesting to compare the results running 0.7 on a tree just opened in fh direct from TMG with the results running that same tree in its fh version - It has been in fh and has been added to in fh since April 2017. I used the same TMG GEDCOM for both.

First the direct tmg import

WILTMG7.JPG (29.29 KiB) Viewed 12970 times

and here is the version that has been worked on in fh

wilfh7.JPG (29.59 KiB) Viewed 12970 times

[tatewise]

As I said earlier, all those tests have poor results, as too few Sort Dates are derived from rule 2) GEDCOM File _SDATE.
The other rules should hardly ever be used, so I am not happy with those figures.

As I said earlier, the Result Set of Unused GEDCOM Tags must be interrogated to discover why the facts are not matching.
Please post the lists you saved.

The problem with Plugin Version 0.8 is that it is handling Family records differently and it seems that FH Project Family records are not matching TMG GEDCOM Family records.
If you remember, unlike Individual records, the Family Record Id don't match, so the Plugin uses the Family member links (HUSB, WIFE, CHIL) to perform the match.

The attached Order Facts by Sort Date Plugin Version 0.8 Date 23 Dec 2017 produces an error message every time a Family does not match.
[ Attachment deleted as superseded by many later versions. ]
Please note the family member Rec Id, and try to find the matching GEDCOM Family record.
i.e. Search for 1 HUSB @I123@ where 123 is the HUSB Rec Id (remember after @ is capital letter I).
Then check the WIFE and CHIL for matching Rec Id.
When found please post a copy of those GEDCOM lines.

[E Wilcock]

As I said earlier, the Result Set of Unused GEDCOM Tags must be interrogated to discover why the facts are not matching.

I didnt know what you meant by interrogate.

.Please post the lists you saved.

Again, I didnt know which format to save them in to post here.

I will down load the new plug in but what file should I look at? Or all four of them again?

[E Wilcock]

Test 4.
I opened a new copy of the TMG sample file in fh and ran the latest plug 08 of 23 Dec using the same gedcom as previously

This is the report

sample1537.JPG (28.69 KiB) Viewed 12967 times

There were 6 error notifications which I have had to copy by hand and retype here - so allow for user error.

Alexander husband no 17 and Martha Ferguson wife 18 with children nos 15,27,28,29,10,30,31.32.33.63.
Elizabeth Kelly wife no 26 with child no 23
Jacob Keebler Husband 50 and child 21
John Keebler Husband 51 with child 21
Ann Haws wife 52 with child 21
Mary Haws wife 52 and child 21

Sorry I now think this is what you want - copied from the TMG Gedcom for the first family

0 @F8@ FAM
1 HUSB @I17@
1 WIFE @I18@
1 CHIL @I63@
1 CHIL @I15@
1 CHIL @I27@
1 CHIL @I28@
1 CHIL @I29@
1 CHIL @I10@
1 CHIL @I30@
1 CHIL @I31@
1 CHIL @I32@
1 CHIL @I33@
1 MARR
2 DATE ABT 1800
2 _SDATE ABT 1800
2 SOUR @S5@
3 PAGE - first child born 1802.
2 SOUR @S11@
2 SOUR @S27@
3 PAGE citing Alexander/Ferguson Bible
0 @F9@ FAM
1 HUSB @I12@
1 WIFE @I13@
1 CHIL @I14@
1 CHIL @I16@
1 CHIL @I11@
1 MARR
2 DATE BEF 1808
2 _SDATE BEF 1808
2 SOUR @S5@
3 PAGE first child born 1808
2 SOUR @S12@
0 @F10@ FAM
1 HUSB @I14@
1 WIFE @I15@
1 MARR
2 DATE BEF 1830
2 _SDATE BEF 1830
2 SOUR @S2@
2 SOUR @S5@
3 PAGE first child born 1830
0 @F11@ FAM
1 HUSB @I19@
1 WIFE @I20@

I will go on with the others.

[E Wilcock]

Elizabeth Kelly wife no 26 with child no 23


Elizabeth Kelly wife no 26 with child no 23I cant find a family record for @I26@ She seems to be an unwed mother - no father entered.

Jacob Keebler Husband 50 and child 21
0 @F22@ FAM
1 HUSB @I50@
1 WIFE @I53@
1 MARR
2 DATE 12 OCT 1812
2 _SDATE 12 OCT 1812
2 PLAC Washington County, Tennessee
2 NOTE bond: James Keebler
2 SOUR @S19@
3 PAGE 9
1 MARR
2 DATE 29 OCT 1812
2 _SDATE 29 OCT 1812
2 PLAC Washington County, Tennessee
2 NOTE bond: Jacob Keebler, James Keebler
2 SOUR @S34@
3 PAGE 10
0 @F23@ FAM


John Keebler Husband 51 with child 21
Ann Haws wife 52 with child 21
Mary Haws wife 52 and child 21
Jacob Keebler Husband 50 and child 21
John Keebler Husband 51 with child 21
Ann Haws wife 52 with child 21
Mary Haws wife 52 and child 21

Since child 21 is clearly a problem this is the Gedcom for
0 @I21@ INDI
1 NAME Benjamin Franklin /Keebler/
2 SOUR @S7@
3 PAGE Letter of 21 Feb 1971. No source given.
1 SEX M
1 CHAN
2 DATE 24 JUN 2000
1 ANCI @SUB1@
1 DESI @SUB1@
1 BIRT
2 DATE 09 NOV 1819
2 _SDATE 09 NOV 1819
2 SOUR @S7@
3 PAGE Letter of 21 Feb 1971. No source given
1 RESI
2 DATE BET 1844 AND 1854
2 _SDATE BET 1844 AND 1854
2 PLAC Limestone, Washington County, Tennessee
2 NOTE near Johnson City.
2 SOUR @S2@
1 DEAT
2 DATE BEF 1860
2 _SDATE BEF 1860
2 SOUR @S10@
3 PAGE Benj. F. Keebler - no entry found. Son Henry Clay living with John and S
4 CONC arah (Keebler) Squibb.
1 FAMS @F3@
1 FAMS @F12@
0 @I22@ INDI

[E Wilcock]

Test 5.
A repeat of test 3 Above with a file WIL brought in from TMG.
The overall results were as above.
However only one error message came up.

Joseph 62, Harriet 278 and children 473 and 539.

0 @I62@ INDI
1 NAME Joseph /Wilcock/
1 SEX M
1 CHAN
2 DATE 16 JAN 2011
1 BIRT
2 DATE 07 OCT 1821
2 _SDATE 07 OCT 1821
2 PLAC Hemsworth, Yorkshire, UK
1 CENS
2 DATE 07 APR 1861
2 _SDATE 07 APR 1861
2 PLAC 47 Potovens Road, Stanley with Wrenthorpe, Wakefield, Yorkshire
2 NOTE Joseph was listed as 39 Agricutural Labourer.
2 _SHAR @I473@
3 ROLE Witness
2 SOUR @S118@
1 DEAT
2 DATE AFT 1881
2 _SDATE AFT 1881
1 NOTE possible daughter only as always living with her grandparents.
2 _DATE 2010
2 _SDATE 2010
1 FAMS @F44@
1 FAMC @F45@
0 @I63@ INDI
1 NAME /Mary/

and his family entry
0 @F44@ FAM
1 HUSB @I62@
1 WIFE @I278@
1 CHIL @I539@
1 CHIL @I473@
1 MARR
2 DATE ABT 1851
2 _SDATE ABT 1851


I hope this helps - but I dont think it gets to the root of the problem?

[E Wilcock]

I realise a bit late that I should also have copied the wife and child bits of the gedcom. Will try to do so later.
At least I understand these @I numbers now