* Import from TMG with Sort Date in GEDCOM

[E Wilcock]

Linda - I agree completely. It is and was for Calico Pie to do. To hold to such a narrow version of GEDCOM seems to be as perverse as extreme theology. That was why I personally took the decision not to bring any more of my TMG Projects into fh until Calico Pie took action. I was hoping a new version of fh with the upgrade might materialise before Christmas. But it didnt.

However for the last 2 years (after much help from Mike) I have been working on a research project started new in fh -and after entering about 3,000 records, I found it hard to return to TMG source entry. That was my reason for importing any tree about which I received a query.

The tree used with Mike to examine this plug in is a different matter. It is my largest family tree and was imported to TMG where it always had issues. If one added too many new people at a time, it skipped to a different individual and sometimes froze and drove me mad. The excellent Tech help at TMG were unable to find anything wrong. So when it skipped on me yet again a couple of weeks ago, I decided to rescue it and open it as an fh text file. It was undoctored before opening and a brief look shows that it does have issues - with census entries.

I have three other (manageable and problem free) TMG family trees in fh -which I can run with this plug in. If I do that, what would you suggest I look for to show any errors? I did look at one tree and told Mike I saw none, other than the marriages.

[E Wilcock]

Mike, I downloaded the 6 plug in again.
I made a fresh copy of my Sadler tree. which is the one we have been using. And ran the plug in using the TMG gedcom.

This is an interim report . The news is not brilliant.

I received notice of 3 families where the numbers in TMG did not match the family ID in fh.

So I made a fresh copy and ran it again, and the same result which this time i copied from the error notices.

TM Family ref id 13 matches fh family ref 16
TM Family ref id 14 matches fh family ref 130
TM Family ref id 12 matches fh family ref 226

I once again snipped the overall report window (1812 refers to todays date.)

1812Sadler.JPG (26.1 KiB) Viewed 13106 times

I have not yet opened the Project in fh to look at all this. And will need to go out shortly.

[tatewise]

I also agree that it would be better if Calico Pie designed the TMG direct import to include _SDATE sort dates, but they haven't. So I wondered if a Plugin could do it, and have shown that it can, but we are now investigating the exceptions where it is not quite working. That is how it goes with software and unfortunately I do not have a TMG licence, so must rely on you to help test and diagnose the problems.

As you may have overlooked, I explained that the messages about TMG Family Record Id matching FH Family Record Id are NOT errors, but proof that the Fam Rec Id in the TMG Gedcom are correctly matched up with the FH internal Family Rec Id. It only produces such messages for TMG Fam Rec Id 12 to 14.

@Evelyn ~ Please check the Engagement and Marriage events under discussion.

@Linda ~ Please post snippets from TMG Gedcom for those Will and ResearchSource facts and screenshots of the fully expanded All tabs for the same facts, just like Evelyn's postings but for INDIvidual records. Only 3 attachments are allowed per posting so you may have to post more than once.

[E Wilcock]

Yes Mike in the property box facts tab they are in the correct order with the engagement and then the wedding, for both wife and husband.
And in the All tab of the family records (reached as you told me how) The engagement has a sort date of 16 Sept 1924 and the marriage of 31 October 1924.
And no. You are correct. I didnt understand the significance or otherwise of those messages - I just reported back to you what I had seen.

[E Wilcock]

Unlike Linda, I have not looked in detail at what is happening to sort date order where I have a number of individual events constituting a biography.
I suspect there may be many instances where a TMG sort order is not exactly replicated. This example comes from the first record I picked, somewhat at random.
In TMG the career of M T Sadler born 1801 runs in the following order with his medical education in London followed by his move to work in Barnsley:
Occupation 1822 sort date 1822 Licensee of the Society of Apothecaries
Education circa Jan 1822 sort date circa Jan 1822 United Hospitals of St Thomas and Guys
Address 1823 sort date 1823 14 Maze Pond, Borough London., in lodgings
Occupation 1823 sort date 1823 Member of the Royal College of Surgeons
Occupation either 1823 or 1824 sort date either 1823 or 1824 29 Church Street Barnsley In practice for a short time with Dr Dow at the house Occupied in 1872 by Mr Stawman, another doctor.
Occupation 23 March 1824 sort date 23 March 1824 Barnlsey His uncle the linen dealer Benjamin Sadler sent him £135 to pay to Mr Starr. Dr Sadler took over the practice and dwelling of Mr Starr, probably an army surgeon.

In fh the order changes so that the Occupation (in Barnsley) dated either 1823 or 1824 comes before the 1823 address in London.

MTSadler.JPG (87.48 KiB) Viewed 13060 times

The TMG Sample file for the UK edition was written by a UK user, Caroline Gurney, and I cannot think that it would breach TMG copyright for us to open that sample directly in fh, nor to share a GEDCOM exported from it. After all we all share data all the time. If the actual content is copyright, permission could be requested just for this particular purpose. http://www.carolinegurney.com/

[E Wilcock]

I think it worth adding that the direct imports of my Sadler tree into both fh (not running the plug in) and rm also have this "wrong" order.
My fault for using that either or date, I suppose

[tatewise]

As usual, please post a Gedcom snippet for those facts, so I can see what is going on.
It looks like some of the facts are not picking up the Sort Date from the TMG Gedcom.

[E Wilcock]

Sorry - I thought my transcription from TMG would serve


0 @I2@ INDI
1 TITL Dr
1 NAME Michael Thomas /Sadler/ snr.
1 SEX M
1 CHAN
2 DATE 14 AUG 2009
1 ANCI @SUB1@
1 RESI
2 PLAC Church Street, Barnsley, -Yorkshire, -UK
2 NOTE Surgeon
2 SOUR @S709@
1 BIRT
2 DATE 21 MAY 1801
2 _SDATE 21 MAY 1801
2 PLAC Dovebridge, Derbyshire
2 NOTE previous suggestion of Snelston thought to be wrong. Dovebridge on 1871 c
3 CONC ensus.
2 SOUR @S3@
2 SOUR @S2@
2 SOUR @S689@
3 PAGE gives day and month.
1 BAPM
2 DATE 05 JUL 1801
2 _SDATE 05 JUL 1801
2 PLAC High Street Wesleyan, Uttoxeter, Staffordshire, England
2 NOTE IGI
2 SOUR @S10@
1 OCCU apprenticed to a Surgeon.
2 DATE 1817
2 _SDATE 1817
2 PLAC Melbourne, Derbyshire
2 SOUR @S3@
1 EDUC
2 NOTE apprenticed to Surgeon Apothecary, Richard Beaumont Child for four an
3 CONC d a half years, a premium being paid of £84.
2 DATE 03 JUN 1817
2 _SDATE 03 JUN 1817
2 PLAC Melbourne, Derbyshire
2 SOUR @S689@
1 OCCU Licensee of the Society of Apothecaries
2 DATE 1822
2 _SDATE 1822
2 SOUR @S690@
1 EDUC completed his medical education.
2 DATE ABT JAN 1822
2 _SDATE ABT JAN 1822
2 PLAC United Hospitals of St Thomas and Guys
2 SOUR @S689@
1 _ADDR
2 DATE 1823
2 _SDATE 1823
2 PLAC 14 Maze Pond, Borough, London
2 NOTE in lodgings.
1 OCCU Member of the Royal College of Surgeons
2 DATE 1823
2 _SDATE 1823
2 SOUR @S690@
1 OCCU
2 NOTE In practice for a short time with Dr Dow at the house occupied in 1872 b
3 CONC y Mr Stawman, another doctor.
2 DATE EITHER 1823 OR 1824
2 _SDATE EITHER 1823 OR 1824
2 PLAC 29 Church Street, Barnsley, -Yorkshire, -UK
2 SOUR @S677@
3 PAGE 12 Aug 1872.
1 OCCU
2 NOTE His uncle the linen dealer Benjamin Sadler sent him £135 to pay to Mr Sta
3 CONC rr. Dr Sadler took over the practice and dwelling of Mr Starr, probably a
3 CONC n army surgeon.
2 DATE 23 MAR 1824
2 _SDATE 23 MAR 1824
2 PLAC Barnsley, -Yorkshire, -UK
2 SOUR @S689@
1 OCCU Doctor, surgeon.
2 DATE AUG 1824
2 _SDATE AUG 1824
2 PLAC Barnsley, -Yorkshire, -UK
2 SOUR @S3@

[tatewise]

The actual GEDCOM is crucial, as your transcription omits key details, and in software those precise details really matter.

The problem with the Address fact Sort Date is that in the TMG GEDCOM it has the UDF tag _ADDR that does not match the FH custom fact Address so the Sort Date is not transferred.
Is that fact tag something you have changed after TMG direct import, or is it an automatic change by FH?
If the latter, then the Plugin will have to be adapted take account of such auto-changes.

The problem with the Occupation in 29 Church Street, Barnsley is the EITHER ... OR ... date format.
That is NOT a valid GEDCOM Date, but now I know about it, the Plugin can be adapted to cope with it.
I suspect during the TMG direct import the DATE field is altered by FH to use just the EITHER date and discard the OR date.
BTW: Are there any other TMG Date formats that don't have equivalents in the FH Date Assistant?

Another consideration is the order of Sort Dates as defined by TMG.
It seems from the TMG GEDCOM snippet that the following order applies:
21 MAY 1801 in FH is 21 MAY 1801
05 JUL 1801 in FH is 05 JUL 1801
1817 in FH becomes 31 DEC 1817 that sorts after the next date 03 JUN 1817
03 JUN 1817 in FH is 03 JUN 1817
1822 in FH becomes 31 DEC 1822 that sorts after the next date 31 JAN 1822
ABT JAN 1822 in FH is 31 JAN 1822
1823 in FH becomes 31 DEC 1823
1823 in FH becomes 01 JAN 1824 +1 day on previous sort date
EITHER 1823 OR 1824 will be treated as 1823 and become 02 JAN 1824 +1 day on previous sort date
23 MAR 1824 in FH is 23 MAR 1824
AUG 1824 in FH will be 31 AUG 1824
In other words TMG appears to assume 1st day of month and January if those parts are missing, so the Plugin needs to apply those rules, which are different from the FH rules.

[DonF]

Mike, this may help you (long post!).
These are the date ordering rules for TMG -

All dates which are not irregular are sorted chronologically, according to rules illustrated by the following sorted example:
after 1974
between 1974 and 1975
before 1975
say 1975
circa 1975
1975
after 1975
between 1975 and 1976
bef June 1975
circa June 1975
June 1975
after June 1975
between June and July 1975
From June 1975 to July 1975
June 1975 or July 1975
before 30 June 1975
30 June 1975
after 30 June 1975
July 1975
before 1 July 1975

In general, the order of sort is: before, say, circa, no modifier, after, between, or, from..to, given that all other elements are the same. A date with a question mark will sort immediately after the identical date form without the question mark. Notice that a more specific date form always sorts after a less specific date form. A date is usually sorted at its earliest possible appearance.

So, dates can be displayed with these modifiers and they sort in this order:
Date before date:
before date
bef date
b date
ante date

Date based on some other event date:
say date
s date
estimated date
est date

Date within 20 years plus or minus:
circa date
cir date
c date
about date
abt date

Date after date:
after date
aft date
a date
post date

Date between two dates:
between date and date
bet date and date
btw date and date
date-date
date .. date

One date or another date:
date|date
date or date

From one date to another date:
from date to date
date to date

Don

[Linda Reinfeld]

Mike -

Sorry to take so long - it's a very busy time of year, and I had written a reply last night & lost it all....

Problem: Will - in TMG date May 1658 sort date Nov 1658. In FH sort date May 31, 1658.

GedCom for this person

0 @I1890@ INDI
1 REFN 1890
1 NAME Elizabeth /Hutchins/
1 TITL Widow
1 NAME Elizabeth /Curtiss/
1 SEX F
1 CHAN
2 DATE 22 NOV 2017
1 BIRT
2 DATE EST 1593
2 _SDATE SAY 1593
2 SOUR @S204@
1 IMMI
2 DATE BET SEP 1638 AND JUN 1639
2 _SDATE SEP 1638
2 PLAC Roxbury, Massachusetts
2 NOTE with two of their sons
2 _SHAR @I1891@
3 ROLE with parents
2 SOUR @S1511@
1 EVEN
2 TYPE NoteSS
2 _SDATE SEP 1639
2 PLAC Stratford, Connecticut
2 NOTE Elizabeth moved to Stratford with her sons John, William and Thomas, wher
3 CONC e she became a property owner.
2 SOUR @S1511@
1 EVEN
2 TYPE NoteSS
2 _SDATE SEP 1639
2 NOTE Elizabeth never remarried, and was always known as the Widow Curtiss
2 SOUR @S1511@
1 DEAT
2 DATE JUN 1658
2 _SDATE JUN 1658
2 PLAC Stratford, Connecticut
1 WILL
2 DATE MAY 1658
2 _SDATE NOV 1658
2 NOTE which was probated in Fairfield 4 Nov 1658.
1 FAMS @F259@

I'm sorry I don't know how to post a clip of the property box, the All tab.

My second problem, ResearchSource tag with TMG sort date 1930 and FH date Apr 19, 1895 - turns out this tag was not in the GedCom! I was very surprised to see that and will have to investigate.

Thanks -

Linda Reinfeld

[E Wilcock]

Thank you Don.
Mike, my instant reaction last night was to say that I could not answer your questions nor pursue this at the moment. The tree we have been looking at has been repeatedly exported from one software and imported into the next. Some non standard date phrases have been entered as text. And like Linda I can if I choose continue to work on it in TMG.

I do however have three smaller trees San, W and R, that were brought into fh before I appreciated the difficulties with sort dates and gedsite. Those trees have been worked on in fh and cannot go back to TMG. So it would really help me to work on those with you.

I have today exported a TMG gedcom with sort dates for each of them. And have made copies of the three fh projects.
I will check I have the latest version of the plug in and run it today.

I feel however that it must be possible to run a more systematic series of checks against the fh sort date project.
Any suggestions from yourself or from Linda? For instance it might be possible for fh search to find any non-standard date entry? Or a search through the GEDCOM file to do the same.
The same goes for any personal quirks of my date entry.
Please let me know,
Most (if not all) the sort dates in these projects were generated by TMG and my possibly inconsistent date entry has been faithfully copied from the date field to the sort date field.

[E Wilcock]

This reports on my San tree 367 people.
The first person, E S , I looked at in the Copy after running the plug in has his residence and occupations in the correct order (which they are not in the fh file)
The next person, IS , has his residences (below) in the wrong order clearly due to my poor entry of the dates. I see why you wanted to work from my sloppy trees rather than from the sample. But it would probably be simpler for me to edit the date in fh than for you to adjust the plug in?
.

SanIs.JPG (38.98 KiB) Viewed 12976 times

I am having problems with the gedcom copy and paste so will post this now - two separate extracts.
1 RESI
2 DATE betw c1901-c.1905
2 _SDATE BET 1901 AND 1905
2 PLAC 52 Smithdown Rd.
2 SOUR @S19@

1 RESI
2 DATE ABT 1910
2 _SDATE ABT 1910
2 PLAC 61 Bamber St, West Derby, Liverpool
2 SOUR @S3@
1 OCCU Glazier Master
2 DATE 1911
2 _SDATE 1911
2 SOUR @S13@
1 RESI
2 DATE 1914
2 _SDATE 1914
2 PLAC 51 Whitefield Road (or 57)
2 SOUR @S23@

[E Wilcock]

I have been out but thinking about this all day Mike. It seems to me that discussion may have moved from your plug in to users' date entry and the formatting of dates in the fh project.

Rather than have your plug in capable of handling any dates, it might be better to tidy the dates in a project?

Would it be possible to design a custom query that could pull out any irregularly entered dates in an fh Project?

Even a quick fh text search for " shows me where a date in a tree has been put into inverted commas.
If I could find the records that have imported with date phrases or questionable dates, I could edit them to an acceptable date and then your plug in would have no problem.

[jbtapscott]

Don't know if you have looked at it, but there is a Plugin (from Calico Pie) called "Find Date Phrases" which states in the description "Search all Facts and list all Dates which contain Date Phrases rather than dates. Very useful as a base to clean up badly formatted dates from imported GedCom files"

[E Wilcock]

Thanks a lot Brent. No. I looked at the queries and query design but didnt look at the plug in list.
I will do so - tho it may be a day or two.

[tatewise]

Actually the Fix Date Fields Plugin, does much the same, but is also able to fix many of the Date formats used by other products, but not recognised by FH. Any remaining unfixable dates are listed in the Result Set. Both Plugins are recommended in how_to:index#importing_to_family_historian|> Importing to Family Historian under how_to:fixing_dates_date_phrases|> Fixing Dates and Date Phrases.

However, whereas most unrecognised TMG Dates become a Date Phrase in FH, the EITHER Date OR Date format is transformed to the first Date in FH, so my Plugin must do likewise in both cases for the Facts to match. The correction of Dates should be dealt with by other Plugins.

I have updated the Order Facts by Sort Date Plugin to Version 0.7 Date 20 Dec 2017.
[ Attachment deleted as superseded by many later versions. ]

A major change is to produce a Result Set of each TMG GEDCOM Fact that does NOT match with any FH Fact, so they can be easily investigated.
It caters for abnormal date formats such as betw c1901-c.1905 that become a Date Phrase.
It adjusts for EITHER ... OR ... dates.
It sorts TMG Sort Dates without a day &/or month before other dates.

@Evelyn ~ It would help if you could investigate the TMG _ADDR versus FH Address Fact problem, and how one has transformed into the other as I mentioned earlier.

@Linda ~ To post a screenshot see info:forums#posting_topics|> Posting Topics.

[Linda Reinfeld]

Hi Mike -

I ran the conversion with a new FH file (direct import from TMG) and an new GEDCOM.
Unfortunately the sort date for the will is still incorrect. In TMG it is Nov 1658. In FH it is May 31, 1658

GEDCOM
0 @I1890@ INDI
1 REFN 1890
1 NAME Elizabeth /Hutchins/
1 TITL Widow
1 NAME Elizabeth /Curtiss/
1 SEX F
1 CHAN
2 DATE 22 NOV 2017
1 BIRT
2 DATE EST 1593
2 _SDATE SAY 1593
2 SOUR @S204@
1 IMMI
2 DATE BET SEP 1638 AND JUN 1639
2 _SDATE SEP 1638
2 PLAC Roxbury, Massachusetts
2 NOTE with two of their sons
2 _SHAR @I1891@
3 ROLE with parents
2 SOUR @S1511@
1 EVEN
2 TYPE NoteSS
2 _SDATE SEP 1639
2 PLAC Stratford, Connecticut
2 NOTE Elizabeth moved to Stratford with her sons John, William and Thomas, wher
3 CONC e she became a property owner.
2 SOUR @S1511@
1 EVEN
2 TYPE NoteSS
2 _SDATE SEP 1639
2 NOTE Elizabeth never remarried, and was always known as the Widow Curtiss
2 SOUR @S1511@
1 DEAT
2 DATE JUN 1658
2 _SDATE JUN 1658
2 PLAC Stratford, Connecticut
1 WILL
2 DATE MAY 1658
2 _SDATE NOV 1658
2 NOTE which was probated in Fairfield 4 Nov 1658.
1 EVEN
2 TYPE ResearchSource
2 _SDATE 1699
2 NOTE Curtiss, FG: A Genealogy of the Curtiss Family quoted in Americana XXXII
3 CONC ; Americana XXXII No 1 1938; Orcutt, Rev S: A History of the Old Town o
3 CONC f Stratford p 1182 quoted in Americana XXXII; Jacobus, DL: Old Fairfield
1 FAMS @F259@

Property Box in FH


I used the Windows Snip Tool. I can print the snip, save the snip, copy the snip and paste into a word document,
but I can't paste it into here. Sorry

FH Sort Date
Title Sort Date
Date May 31, 1658

Linda Reinfeld

[tatewise]

Linda, as it says in info:forums#posting_topics|> Posting Topics:
"Drag & Drop a file to text box will work, but Copy & Paste a file or PrtScn & Paste clipboard will not."

You must save the snipped screenshot to a JPG/PNG file perhaps to your Desktop, then either use the Attachments tab or use Drag & Drop to get the file into your posting.

[Linda Reinfeld]

Mike -

I looked at the result set of the plugin, and that shows the date as May 1648 and the s_date as Nov 1648.
So the date just isn't making it into the Property Box.
The only 4 facts/tags for Elizabeth Hutchins were Will, Death, Immigration & Birth. As you can see from the GEDCOM she had more tags than just those 4, and undated ones.

Linda

[Linda Reinfeld]

Capture.JPG (65.75 KiB) Viewed 12907 times

It doesn't look like it made it.
Mike - why is it so important you actually see the property box?
I have indicated what the values are.

Linda

[tatewise]

It is important to see the Property Box with its All tab fully expanded without any [+] signs.
The Plugin matches the GEDCOM Fact details with the FH Fact details using Date & Place, etc.
The reason a Sort Date does not make it into FH is that matching is failing, and I need to understand why.

Please post the All tab fully expanded without [+] signs BEFORE running the Plugin.

To fully expand tab, just click on Hutchins, Elizabeth twice.

BTW: You say you "looked at the result set of the plugin, and that shows the date as May 1648 and the s_date as Nov 1648"
Is that a typo and should be 1658 or is the Result Set really saying 1648?

I think I see the cause of the mismatch problem.
Please temporarily change Tools > Preferences > General > Preferred Standard Date Format to dd mmm yyyy and see if that fixes this unused Sort Dates problem.
If it does then I know how to adapt the Plugin to rectify the problem.

[Linda Reinfeld]

Eliz_Hutchins.JPG (68.25 KiB) Viewed 12902 times

Sorry Mike I didn't understand you wanted the All tab before conversion.

[Linda Reinfeld]

Mike -

I changed the date format, ran the plugin - and the will was fixed!.
I shall check others later

Thanks -

Linda

[E Wilcock]

@Evelyn ~ It would help if you could investigate the TMG _ADDR versus FH Address Fact problem, and how one has transformed into the other as I mentioned earlier.

Mike - You are doing wonders. When I went to bed last night for the first time I felt the TMG import was going to be solved.

But I didnt understand your question above, didnt know what to investigate and I couldnt deal with it. It applied to a problematic project that has been through several software programs in its time. I had done nothing to tidy before or after import. i.e. I hadnt sorted out or renamed Tags and facts. And I couldnt cope with this fh issue.
It may well be that after Christmas I can, but, as I posted yesterday, my continuing on this Thread means limiting it to the three projects I have in fh.
I am about to run the date plug in against my husband's tree and then I am going out to work.
And I am so muddled, I dont even know or remember which plug in I am supposed to use?
I know a great deal about my own subject and cooking. But I am not a computer person. I need it to be simple. I need to be told to do A, B and C.