* Sort dates with plugin for TMG import

Importing from another genealogy program? This is the place to ask. Questions about Exporting should go in the Exporting sub-forum of the General Usage forum.
Post Reply
avatar
Linda Reinfeld
Famous
Posts: 124
Joined: 04 Nov 2014 17:34
Family Historian: V7

Sort dates with plugin for TMG import

Post by Linda Reinfeld »

I am using FH v6.2.7, plugin Order Facts by Sort Date version 1.1.

Everyone of my tags/facts in TMG has a sort date. Therefore all of the PlugIn results should be using Rule 2.

When I run the plugin, I get 32424 records in result set - about half say GEDCOM fact is missing and the other half say GEDCOM fact not used by Project. The messages are not quite true. The tag/fact that says "missing" is then shown as "fact not used".

One set of problems: in TMG my date and sort date are May 1931.
The record that says GEDCOM fact is missing has the following fields:
Record Name: Richard Collier Sargent; Rec Id: 5; Fact Summary: Education; Fact Date: May 1931; Sort Date: May 1, 1931; Fact Place: Phillips Academy, Andover, Massachusetts; Help & Advice: GEDCOM fact is missing.
The record that says GEDCOM fact not used by Project has the following fields:
Records Name: Richard Collier Sargent; Rec Id: 5; Fact Summary: 1 EVEN 2 TYPE Education; Fact Date: 2 DATE May 1931; Sort Date: 2_SDATE May 1931; Fact Place: 2 PLAC Andover, Massachusetts; Help & Advice: GEDCOM fact not used by Project
The Record in the GEDCOM that contains the sort dates:
1 EVEN
2 TYPE Education
2 DATE MAY 1931
2 _SDATE MAY 1931
2 PLAC Andover, Massachusetts

In TMG and in the GEDCOM I created the sort date is May 1931. However the plugin has chosen to change it to May 1, 1931. This happens with dates that have just a year or just a year and a month. The date that should be in the sort date field in FH is May 1931. It is clearly that date in the GEDCoM that contains the sort dates.

It seems to me as though you are matching on Fact Place, which is not the same (due to the Gedcom limitations). I would have hoped the match would be on record name, ID, Fact Summary Date, and then use that sort date.

Thanks for your help

Linda Reinfeld
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Sort dates with plugin for TMG import

Post by tatewise »

OK, it is quite a complex process for the Plugin to match up facts in the TMG GEDCOM file and the FH Project.

In your example it is saying the standard Education attribute with tag 1 EDUC exists in the FH Project but is missing from the TMG GEDCOM.

Also the custom Education event with tag 1 EVEN 2 TYPE Education exists in the TMG GEDCOM but not the FH Project.

It's easy for you and I to see they should be paired up, but the Plugin is not so intuitive, and dare not make rash assumptions.

In TMG those custom Education facts are associated with a Master Tag Types definition that says to use the 1 EVEN 2 TYPE Education form when exported to GEDCOM instead of the 1 EDUC tag.
However, when you imported from TMG to FH they should have migrated across as 1 EVEN 2 TYPE Education.
That was discussed in Education tag - attribute/event - import from TMG (14960).
I wonder if you've used the Change Any Fact Tag Plugin to convert them to standard Education attributes 1 EDUC.
If so, that is why they no longer match the TMG GEDCOM containing sort dates.

Can you put things into a time-frame ~ when did you import the FH Project from TMG, and what processing have you performed on it until you have created the TMG GEDCOM for sort dates.

It says in plugins:help:order_facts_by_sort_date:order_facts_by_sort_date|> Order Facts by Sort Date Plugin that "Such files should be generated shortly after migrating from another product such as TMG. See how_to:import_from_tmg#import_fact_sort_dates|> Import Fact Sort Dates from TMG for details."
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Linda Reinfeld
Famous
Posts: 124
Joined: 04 Nov 2014 17:34
Family Historian: V7

Re: Sort dates with plugin for TMG import

Post by Linda Reinfeld »

Hi Mike -

I'm sorry to have used the Education tag as an example. To me the difference between attributes and events is totally confusing. The way I have it set up now is sort-of working, so I don't think I will change anything. Usually the sort date is OK with those tags, it's just that you don't take the sort date from the sort date field. Such as a year only, or a year and a month. You add on a day.

I have a bigger problem with another tag - called Story. It has ended up with an incorrect date.
The record that says GEDCOM fact is missing has the following fields:
Record Name: Barbara Lois Male; Rec Id: 3; Fact Summary: Story; Fact Date: (empty); Sort Date: September 2, 1940; Fact Place: April 2017; Help & Advice: GEDCOM fact is missing. (the Fact Place data is OK, it is correct, the memo/notes start with a date)
The record that says GEDCOM Fact not used by Project has the following fields:
Record Name: Barbara Lois Male; Rec Id: 3; Fact Summary: 1 EVEN 2 TYPE Story; Fact Date: (empty); Sort Date: 2 _SDATE 1997; Fact Place: Empty; Help & Advice: GEDCOM fact not used by Project

I don't know why the incorrect sort date was put into the field. I hope I have described it in enough detail.

I must say it is quite frustrating. In my system I have sort dates for every tag/event. It is a much more difficult task to match sort dates after the fact - much easier to just take the sort dates at the time of the direct conversion.

Sorry not to have answered your questions. I imported my data and immediately ran the sort date plugin. Well, actually first I ran 2 plugins, Move Citation Tet Fields, and Move Labelled Notes to Sort Fields. I have not made a final conversion to FH - I am still trying to get it to work.

Linda
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Sort dates with plugin for TMG import

Post by tatewise »

In FH it is best to use explicit Sort Dates to ensure the sort order is well defined.
The Plugin has an option to choose whether imported Sort Dates missing a day or month come before or after other Sort Dates. However, you can change any Sort Date afterwards.

On closer inspection I have realised why the GEDCOM and Project facts don't match for both your Education and Story examples. Their Place fields are different.

So we need to investigate why the Place fields in the imported Project differ from the Place fields in the TMG GEDCOM.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Linda Reinfeld
Famous
Posts: 124
Joined: 04 Nov 2014 17:34
Family Historian: V7

Re: Sort dates with plugin for TMG import

Post by Linda Reinfeld »

Hi Mike -

When I created the GEDCOM for the sort-date match I used what is called 'short place field', which is city, county, state. I never thought that the GEDCOM match would rely on place data. I am rerunning the GEDCoM extract for sort dates and choosing to import all fields in the place. I will then reconvert - do a direct import into FH and then use the latest GEDCOM for sort dates. and then see what happens. This process takes a while......

Linda
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Sort dates with plugin for TMG import

Post by tatewise »

Does the advice at how_to:import_from_tmg#import_fact_sort_dates|> Import Fact Sort Dates on how to use the File > Export > Gedcom (*.ged) Wizard dialogue need improving to avoid choosing 'short place field'?

The Plugin needs to compare several fields, otherwise it cannot differentiate between multiple instances of similar facts such as Census or Occupation for one person. If say two Residence facts have no Date and just a Place field, then that is the only way to associate them with the desired Sort Date.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Linda Reinfeld
Famous
Posts: 124
Joined: 04 Nov 2014 17:34
Family Historian: V7

Re: Sort dates with plugin for TMG import

Post by Linda Reinfeld »

Hi Mike - I redid the import and matching GEDCOM sort date file and came up with very different results - so I think that all place levels should be identified in the Knowledge Base instructions. (Actually, did you do that already? I saw it mentioned in the instructions. I had never looked at the instructions because I was on the forum as you were writing the plug-in).

Actually it seems to me that in TMG if you don't have a date or a sort date then you don't care where the record is placed. So if you have 2 residence tags with no dates then whether their place fields are different doesn't matter, the author doesn't care (or wants them to sort to the top or the bottom).

Anyway, my new numbers
122187 facts assigned LMO sort date
121571 used rule 2
489 used rule 4
127 used rule 5

As you can see these are better numbers.

I am working my way through them.
The first thing I see is that old-style dates are a place that doesn't match. But that doesn't seem to be much of a problem.
The correct sort date is in the sort date field - it's just that the 2 different records are identifies as not matching.

Still working on the rest of the data exceptions.
thanks, Linda
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Sort dates with plugin for TMG import

Post by tatewise »

That is good news. So it seems that the TMG 'short place field' setting is critical.

Could you please review how_to:import_from_tmg#import_fact_sort_dates|> Import Fact Sort Dates and identify exactly which Step and Option screen numbers apply to that 'short place field' setting and exactly how it is worded, so I can update the instructions.

Regarding fact matching, I accept that if there is no Fact Date and no Sort Date then the author does not care about the order, but is it possible in TMG to set a Sort Date even when there is no Fact Date, because in that case matching the Place field is important?

Not sure what you mean by 'old-style dates' not matching. Is that an area where the Plugin matching could be improved?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Sort dates with plugin for TMG import

Post by jbtapscott »

Yes Mike, in TMG you can have a Sort Date and no Fact Date - it was the way to get facts sorted in the order you wanted even if there were missing Fact Dates
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Sort dates with plugin for TMG import

Post by tatewise »

Thanks Brent. I thought so.
Can you provide the details I requested for the TMG 'short place field' setting, so the [kb]|[/kb] can be updated.
Unfortunately, I don't have a licenced copy of TMG to experiment with.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Sort dates with plugin for TMG import

Post by jbtapscott »

TMG Help for Export to Gedcom (Step 7: Option Screen 3) just states the following:

"
Places

You can choose whether to export only the Short Place fields or you can check which of the place fields will be exported, whether a comma will be exported when a field is missing, and, if Commas when missing is checked, then you specify whether to trim leading and trailing spaces which may have been inadvertently included in the data.
"
The Place Fields offered if you choose the second option are:
Addressee
Detail
Village/Area
Town/City
County/Region
Country
Postal Code
Phone
Coordinates
Temple (TEMP)

Elsewhere, the Help says:

"Short Place

A short place is an additional way of referring to places. It does not include all of the place fields and is used primarily in charts and other places where the entire set of place fields will not fit. The default is [CITY], [COUNTY], [STATE].
"
and
"
Short Place Template

You can change the fields that will be included in the default Short Place. A Short Place is an additional way of referring to places. It does not include all of the place fields and is used primarily in charts and other places where the entire set of place fields will not fit. The default is [CITY], [COUNTY], [STATE].
"
I'm afraid it is so long ago that I used TMG that I don't recall how I used these options within the program.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
avatar
Linda Reinfeld
Famous
Posts: 124
Joined: 04 Nov 2014 17:34
Family Historian: V7

Re: Sort dates with plugin for TMG import

Post by Linda Reinfeld »

Mike - In the instructions for the Plugin - Import Fact Sort Dates - this is the part about places:

Then at Step 7: Option screen 3 ensure Place Levels has every item ticked that your project uses, such as Addressee, Detail, Postal Code & Phone, but NOT Commas When Missing.

This seems to identify what is needed quite well.

Old-style dates - this refers to the dates that were created before the change to the Gregorian calendar. From the TMG help:
Dates between 1583 and 1752 may be considered as Old Style dates. The Gregorian calendar was introduced by Pope Gregory XIII in 1582 to correct an error in the Julian calendar. In that year, the day that would have been October 5 was called October 15 and other changes were made to prevent future errors. In addition, the beginning of the year was moved from March 25th to January 1st.

The US (and Great Britain) didn't adopt the new date format until 1752. By that time there was a difference of 11 days.

So a date is entered into TMG as Birth - 20 Jan 1680/81 . This actually means the person was born in 1681, but the date of record was 1680. The correct date is in FH, so there isn't a problem.

Thanks -

Linda
Post Reply