* Advice needed with handling fields on importing

Importing from another genealogy program? This is the place to ask. Questions about Exporting should go in the Exporting sub-forum of the General Usage forum.
Post Reply
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Advice needed with handling fields on importing

Post by Kim Travis »

Starting long process of seeing if I can port data from Peditree and have three questions at this stage.
- Where is there a Gedcom 5.5 standard description, as used in FH, so I can know what field names FH is expecting?
- Within Sources I use a field called Note, yet on import into FH the data from this field have been put into a field variously called "Actual text" or "Text from Source" (I assume these two names are synonyms but haven't yet found information of field name synonyms) rather than the Source Note field. Any idea why this should be?
- In FH Places seem to exist at the same level as Sources and Individuals and Marriages. Any tips for dealing with this as I'm importing from software that doesn't manage Places like this. I got thousands of errors on import "Excluded: link to non-existant Place record (0)"

Thanks,
Kim
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

Welcome to the FHUG Kim.

As a newcomer I advise you study how_to:key_features_for_newcomers|> Key Features for Newcomers and all its links into our Knowledge Base which includes a link to glossary:gedcom|> GEDCOM that links to a downloadable GEDCOM 5.5 PDF.

Where FH puts input data depends entirely on the GEDCOM data structure exported by PediTree.
Its Getting Started Guide says in Chapter 6 F Page 48 the "choice of GEDCOM 5.5 is for transferring information from PediTree databases to other family history programs that use this standard." Did you do that?

The reason the PediTree Source Note inputs to Text From Source is that the PediTree GEDCOM is using the TEXT tag instead of the NOTE tag in its exported SOUR records.

FH does implement a non-standard extension to GEDCOM where Place records are linked to every Place field with the same Place name. However, those records will get created automatically on import provided the GEDCOM PLAC field tags are used correctly. If necessary you can post snippets of GEDCOM and we can analyse what is going on.

I hope to install the PediTree free trial and that may give me some clues as to what its exported GEDCOM 5.5 looks like.

I have done that and migrated the Elton55 sample database, some PLAC tags have no place name and those are causing the EXCLUDED: link to non-existent Place record (0) exceptions, so they can be ignored. Otherwise, there was just a minor Child to Family link validation error.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Re: Advice needed with handling fields on importing

Post by Kim Travis »

Yes I'm using Gedcom5.5 (or trying to, but that's another story). Thanks for your help. I tried to cherry-pick which bits of FH help to read, especially the introductory stuff, as I'm very familiar with most of the concepts, just unfamiliar with how FH deals with them. I guess I need to read it all and get back only on the questions that this doesn't answer,
Thanks again,
Kim
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

I have experimented further with the Elton55 database.

It would be advisable NOT to Convert GRO Quarter Dates when exporting the GEDCOM 5.5 file.
That allows them to be corrected into valid FH/GEDCOM Quarter Dates using a Plugin, which is not possible if they are converted by PediTree.
I have updated my Fix Date Fields Plugin to cater for those PediTree Quarter Dates but it is not published yet.
See how_to:fixing_dates_date_phrases|> Fixing Dates and Date Phrases.

BTW: In case it helps your understanding, FH operates entirely within the concepts and structures of GEDCOM 5.5 to the extent that the FH Project database file is a GEDCOM file.

I have assumed that you are using File > Project Window > New Project > Import a GEDCOM File to create your FH Project from the exported PediTree GEDCOM file. See how_to:index#importing_to_family_historian|> Importing to Family Historian.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Re: Advice needed with handling fields on importing

Post by Kim Travis »

Thanks for the tips. Yes that's how I created my FH project, but I'm just exploring and learning at the moment how I'm going to manage the dozens of issues that porting the data will create - some will need fixing before loading into FH and some after. By the time I come to moving the data for real I'm sure you extended plug-in will be available. Thanks for the info re Quarter dates - I was wondering about that,
Kim
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

Those are excellent tactics Kim. The fixing before and after strategy is widely used by most migrants into FH.

BTW: You may not have spotted that FH imports some data as UDF that are partially hidden.
So some of your PediTree data may have imported into FH without your knowledge.
To see UDF open the All tab and expand the data so no [+] marks remain and the UDF appear with * bullets:-
All tab UDF
All tab UDF
AllTabUDF.png (17.69 KiB) Viewed 11455 times
See how_to:handling_unrecognised_data_fields|> Handling Uncategorised Data Fields (UDF) for full details and advice on how to locate all UDF quickly.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Re: Advice needed with handling fields on importing

Post by Kim Travis »

Thanks for all your advice - much appreciated. I'm nearly there with my first Peditree database. Then I'll work with Peditree for a bit to check my choices of handling non-standard fields were OK. Then I'll have to start thinking about merging in my 2 other databases.... I've been putting off this whole task for about 10 yrs, but so far so good,
Kim
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Re: Advice needed with handling fields on importing

Post by Kim Travis »

I've managed to import my 3 Peditree databases into a single FH database at last. Today I ran the Fix Date Fields Plugin. It did a really good job of fixing dual years and quarter dates that Peditree specifies in an incorrect way. However, when a dual year or quarter date was part of a date range it didn't cope very well. For example it changed
"BET 03 Jul 1840 - Q1 1849" (the field includes these quotation marks)
to
"Frm bet 03 Jul 1840 to q1 1849"
From what I can gather the best result would be
btw 03 Jul 1840 and q1 1849

Date ranges without a dual date or quarter date were handled fine. I'm not familiar with the scripting language used for Plugins, but I may need to get into it. Alternatively, perhaps this represents a further opportunity for improvement to this excellent Plugin.

For now I'll go back to the backup I made before running the Plugin

Kim Travis
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Re: Advice needed with handling fields on importing

Post by Kim Travis »

...forgot to mention that another rare form of incorrecty fixed date is illustrated by this (actually the only) example. After running the Plugin the "fixed" date is
"Bef Q1 1647/48"
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

Hi Kim,
The dates "BET 03 Jul 1840 - Q1 1849" and "BEF Q1 1647/48" are in an eccentric format.

The "quotation marks" are the way that GEDCOM allows invalid dates to be held as a text string known as a Date Phrase.

The problem is that Q1 1849 is itself a Date Range that runs from 1 Jan 1849 to 31 March 1849.
So those BETween and BEFore dates are both ambiguous and invalid GEDCOM format.
e.g.
What does Between 03 Jul 1840 and from 1 Jan 1849 to 31 March 1849 mean?

If you don't have too many, then it would be best to manually correct them either in PediTree, or FH which is probably best as it will advise whether the Date format is acceptable or not.
If too many for that, then you can use the Search and Replace Plugin on just Date fields to change say Q1 to 1 Jan but ask for advice how to make that work as the "quotation marks" must also be removed.

Due to the ambiguity explained above, it is not obvious how the Fix Date Fields Plugin could convert those unusual dates reliably.
i.e.
Should it be Between 03 Jul 1840 and 1 Jan 1849 or maybe Between 03 Jul 1840 and 31 March 1849 ?
And should it be Before 1 Jan 1647/48 or otherwise Before 31 March 1647/48 ?

BTW: If you don't Close FH you can revert to the pre-Plugin state by using Edit > Undo Plugin Changes.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Re: Advice needed with handling fields on importing

Post by Kim Travis »

Yes is see. (I could quibble that a quarter is no more a date range than a year is - they are just a different unit of time and fail because the only unit of time recognised in FH is the day, and they are undeniably a range of days.) Thanks for the explanation and the suggestions about how to fix it. But I would say that this argument should not apply to date ranges which include a dual date, for example "BET 09 Feb 1632/3-28 Oct 1637" could be converted into a valid range - couldn't it? Or does FH view a dual date to be a date range (I hope not)?

Thanks
Kim
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

The problem there is twofold.
1st that is not a valid double date as the year should be 1632/33
2nd is the lack of spaces around the hyphen -

BET 9 Feb 1632/33 - 28 Oct 1637 is OK
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Kim Travis
Diamond
Posts: 51
Joined: 04 Jan 2018 21:17
Family Historian: V6

Re: Advice needed with handling fields on importing

Post by Kim Travis »

OK, I'll edit these few hundred manually,

Kim
User avatar
Tony Jones
Diamond
Posts: 65
Joined: 14 Oct 2016 09:27
Family Historian: V7
Location: Berkshire, England
Contact:

Re: Advice needed with handling fields on importing

Post by Tony Jones »

I found I had a few dozen date fields in quotes from an old import and recently use the Search and Replace plugin to sort them out. I'm sure there's a clever way to do it, but I just did it in several simple stages. The hardest was converting double space to single space - I figured out %s meant any white space character, but when I keyed a space in the 'to' box it didn't display, leaving me very confused!
Using an iMac Ventura Intel via Parallels v19 / Win 10 64-bit
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

To explain Tony's reference to %s that is an LUA Pattern Mode code for any number of white space characters.

The drawback is that it would match every solitary space character too, and replace that with one space, so rather unnecessary.

It would have been better to use Plain Text Mode and enter two space characters in the Search box.

Although the space characters are not visible, you can tell they exist by putting the cursor in the box and clicking the left & right arrow keys and notice the cursor move over the space characters.

Finally, if Date fields are in "quotes" they are Date Phrases and will not become fully fledged Dates unless the "quotes" are removed, or the Fix Date Fields Plugin can convert them.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Tony Jones
Diamond
Posts: 65
Joined: 14 Oct 2016 09:27
Family Historian: V7
Location: Berkshire, England
Contact:

Re: Advice needed with handling fields on importing

Post by Tony Jones »

I did, indeed use %s%s to match two white spaces! Coming from MS Word, I was used to %s just meaning the space character, so I originally tried to swap %s%s for %s!

I'm just beginning to scratch the surface of what FH can do under the hood, so expect lots of questions!
Using an iMac Ventura Intel via Parallels v19 / Win 10 64-bit
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

Sorry Tony, I confused everyone by saying %s matches any number of white space characters when it only matches one.
You are correct it needs %s%s to match two spaces, but two spaces would work just as well in Plain Text Mode.
The Plugin Help and Advice for LUA Pattern Mode refers you to plugins:understanding_lua_patterns|> Understanding Lua Patterns that are similar to Regular Expressions but are not exactly the same.

BTW-1: There is no need to repeat the immediately preceding message you are replying to.

BTW-2: Although you have posted a few times I don't recall advising you study how_to:key_features_for_newcomers|> Key Features for Newcomers which although you are not exactly a newcomer does refer to some of the newer FH V6 features. Lots of questions expected.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Tony Jones
Diamond
Posts: 65
Joined: 14 Oct 2016 09:27
Family Historian: V7
Location: Berkshire, England
Contact:

Re: Advice needed with handling fields on importing

Post by Tony Jones »

Thanks Mike. I have read a number of things on the site, but I'm one of those who needs to fiddle with software to get the hang of it, as well as just reading.
Using an iMac Ventura Intel via Parallels v19 / Win 10 64-bit
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Advice needed with handling fields on importing

Post by tatewise »

That is why I usually say 'study' and not 'read', because many of the topics have tutorials or videos, and benefit from practical experiments to reinforce the learning process. So 'read' and 'fiddle' away, but then 'read' again.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply