* FTM to FH6 date format corrections

[LarryC]

I have many Fact problems after converting an approximate 100k person FTM GEDCOM to be used in FH6. Please help me with the following:

• (1a) Change all Fact Date entries "UNKNOWN" to "unk".
  
  (1b) Change all Fact Date entries "BET" to "btw" and "AND" to either "and" or "&".
  
  (I have tried find and replace using all filters without success)

Thanks,

Larry Chesebro'

[tatewise]

I suggest you use File > Backup/Restore > Small Backup beforehand, just in case!!!
Remember that if the changes below are unsatisfactory they can be reversed using Edit > Undo Plugin Updates before closing FH.

(1a) Change all Fact Date entries "UNKNOWN" to "unk".

Be aware that this only changes the "Date Phrase" text and will not become a recognised valid Date format in FH.

Use the Search and Replace Plugin.
Ensure Plain Text Mode and Case Sensitive and Whole Words are all selected.
Ensure Search Scope: is All Records & Events/Attributes.
In Basic Filters untick every option except Date fields.
On the Extra Filters tab select No Date Phrase Warnings lower right, otherwise you will get a warning for every single change.
Back on the Major Options tab set the following without "string quotes":
Set Search: box to UNKNOWN
Set Replace: box to unk
Click the Search & Replace button, and click Replace button until satisfied that the changes are OK, then untick Confirm every item found and click Replace once.
When complete a Result Set of all changes will be listed.

(1b) Change all Fact Date entries "BET" to "btw" and "AND" to either "and" or "&".

I assume you are trying to convert Date Phrases to a valid Gedcom Date format.
On inspecting your Chesebro Gedcom it has Dates with the following example format:
"BET <day/month> AND <day/month> <year>"
This has two problems ~ BET should be BTW, and the first <day/month> needs a <year> so the valid format is:
BTW <day/month> <year> AND <day/month> <year>

If that is what you want to achieve then use the Search and Replace Plugin again.
Ensure LUA Pattern Mode is selected.
Ensure only Date fields are ticked, and on Extra Options tab Date Phrase Warnings is selected.
Back on the Major Options tab Copy & Paste the following:
Set Search: box to "BET (.*) AND (.-) (%d%d%d%d)" including the "string quotes"
Set Replace: box to BTW %1 %3 AND %2 %3 without string quotes
Click the Search & Replace button and proceed as above.
To understand more about LUA Pattern Mode see plugins:understanding_lua_patterns|> Understanding Lua Patterns under Constructing a Pattern.

[LarryC]

Thanks!

I successfully converted Date Phrase "UNKNOWN" to "unk".

The "BET" Date Phrase change process "worked" but with an unwanted result. I substituted btw for the replacement BTW entry and the result was "between", lower case as desired but whole word as opposed to abbreviated. I did not save the result and would prefer the abbreviated result.

At 78 years of age I have some problem learning the LUA language. I did carefully read the reference you shared but I am concerned of my ability to pursue "programming" efforts at this time. I would prefer to be self-sufficient but for now please accept me as a "user" only.

With that in mind, trying to complete the 3rd portion of my request, which I do agree is preferred, may I add to the original request these requests for your help:

• (1.1) Is there an option or process to remove unwanted Date Phrase phrases? If so, would that allow the BTW result to be abbreviated lower case? Or is there at least a way to force the abbreviated lower case result of the BET change?
  
  (1.2) How can I change dates like "1-16 Jun 1980" which FH6 apparently requires it entered "1 Jun 1980 to 16 Jun 1980"? And have FH6 automatically record future similar dates when entered "day-day""month""year"?
  
  (1.3) How can I change dates with known days and months but not years, e.g. "22 Dec" to something like "22 Dec unk yr"? And have FH6 automatically record future similar dates when entered: "day""month""?" ?

[tatewise]

Larry, I have put these postings into a new thread of its own.

(1b) The "BET" Date Phrase change worked exactly as expected.
What you are seeing is NOTHING to do with using btw instead of BTW (you could also use between or BeTwEeN or BETWEEN or Btw).

As I have said before, FH enforces GEDCOM Date formats, which are case insensitive.

What you are seeing in the Plugin Result Set is the LONG Date display format where all GEDCOM abbreviations including Months are expanded into full words.
So the GEDCOM Date BTW 23 JAN 1878 AND 30 SEP 1901 is displayed as between 23 January 1878 and 30 September 1901.
If you double-click on that displayed Date you will see the COMPACT display format in the Facts tab.

The Search and Replace Plugin will allow the Replace: box to use either LONG or COMPACT format for dates in upper or lower case. So Between %1 %3 and %2 %3 works the same as BTW %1 %3 AND %2 %3.

In FH there is a degree of flexibility in how Dates are displayed, but the underlying format is still the GEDCOM compact form. See the FH Help > Family Historian Help > Index tab, enter Dates, and then inspect the Dates Format page.

You may even see Dates such as BTW APR AND JUN 1830 displayed as Q2 1830 and they can be entered in that format. But that is governed by Tools > Preferences > General > Use Quarter Date Display Format. FH does allow a variety of Date entry formats, but they cannot be extended or customised. I am sure FTM is much the same.

FYI: LUA Language is different from LUA Patterns and I don't expect you to understand either, but other users who read these threads like to know more about the topic being discussed.

(1.1) If you investigate Dates as discussed above you will find the btw form is quite common.

(1.2) To change Dates like "1-16 Jun 1980" use the Search and Replace Plugin as for the BET to BTW change.
Set Search: box to "(%d+)%-(%d+) (%a+ %d%d%d%d)"
Set Replace: box to from %1 %3 to %2 %3
You cannot customise FH Date entry formats. Does FTM allow you to customise Date entry formats?

(1.3) To change dates without years use the Search and Replace Plugin as for the BET to BTW change, but with No Date Phrase Warnings selected on Extra Filters tab, because they will still be Date Phrases.
Set Search: box to "(%d- ?%u%a%a)" to match "23 Jan", "4 JUL", and "DEC" but not "unk" or "UNKNOWN".
Set Replace: box to "%1 unk yr"
You cannot customise FH Date entry formats. Does FTM allow you to customise Date entry formats?

Having completed those changes I suggest you run the Find Date Phrases Plugin to list all Date Phrases not recognised as GEDCOM Dates. This will include "unk" and "22 DEC unk yr", etc, plus a few others that may need adjusting by hand.

One overriding consideration is how Dates will appear in Narrative Sentences in Reports and in Diagram boxes, etc.
You might like to create a few sample Reports and Diagrams before finalising your Date Phrase styles.
e.g.
She died "22 DEC unk yr".
She married Lambert "unk".
are not very readable, so the following may be better:
She died "on 22 DEC year unknown".
She married Lambert "in year unknown".

I suggest you eventually run the Show Project Statistics Plugin and check its Exception Report because I spotted unusual years such as 999.

I have run that on your Chesebro Project and that shows another Date format problem:

(1.4) BET 21 AND 29 JUN 1980 is taken as between 0021 and 29 Jun 1980
i.e. The first date is the year 0021
Use the Search and Replace Plugin as for the BET to BTW change above.
Set Search: box to between 00(%d%d) and (%d+) (%a+ %d%d%d%d)
Set Replace: box to BTW %1 %3 AND %2 %3

I intend to document these Date correction techniques in how_to:fixing_dates_date_phrases|> Fixing Dates and Date Phrases.

[LarryC]

I reviewed the Dates help topics and understand the formats but do not understand the options - if there are any extensive default options.

FTM options include default settings for full or abbreviated, "day, month, year" or "month, day, year"; various variations for before, after, and between; and approximate including "c", "ca", "circa" or "approx."so regardless of how entered, the result is how optioned. And then, whether in reports or charts, the format result is the same format. Really simple and effective once defined.

My preference for all formats would the Abbreviated as defined with the "ca." for approximate and no Quarterly option.

Am I correct to learn the only default options are for the circa, quarter date, and Date Phrase formats?

FTM does accept dates as 21 - 30 Dec 1990 and are apparently problematic in FH. I would expect FH to recognize an entry of 21-30dec1990 at least as 21 Dec - 30 Dec 1990 not as it does - 21 A.D - 30 Dec 1990. Apparently the initial month and year must also be entered which seems not very efficient.

The date 999 is a good year as all years beginning with 1, for the 1st year A.D. should be OK.

[tatewise]

Firstly, I advise that you spend some time becoming more familiar with all the features of FH.
Use the how_to:key_features_for_newcomers|> Key Features for Newcomers especially the Documentation & Tutorials.

The PDF book Getting the Most From Family Historian 5 covers Date entry in Chapter 2.
See also the FH Help > Family Historian Help > Index tab, enter Dates, and then inspect the Date Entry Assistant Dialog page.
FH offers many Date entry formats similar to FTM, but just like FTM you cannot add any new Date entry formats.
Regardless of how entered, Dates are saved in GEDCOM Date formats, which improves exportabiity to other products.

The global Tools > Preferences options for Dates are the ones you mention.
The Date format used in Property Boxes, most Reports, and by default in Diagrams is the Compact Date format that you seem to prefer.
However, in Narrative Reports the Long Date format is used by default, because it reads better in narrative sentences, but can be changed in Tools > Fact Types to use any Date format for each separate Fact.
In Diagrams, each Diagram Text Scheme can be customised to use any Date format.

Agreed, FH does not accept dates such as 21 - 30 Dec 1990 in the same way as FTM.
But if FH did, they would be converted to GEDCOM Date format btw 21 Dec 1980 and 30 Dec 1990.

Have you tried any more of the Search and Replace changes?

[LarryC]

FH is not very intuitive and the introduction and help information is very difficult to interpret for just a "user" - but I am trying having read a fair amount of what you mentioned but, the problem is that rather being instructed how to do something the help is how it results or the end result. And, this is the biggest problem:

• "Note that ƒh V6 does not yet include the Getting the Most From Family Historian book nor its Tutorial Files, but you can download the V5 copies, and much is still applicable".

I believe it to be counterproductive to use V5 reference when V6 may not be compatible causing even more confusion.

Without the great help of FHUG, or at least you as a FHUG representative, FH is very difficult to use in more detailed and extensive databases. And, as is so exemplified with all the plugins, FH requires intricate modifications just to interface or be exchanged with other GEDCOM driven applications and uses. I am confident that in time I will appreciate many of the "extra" features of FH but for now I miss some necessary simple options and features, e.g. a rudimentary spell checker.

I do additional FTM to FH6 "clean-up" for all, or at least most, of the returned list upon completion of a change. I should finish the original request for help (3) tomorrow or Thursday.

My continued thanks,

[tatewise]

Getting the Most From Family Historian 6 and new Tutorials are promised to be available in FH V6 free upgrade soon.

The Plugins feature was requested by FHUG users, and allows new capabilities to be added without Calico Pie needing to issue new software updates. It is extremely powerful and useful.

If other GEDCOM driven applications were actually GEDCOM compliant then things would be a lot easier.
But the GEDCOM specification allows user-defined structures to be added, so each application has its own dialect, which complicates things.

Here we all agree that some rudimentary word-processor features such as a spell-checker and font styles are essential.
See and Vote for:
Wish List Ref 20 Word-processing facilities for note and source text (italics, bold, tables, etc)
Wish List Ref 79 Spell Checker

[LarryC]

I tried reviewing more of your references. I understand a bit more but have yet to find FH Help instructive enough to improve my understanding of accepted entry defaults - either FH or GEDCOM. The more I learn the more frustrated I become. So.....for now I hope to depend on your knowledgeable advice and support as I try to convert from FTM to FH.

I cannot do the (1.3) dates w/o years. The Replace Value result is "?1 year unknown" - apparently no month and year is retained.

I ran both Find Date Phrases and Show Project Statistics which generated or added to these questions:

• (1.3.1) What is the GEDCOM standard or default entry for an unknown date? If none, why then are blank event dates, e.g. Birth, Death, and etc. reported in the Exception Report of the Show Project Statistics?
  
  (1.3.2) What is the GEDCOM standard or default entry for the unknown Gender of a child?
  
  (1.3.3) Why are persons either living or presumed to be living but at least not known as dead listed on the Exception Report of the Show Project Statistics?
  
  (1.3.4) Why am I, as the submitter, listed in the Exception Report of the Show Project Statistics as "Ident Permanent Record Nu: 1"

[tatewise]

cannot do the (1.3)
I suspect because you have Replace: box set to "?1 year unknown" instead of "%1 year unknown".

I presume that the (1b), (1.1), (1.2) and (1.4) Date fixes are all OK?

Show Project Statistics
The first thing to say is that some of the Exception Report Items may be ignored.
They are simply things to review based on past experience, and can either be corrected or ignored as necessary.
See the Plugin Help & Advice pages for more information.

(1.3.1) Are you referring to the reports of No Birth, Baptism, or Christening Event and No Death, Burial, or Cremation Event nor Living Flag? They are NOT saying there is no Date they are saying there is no Event at all. Every Individual must have been born and died, unless they are still living, so they ought to have an Event to say so or a Living flag, even if there is no data. But if you are happy to leave them out, then those reports can be exclude by unticking them on the Plugin Options tab.

(1.3.2) If the Gender is unknown then it should be left undefined. If the reports are all OK, they can be excluded as above.

(1.3.3) They are listed because living Individuals should have a Living flag, so that Privacy options can be applied to hide their details in Reports, Diagrams, etc, that might be published publicly. Exclude the reports as above.

(1.3.4) The Permanent Record Number field (and the Automatic Record Id field) exist in every record, but are rarely used and hidden away on the All tab. So the report is simply bringing them to your attention.

[LarryC]

I apologize for the late response. For some reason I did not get an email notice of your response. I just was double checking today when I found this. I kept busy making some simple FH Fine and Replace efforts and made a WEB page to see the value. After more than 60 hours of FTP effort over a 100mb bandwidth Internet connection I was able to post 66,294 total files for my family's 100 generation (limited by FH from more than 110 recorded) ancestral record with an 11 generation immigrant ancestor descendant record! FH should consider partnering with some entity like TNG and Simply Hoisting (they partner for producing and hoisting GEDCOM based genealogy records) for really compact, detailed, feature rich WEB page options in FH. MY 2 database with over 110 generations and much more features only requires 2,932 files. http://chesebro.info/fh/index.html

I did make the mistake "?1 year unknown" I succeeded with that effort using the correct input.

For the Show Project Statistics questions:

I was referencing dates for birth, death and etc. I will have to review later why there is not at least an "unk" birth entry assuming that will create the Event. For privacy reasons I had hoped there would be an option to "privatize" all living persons only when exporting to a GEDCOM saving time, effort and space not having to enter something for every person. I did not want to do all export to GECOMs with living tags as my destination online locations do that for me and the vast majority of personally shared GEDCOMS are not privatized. What are my options?

What is the GEDCOM 'standard' for unknown gender? FTM did accept "?" and in turn the online locations also. Can the "?" or similar qualifier be applied in FH via a plugin?

Anytime you are ready to proceed with the original problems (1) and (2) is up to you as I have to take time to review if anything should, or can, be done with the Show Project Statistics items.

[tatewise]

For privatisation there is the Clean Living People Plugin that will privatise any Individual with the Living flag, or with a Birth Date after a nominated year and no Death Date, or no dates at all, plus various other options.
See also fhugdownloads:livingpeople|> Downloads and Links ~ Living People Identification.

If gender is unknown then GEDCOM requires the Sex field to be left blank, because the only valid values are Male and Female, and other values such as ? are not allowed.

Will get back to original problems (1) and (2) tomorrow.