* GEDCOM 5.5.1

[wianb]

Did not quite know where to ask this so please do move it if needed.
Question - when or will FH adopt/change to GEDCOM 5.5.1. Are there any plans to do so?

[Jane]

You would have to ask Calico Pie, but they are not normally forthcoming with any future plans. I suspect as 5.5.1 has never been generally adopted, FH will not be in any hurry to implement it as there are so few other programs which use it.

They did join the https://fhiso.org/ but that group has not actually put out any standards yet

[tatewise]

The adoption of GEDCOM Draft 5.5.1 is a bit mixed, and even the implementation of GEDCOM 5.5 is less than perfect in many products.

Why are you interested in GEDCOM Draft 5.5.1?

[JohnnyCee]

I am not sure why it matters to the OP whether FH uses GEDCOM 5.5 or GEDCOM 5.5.1. As a developer, I'd prefer that FH use 5.5.1. It would make things easier for me if all the major GEDCOM products use the same version. When things are easier for developers, that translates into benefits for all users.

GEDCOM 5.5.1 is called a draft because the document says so, but practically speaking, it's not a draft. Shortly after publishing the document, FamilySearch began using 5.5.1 in production software (PAF 5.0) and so the same people who wrote that it was a draft were using it as if it was not a draft.

Regardless of whether its a draft or not, 5.5.1 has several less flaws than 5.5, and that should be reason enough to use it.

Tamura Jones, Keith Riggle, and others consider it the defacto standard:

https://www.tamurajones.net/GEDCOM5.5.1IsntADraft.xhtml
https://genealogytools.com/why-all-gene ... com-5-5-1/

[Ron Melby]

Among other things the LDS based FamilySearch has supported it since it came out, for all practical and real purposes, it IS the standard, only a few outliers do not support it, and lets face it, running unicode but not supporting 5.5.1 is sort of non-standard anyhow. it would be so much easier to do real world stuff like Addresses and phones and emails and lat longs portably in the gedcom, because after all, you could transform it and note it out to 5.5 on an export, by a check box if necessary....

[tatewise]

I agree it would be better if all products worked to the same GEDCOM specification, but at present I am not aware of any product that fully implements either GEDCOM 5.5 or 5.5.1 and sadly that includes FH

Apart from the arguable misuse of UTF8 character encoding, in my experience more current (not defunct) products work closer to GEDCOM 5.5 than to 5.5.1 that requires the fields FACT, EMAIL & WWW in both Facts & Repositories, FILE in Media records, and MAP/LATLONG geocoding.

[AdrianBruce]

... As a developer, I'd prefer that FH use 5.5.1. It would make things easier for me if all the major GEDCOM products use the same version. ...

I think we'd all totally agree with that.

... Regardless of whether its a draft or not, 5.5.1 has several less flaws than 5.5, and that should be reason enough to use it. ...

But we know also that there are several flaws with the 5.5.1 draft - the introductory list of changes is so far from reality as to make it clear that the manual, at least, for 5.5.1 was well short of being the finished product. Also there is a complete muddle over the generic EVEN(T) where the example given blatantly contradicts the standard. This is possibly in connection with the proposed generic FACT tag. Does this matter? Well, quoting Mike Tate of this parish:

... it seems many developers have worked from the faulty example rather than the formal specification ...

When this discrepancy was found, several bloggers stated that it was "obvious" which one of the two contradictory elements (definition and example) was correct. Obvious? Like heck - it's utterly arbitrary which you choose, and only a psychic could divine which was the intended final format.

... Tamura Jones, Keith Riggle, and others consider it the defacto standard ...

Standards are not made by small external groups deciding for themselves.

Look, please don't misunderstand me - I'd prefer that we had 1 standard, and theoretically 5.5.1 should be closer to what we need. I also have nothing but admiration for anyone who tries to write their software to deal with the multiplicity of variations of GEDCOM. But what I do want to do is highlight that anyone with a concern for rigour and risk-reduction is likely to want to approach 5.5.1 with a very long stick.

[JohnnyCee]

But what I do want to do is highlight that anyone with a concern for rigour and risk-reduction is likely to want to approach 5.5.1 with a very long stick.

I am aware there are flaws in the 5.5.1 spec. There are flaws in all of the GEDCOM specs. Still, developers somehow manage to make products that work well. 5.5.1 has features that no one wants to give up, like support for UTF-8. In the other direction, 5.5 has support for features that everyone wants to forget, like media objects encoded in the BLOB record. (The spec for that is flawed and impossible to implement reliably, and I am not aware of any programs that support it.)

[tatewise]

John, apart from GedStar Pro and Legacy (partially) what other products support GEDCOM 5.5.1 fairly well?

The other products that I have come across, either support GEDCOM 5.5 fairly well, or don't support GEDCOM well enough to identify which variant it is. They all say it is 5.5 in their header.

[JohnnyCee]

Mike,

Most programs support multiple versions for import. Some support multiple versions for export. None provide full support, as far as I know, and that may not really be possible in any version of the spec given the flaws in each of them.

FH supports some 5.5.1 features, for example, though I think that’s only on import. I use your export plug-in so I am not very familiar with the native FH GEDCOM export feature.

My bias for 5.5.1 is not based on current market adoption, it’s more practical: 5.5.1 has some useful and widely adopted features that no one wants to lose and many programs adopt some or all of them. It doesn’t make sense to start from 5.5. We would all benefit from all products at least starting from the same spec, even if that spec is flawed and even if no product implements it 100% fully/correctly. So, let’s agree on 5.5.1.

I know it’s mostly wishful thinking, but “it’s pretty to think” that we could all go to 5.5.1 and then—dare we think it—agree on some corrections/amendments.

[tatewise]

I am aware of 'others' claiming 5.5.1 is the de facto standard, and it does have some advantages.

FH supports the 5.5.1 encoding in UTF8 format for both import & export.
On export, the Destination: Family Tree Maker option converts each Media record tag _FILE to the 5.5.1 tag FILE although that is not documented.

You say "5.5.1 has some useful and widely adopted features ... and many programs adopt some or all of them."
Could you identify those products, as apart from UTF8, I don't believe its features have been adopted widely nor by many programs.
From work with my Export Gedcom File Plugin, I know of support in GedStar Pro (most features), Legacy (some features), and MyHeritage (few features), but not aware of any options in any others that genuinely support 5.5.1 for import or export, and most do not even support 5.5 very well. The first step would be to use the Header VERS tag correctly.
With enough evidence of wider support, I was hoping to influence Calico Pie to improve FH support for 5.5.1.

The FHISO could have clarified GEDCOM features based on de facto usage so that developers had a flawless standard to work to when adding features, but they have not done so.

[JohnnyCee]

Mike,

MAP is probably second after CHAR UTF-8 as the most widely-adopted 5.5.1 feature. A "non-strict" FH GEDCOM 5.5 export uses MAP but under a custom _PLAC record. FTM, RootsMagic and Legacy export to GEDCOM 5.5.1 and use MAP.

From what I can tell, they all follow the 5.5.1 rules for the format of the LATI and LONG values, i.e, a "direction" character followed by a fractional degree. Legacy (and maybe others, I didn't check) will render values that are more precise than allowed by 5.5.1. For example, I have a test file exported from Legacy with "4 LATI N44.9494438888889" but 5.5.1 specifies "Size=5:8" so the value should be limited to "4 LATI N44.9494". The 5.5.1 limits are too restrictive, a typical but unfortunate flaw. With the example coordinates from a Legacy file, the truncated coordinates were 29.5 ft (~9.5 m) away from the non-truncated coordinates. That could make a difference when trying to locate a poorly-marked grave, for example.

The other new tags (EMAIL, WWW, etc.) may be more popular than MAP; I didn't do a survey. I discount those items because many programs have non-standard ways of rendering that data, such as "_URL" and the like, but they may actually be more widely adopted than MAP/LATI/LONG.

I assume that Calico Pie and the other major genealogy program publishers see insufficient direct benefit to making changes to their GEDCOM handling. For Calico Pie, it's unlikely that announcing a change to GEDCOM 5.5.1 is going to trigger a surge in sales. The lack of specific motivation for publishers leads to the awful situation we have now where two stagnant, flawed standards have been allowed to stifle progress for 20 years.

Some agitators have tried to throw out GEDCOM and replace it with something new and/or markedly different. The results suggest that won't fly; software publishers have not adopted any of the proposals. My suggestion is much more modest: get the major players to 5.5.1, then make some refinements to it. Baby steps, and even that probably won't gain traction. <sigh>

[JohnnyCee]

Mike,

You questioned whether it was accurate for me to say, "5.5.1 has some useful and widely adopted features ... and many programs adopt some or all of them."

As I described above, Family Historian, Family Tree Maker, RootsMagic and Legacy support two of the most important GEDCOM 5.5.1 features, CHAR UTF-8 and MAP/LATI/LONG. While that may not be "many programs", those are the four leading programs for English-speaking users, and some people whose primary language is not English use them.

FTM, RM, and Legacy have adopted 5.5.1. Whether they use all the 5.5.1-specific features doesn't matter because (A) they use some of them and (B) they use the most important feature, CHAR UTF-8.

Here are some quick comments on other programs.

My test files for Reunion use GEDCOM 5.5. I do not have access to the program so I do not know if Reunion has an option to export to 5.5.1. Reunion does such a poor job of conforming to the GEDCOM 5.5 spec that it's almost not worth including in the discussion. Its position as the leading Mac-based program is the counter-argument.

As one small example of its GEDCOM issues, the test files I have use "1 CHAR IBM WINDOWS", which is invalid. Perhaps Reunion also supports valid CHAR values and the users who created my test files did not configure the options correctly. Even so, Reunion should not use "1 CHAR IBM WINDOWS"; if they are going to offer an option that does not comply with the GEDCOM 5.5 spec, they should implement UTF-8!

Other issues are incorrect use of CONT tags and many custom tags that are not prefixed with "_".

I assume that Reunion's position as the most popular program for Macs means the publisher can get away with ignoring the program's GEDCOM issues.

GRAMPS is the leading FOSS genealogy program. It exports to GEDCOM 5.5.1 and UTF-8; in the version I have (which is recent but may not be the most recent), there are no other options. It supports the MAP/LATI/LONG tags and appears to implement them properly except it allows more precision than specified in the GEDCOM 5.5.1 spec.

Ancestry.com Family Trees claims to export to GEDCOM 5.5 but uses "1 CHAR UTF-8". I don't have any direct experience with this service.

As of May 2018, Brother's Keeper claims to export to GEDCOM 5.5 but uses "1 CHAR UTF-8" and several 5.5.1 tags like EMAIL, FAX, WWW, and MAP/LATI/LONG. I do not have a recent Brother's Keeper GEDCOM export so this information is second-hand.

Heredis claims to export to GEDCOM 5.5 but uses "1 CHAR UTF-8" and includes MAP/LATI/LONG tags. The LATI/LONG values do not conform to the GEDCCOM 5.5.1 spec: they omit the direction character and use negative values for west and south.

I have limited test files for MyHeritage Family Tree Builder. The file I have is not from a recent version. It does not use UTF-8, but it does include several GEDCOM 5.5.1 tags (WWW, EMAIL, FAX).

rootstrust exports to GEDCOM 5.5.1 with 1 CHAR UTF-8. It also supports MAP/LATI/LONG using the 5.5.1 rules. Like other programs, it will exceed the LATI/LONG size to include more precise locations.

I think the above is enough information to support my claim that many program support GEDCOM 5.5.1 features.

[Valkrider]

For information

Reunion 11 only exports Gedcom 4.0, Gedcom 5.5 or Ancestry.com

Ancestral Quest only exports Gedcom 4.0 and Gedcom 5.5

MacFamilyTree & Heredis just say Gedcom export without giving a version.

[tatewise]

Thank you ~ that extends my understanding of what is available.
But as I have said a few times, apart from UTF8 the support is somewhat patchy and sometimes non-conformant, which makes interchange via GEDCOM virtually impossible, especially when the extensive use of custom tags is thrown into the mix.

I admit I had overlooked the support for MAP/LATI/LONG but not sure they all conform to 5.5.1.
e.g. FH and TNG add the MAP/LATI/LONG structure to non-standard _PLAC records instead of PLAC fields.

Tags EMAIL, FAX, WWW may be allowed for Facts but not Repositories, or vice versa, or are at the wrong level.

I would discount Ancestry Family Trees as they don't even support 5.5 very well.

Last time I checked, Heredis only supports UTF8 for the ANSI subset of characters, so does not really count, as most Unicode characters are not accepted.

[AdrianBruce]

... My suggestion is much more modest: get the major players to 5.5.1, then make some refinements to it. Baby steps, and even that probably won't gain traction. <sigh>

Tamura Jones has produced an annotated version of 5.5.1 https://www.tamurajones.net/GEDCOM551An ... tion.xhtml plus later posts referred to in there. To quote him:

There are corrections of errors, commentary about differences with GEDCOM version 5.5, observations on the structure of GEDCOM files, notes meaning of some sections, clarifications of confusing parts and resolutions of contradictions. There is commentary on bad examples within the specification and on some real-world issues the specification never mentions. Several annotation provide advice, guidelines and best practices, often with links to relevant articles.
Obsolete and deprecated sections have been clearly marked as such, but have not been otherwise ignored; those sections do still contain corrections and annotations.

I confess that I haven't gone through it in detail - I downloaded his first, released version, skimmed it to get a sense of it, but unfortunately found a Tamura tirade criticising FamilySearch about their use of Common Law marriages. When I read that closely, it appeared that he'd missed the point wholly by confusing a civil marriage with a Common Law marriage. Understandable if you're starting out from a state that doesn't use Common Law, perhaps, but it put me off. And I've never got round to looking at the later version to see if it's been (as I hope) corrected.

[JohnnyCee]

Thank you ~ that extends my understanding of what is available.
But as I have said a few times, apart from UTF8 the support is somewhat patchy and sometimes non-conformant, which makes interchange via GEDCOM virtually impossible, especially when the extensive use of custom tags is thrown into the mix.

Well, I think it's less patchy than you do, but more importantly, support for 5.5 is also patchy and non-conformant. The inclusion of 5.5.1 features in files that claim to be GEDCOM 5.5 is one small example of that. Also, you continue to exempt UTF-8 as a 5.5.1 feature, but it's very important. Without it, character encoding is a mess.

Interchange with 5.5.1 is certainly not virtually impossible; people routinely move data between RootsMagic, Legacy, Family Tree Maker, other programs, and even FH, using 5.5.1 on one side or the other. The transfers are not without issues, but 5.5 transfers are not without issues.

As genealogy users, we need software publishers to move to 5.5.1 or a lightly modified version of it. Only then will it even be conceivable that we could proceed to something that addresses some of the larger issues with GEDCOM. While the genealogy community has been treading water since 1999, other technological communities have defined and redefined standards, so it's possible.

[JohnnyCee]

Tamura Jones has produced an annotated version of 5.5.1

Yes, I know. I have read it. It's a difficult read, but there are many good corrections and suggestions in it.

[JohnnyCee]

For information

Reunion 11 only exports Gedcom 4.0, Gedcom 5.5 or Ancestry.com

Ancestral Quest only exports Gedcom 4.0 and Gedcom 5.5

MacFamilyTree & Heredis just say Gedcom export without giving a version.

Thanks.

Heredis may not indicate the GEDCOM version in its user interface, but I have test files written by Heredis that include "1 GEDC, 2 VERS 5.5.1" in the header.

[tatewise]

Sorry, I did not mean to imply that UTF8 is not a useful 5.5.1 feature, but that if you put it to one side, the other features are not as coherently supported. As I stated earlier, I agree that 5.5 support is not much different.

Heredis may put VERS 5.5.1 in its header, but I believe it only supports UTF8 for the ANSI subset of characters.
Which means when importing a GEDCOM it does not recognise more than about 250 of the Unicode characters.

[JohnnyCee]

Sorry, I did not mean to imply that UTF8 is not a useful 5.5.1 feature, but that if you put it to one side, the other features are not as coherently supported. As I stated earlier, I agree that 5.5 support is not much different.

I don't understand why we ought to put it to one side.

Heredis may put VERS 5.5.1 in its header, but I believe it only supports UTF8 for the ANSI subset of characters.
Which means when importing a GEDCOM it does not recognise more than about 250 of the Unicode characters.

If Heredis is limited to a subset of Unicode, that's not really relevant to its use of GEDCOM 5.5.1 and the UTF-8 character encoding.

First, most (all?) programs are limited to a subset of Unicode because many are limited to 16-bit code points (which I'll call characters for simplicity), most do not have fonts that support all Unicode glyphs, and most do not handle all the Unicode rules. Granted, programs that use 16-bit Unicode characters internally are far less limited than programs that use 8-bit characters because they can (theoretically) support 65K characters compared to less than 250.

But that's not really the point!

One big issue with pre-Unicode GEDCOM is the lack of certainty about character encoding. ANSEL was not a good choice because (A) developers had to take special steps to read and write it and (B) few non-genealogy programs supported it. Programs used other encodings to support 8-bit characters, but the lack of a standardized encoding meant exchanging data between programs that were both limited to 8-bit characters (adequate for many, many users) was not foolproof.

The UNICODE option in GEDCOM 5.5 was a big step in the right direction, but not ideal. The UTF-8 option in GEDCOM 5.5.1 is better for two reasons: it's more disk-space-efficient and there is no ambiguity: there is only one way to encode characters in UTF-8.

Getting back to Heredis: importing UTF-8 avoids the issues in the prior paragraphs. Using UTF-8 ensures that for the characters it supports, the Heredis import will work properly. I assume that its users are aware that the program's database/UI does not support Unicode and they can decide to move to another program if they require that capability. If users export their data to GEDCOM, it's important that Heredis use UTF-8 so that the characters it supports are properly received by the downstream program.

I am not defending the character set limitation in Heredis. I am asserting that support for UTF-8 in Heredis is a good thing for its users and anyone else who has to process a GEDCOM file written by Heredis.