* Importing citation details

Importing from or exporting to another genealogy program. This is the place to ask.
avatar
PeteW
Gold
Posts: 10
Joined: 10 Dec 2014 12:14
Family Historian: V6

Importing citation details

Post by PeteW » 10 Jul 2015 18:21

I’m trying to import my TMG v9.05 data over to FH v6.0.4, one complication is trying to get citation details in the right place-
TMG contains the following citation fields:
“Citation detail” reference No. etc, of citation within source - unlimited text
“Citation memo” text from source/citation - unlimited text
“reference” limited to 30 characters – not used by me
“Surety” 5 single character fields

FH has the following citation fields:
“Entry date”
“Assessment”
“Where within source”
“Text from source”
“Note”
When using TMG transfer -
FH“Entry date” remains empty – no problem
FH“Assessment” remains empty – no problem
FH“Where within source” remains empty
FH “Text from source” contains TMG “Citation detail” field, preferred it was in FH“Where within source”
FH “Note” contains TMG “Citation memo” field, preferred it was in FH “Text from source”
“Surety” and “reference” don’t appear to transfer over – not a problem for me

After TMG transfer, Is there any way to transfer the text from FH “Text from source” to FH “Where within source” and transfer “Note” content to FH “Text from source”.
I note the custom query "Query:Fact ~ Citation Details” , lists all the FH citation fields for each source but can it be edited?
Thanks for any help

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 10 Jul 2015 20:13

I appears that the TMG to FH mapping is:
Reference => Citation Where within Source
Citation Detail => Citation Text From Source
Citation Memo => Citation Note
(generally TMG Memo fields are treated as FH Note fields throughout.)

I might have expected Surety to map to Assessment but apparently not.

I am not aware of other migrants having reported your problem, so I suspect they have used the fields differently.

A Query reports data as it is.
A Query can be edited to change what it displays, but the underlying data cannot be edited.

A custom Plugin could quite easily be written to move the Citation fields as you request.

However, your use of Citations could be reviewed against the use of Source Records that may offer a better way of recording your Source Documents. See Knowledge Base > Sources. Also consider how you might make use of Knowledge Base > Ancestral Sources and adjust your existing data to match. Some TMG migrants take the opportunity to review their records and re-enter the Sources via Ancestral Sources and claim to often gain a new insight into their records and even discover new relationships.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
PeteW
Gold
Posts: 10
Joined: 10 Dec 2014 12:14
Family Historian: V6

Re: Importing citation details

Post by PeteW » 12 Jul 2015 13:00

A plugin would be of great use, I’m surprised no other TMG users have encountered this problem.
Thanks for your prompt reply.
There are 5 TMG sureties – principal1/principal2/date/place/memo (1 character entry 1,2,3,4,-)
TMG Surety 1 is mapped to assessment and is the only surety transferred. this is adequate
TMG reference is mapped to citation where within source but does not appear to transfer over, as I’ve never used this field it’s not a problem for me.
TMG has the added function of partitioning both the citation details and memo into 9 individual parts by ||, these partitioned parts appear to transfer as 1 sentence with the || dividers. As I’ve never used split citations so again this is not a problem for me.

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 12 Jul 2015 13:56

Perhaps the other TMG migrants have used the fields differently?
My interpretation would be:
  • TMG Reference holds the "reference No. etc, of citation within source" (30 char limit)
  • TMG Citation Detail holds the transcribed "text from source/citation" (unlimited)
  • TMG Citation Memo holds miscellaneous notes (unlimited)
and fits perfectly with the TMG to FH mapping, especially regarding the text format & size of each field.

The GEDCOM/FH field Where Within Source is limited to 248 characters all on one line (i.e. no newline chars).

This could be a problem for the Plugin if your TMG Citation Detail/FH Text From Source field exceeds 248 chars or has multiple lines of text, because it will not transfer to the Where Within Source field without some adjustment/truncation.

Can you please confirm whether your text violates these constraints, and if it does, how you wish it to be handled.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
DonF
Diamond
Posts: 91
Joined: 07 Dec 2014 00:31
Family Historian: V6.2
Contact:

Re: Importing citation details

Post by DonF » 13 Jul 2015 00:07

If I can just make some corrections here.

The field use in TMG is meant to be as follows (and this is from TMG Help):
Citation Detail is used for more specific information in the citation; for example, page numbers, dates of correspondence, etc. The length of this field is unlimited.
Citation Memo: is used to record comments concerning the citation - comments, rationale, etc
Citation Reference: is designed to hold a reference number assigned to the cited information, for example, where you have filed a document.

Thus the TMG -> FH mapping appears to be 'correct', or at least close.

Also TMG Sureties are not as stated above - for each principal1, 2/date/place/memo part there are possible ratings of 3,2,1,0,- or blank. These do not map well to FH Assessment as they are much more sophisticated (and have more possible settings), plus the FH mapping process is undocumented and unexplained (though I have asked support multiple times for clarification).

I thought it might be better to try and move the field contents in TMG before import, but even using the TMG Utility, this would be tricky. It may be better to accept the transfer as is, and adapt one's use of the fields.

avatar
PeteW
Gold
Posts: 10
Joined: 10 Dec 2014 12:14
Family Historian: V6

Re: Importing citation details

Post by PeteW » 13 Jul 2015 11:07

Thanks Don for the TMG help details, which I was going to copy over anyway
TMG help:
TMG_Citation Detail This field is used for more specific information in the citation; for example, page numbers, dates of correspondence, etc.
TMG_Citation Memo The Citation Memo field is used to record comments concerning the citation.
TMG_Reference The Reference field is designed to hold a reference number that you have assigned to the cited information. This tells you, for example, where you have filed a document. 30 Characters

FH help
FH_Where within Source If the source record is a document, you could use this field to store a page reference. When citing non- document sources, however, you can adapt this field appropriately, to simply specify in more detail which aspect of the source is relevant.
FH_Text From Source Again, this field is most obviously appropriate for document sources.
FH_Note Use this field to store any notes you wish to make about the current source citation.

When you compare TMG help and FH help it appears to me that TMG_Citation Detail & FH_Where within Source are comparable fields, TMG_Citation Memo & FH_Text From Source could be used in the same way, this leaves TMG_Reference which appears to be designed for a personal ref. No. and I’m not sure where you transfer to possible to FH_Note., it doesn’t appear to transfer over anyway on my test data.
However, I’m sure there are numerous variations on the use of these fields by both TMG & FH users.

I’ve looked at TMG Utility but as far as I can you can only swap around the various fields on a limited basis, it allows for using split citations but not split memos which would make it easir but still tricky, especially with the TMG_Reference limited to 30 chracters.

In answer to Mike’s question:
My TMG_Citation Detail fields are all less than 248, but a number are spread over 2 or 3 lines, if these were combined into 1 line with a divider eg comma or dash, then that would not present any problems, all would still be less than 248.

avatar
Jim Byram
Newbie
Posts: 3
Joined: 13 Jul 2015 18:09
Family Historian: V6.2

Re: Importing citation details

Post by Jim Byram » 13 Jul 2015 18:15

I see exactly the same issue with the TMG citation fields being imported to incorrect FH fields after doing a direct import with FH v6.04.

It was very easy to repair my FH database by globally editing the FH GEDCOM since my citation data is simple and absolutely consistent.

For many users, however, this would be a show stopper.

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 13 Jul 2015 18:46

Whether the mapping is correct or incorrect is debatable.
My argument goes along the lines that there is one single-line limited length field, and two multi-line unlimited length fields in both products, and both structures are quite closely based on GEDCOM. One of the multi-line fields is called Memo in TMG, which equates to Notes in FH in a number of cases. So by a simple process of association, what could the mapping be for these three Citation fields?

Any mis-mapping is far from a "show stopper", as I will soon demonstrate with a simple Plugin to correct Pete's case, that will be like other Plugins that have adjusted similar mis-mappings for imports from other products.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Jim Byram
Newbie
Posts: 3
Joined: 13 Jul 2015 18:09
Family Historian: V6.2

Re: Importing citation details

Post by Jim Byram » 14 Jul 2015 01:00

Mike,

If you export a GEDCOM from TMG and then import that GEDCOM to FH, the data will be in appropriate fields. <g>

The TMG 'Citation Detail' should go to the FH 'Where within source' field.

The TMG 'Citation memo' could go to either the FH 'Text from Source' or Note field. FH has two fields in that case whereas TMG has only one field.

The TMG surety is used the same way as the FH Assessment data is used. They are not equivalent but have similar functions.

The TMG References field is for a user-created filing index value so that you can locate documents in your filing system related to the citation.

btw... I maintained and wrote the TMG help file beginning with TMG7 until it was closed down.

Jim

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 14 Jul 2015 09:20

Unfortunately, transfer of the TMG 'Citation Detail' into the FH 'Where within Source' field might not work satisfactorily in the general case as discussed with PeteW. It does not matter whether the transfer is performed via GEDCOM or by Direct Import of the TMG database.

However, it depends which GEDCOM tags the TMG GEDCOM export uses for those fields. It would be interesting to know how TMG fits its Citation Details into GEDCOM tags. FH would not change the tags chosen by TMG when importing the GEDCOM.

The TMG 'Citation Detail' field is potentially a long multi-line unlimited length field.

The FH 'Where within Source' field is a single-line field, which is limited to 248 characters in the GEDCOM specification, although FH seems to ignore this limit, but if the GEDCOM were exported, then another product might truncate that field, because it not only exceeds the field length but also the GEDCOM line length, and there may be other unexpected side-effects.

The first problem is how to handle multiple lines of text in the TMG 'Citation Detail'. Perhaps the lines just need to be concatenated together - would that be acceptable?

FH has avoided those issues by transferring long multi-line TMG fields into long multi-line FH fields.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
PeteW
Gold
Posts: 10
Joined: 10 Dec 2014 12:14
Family Historian: V6

Re: Importing citation details

Post by PeteW » 14 Jul 2015 12:57

TMG_Citation Detail - the lines concatenated together would not present any problem for me and the combined length would still be less than 248

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 14 Jul 2015 13:04

I appreciate that Pete, I was responding to Jim's suggestion that is should be the default mapping.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 14 Jul 2015 14:05

Pete, I have attached Move Citation Text Fields version 1.0 dated 14 July 2015.
Just left-click on the link to install it into FH.

It assumes your data only has Citations in Individual & Family records & facts. If there are Citations elsewhere, let me know before running the Plugin.

I advise you take a File > Backup/Restore > Small Backup (Gedcom File Only) before running the Plugin.

Use Tools > Plugins select the Plugin and click Run.
If your database is large it may take a while to complete.

For each Individual or Family record/fact Citation:
  • If Where Within Source is empty and Text From Source is not empty,
    then it moves Text From Source with ~ between lines into Where Within Source.
  • If Text From Source is then empty and local Note is not empty,
    then it moves the local Note field into Text From Source.
Check the changes are OK, and if anything is unsatisfactory use Edit > Undo Plugin Updates before closing FH.

Let me know if there are any problems with using the Plugin.
Attachments
Move Citation Text Fields.fh_lua
(2.43 KiB) Downloaded 121 times
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
PeteW
Gold
Posts: 10
Joined: 10 Dec 2014 12:14
Family Historian: V6

Re: Importing citation details

Post by PeteW » 14 Jul 2015 19:43

Mike,
Thanks for the plugin which worked fine.
Just one point you stated in a previous post that TMG_Reference is mapped to FH_Citation Where within Source
Although I don't use this field, I've entered test data into a couple of TMG_Reference citations but they do not appear to transfer to any citation field in FH.
Perhaps it might be worth considering for completeness for other TMG converts to have a TMG_Reference to FH_citation note within the plugin?

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 14 Jul 2015 20:02

Sorry, my TMG Reference to FH Where Within Source mapping was an educated guess, which proved to be wrong.

The Plugin cannot obtain the TMG Reference, because that data does not exist within FH, only within TMG.

You should Email Calico Pie at support@family-historian.co.uk and suggest how you would like the TMG fields to be imported into FH, but remember it should be appealing to the majority of TMG migrants.

It is feasible to import multiple instances of the Text From Source field or the Note field.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Jim Byram
Newbie
Posts: 3
Joined: 13 Jul 2015 18:09
Family Historian: V6.2

Re: Importing citation details

Post by Jim Byram » 15 Jul 2015 02:07

Mike,

Here's how TMG exports a dummy citation to GEDCOM 5.5.

2 SOUR @S1@
3 PAGE citation detail; citation reference
3 NOTE citation memo
3 QUAY 1

The citation reference export is optional and is appended to the citation detail if selected.

TMG doesn't follow the GEDCOM specs and will extend the PAGE data with CONT lines as necessary to get all of the citation detail data out. If this is done, I assume that it will be clipped by FH.

In TMG, both the citation detail and citation memo fields are memos so have unlimited length (for all practical purposes). Other than the fact that both fields can be 'split' into 9 parts with delimiters, TMG doesn't care how you structure the data in either field.

I would now assume that the FH direct import was designed to deal with the possibility of the citation detail overrunning the 'where within source' field. I can understand that. Rearranging things after import is simple enough to do.

Jim

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 15 Jul 2015 10:36

Thanks for that feedback Jim.

I notice that a TMG Surety is supplied in the QUAY Assessment tag.

Since CONC/CONT tags on a PAGE tag are illegal GEDCOM, the FH GEDCOM import will convert them to UDF tags, but the text data will be retained.
Then a Plugin could reformat the text data as the user requires.

Your assumption about FH direct import from a TMG database would appear to be correct, and to me is a sensible decision.
However, it is a shame that the TMG Reference is not transferred to FH Where Within Source.
It is also disappointing that at least one TMG Surety is not transferred to FH Assessment.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
DonF
Diamond
Posts: 91
Joined: 07 Dec 2014 00:31
Family Historian: V6.2
Contact:

Re: Importing citation details

Post by DonF » 17 Jul 2015 11:03

I would have to differ on the comment that

Code: Select all

It is also disappointing that at least one TMG Surety is not transferred to FH Assessment.

As far as I can tell, all my TMG citations that have a TMG Surety end up with an FH Assessment. Not one I can understand though, but as I've said before I have asked Calico Pie how that map them, with no reply. Mostly they end up with 'Primary Evidence', which led me to believe FH was taking the 1st TMG Surety (namely 3 for the Principal, meaning 'yes, I am sure this pertains to this person') and treating that as the top-level FH Surety. Then I found a citation of 2 sources, with differing sureties against the Principal ( a 3 and a 2) and FH claimed they were both 'Secondary Evidence'.
So I have no idea what the FH Import is doing, but it does transfer something - though the results are meaningless

avatar
PeteW
Gold
Posts: 10
Joined: 10 Dec 2014 12:14
Family Historian: V6

Re: Importing citation details

Post by PeteW » 17 Jul 2015 13:18

I agree with you there Don, that some form of surety comes over.
I've been trying to work out what, it a appears to take into account sureties other than No1, but how it calculates the final assessment is a mystery.

avatar
BarbaraL
Gold
Posts: 11
Joined: 21 Dec 2015 00:37
Family Historian: V6
Location: US

Re: Importing citation details

Post by BarbaraL » 27 Feb 2017 00:05

I have a hypothesis about how the FH Assessment is determined that fits my data. Given a set of 5 TMG Surety values, it appears that FH takes the lowest Surety value and maps it to the FH Assessment as follows:

3 -> Primary evidence
2 -> Secondary evidence
1 -> Questionable
0 -> Unreliable

So, Surety values 33--- would be mapped to Primary evidence; 22-2-, 3-2--, and 33222 to Secondary evidence; 32321 and 3311- would be mapped to Questionable; and so on.

Whether that mapping makes sense would depend on how you use the Surety values.

Barbara

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 27 Feb 2017 00:18

That seems a reasonable hypothesis.
For the benefit of non-TMG users, how do those 5 Surety values relate to the Source Citations, etc.
Also how can TMG users employ them that might make the FH mapping unacceptable?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
BarbaraL
Gold
Posts: 11
Joined: 21 Dec 2015 00:37
Family Historian: V6
Location: US

Re: Importing citation details

Post by BarbaraL » 28 Feb 2017 00:34

In TMG, a Surety value can be assigned in 5 "slots" for each citation (principal 1, principal 2, date, place, and memo) to "indicate the quality of a source in documenting a given fact". TMG defines the values as follows:
"3= an original source, close in time to the event
2= a reliable secondary source
1= a less reliable secondary source or an assumption based on other facts in a source
0= a guess
-= the source does not support the information cited or this information has been disproved"

(I need to revise the notation from what I used before, since "-" is one of the values.) So, where "_" is an unassigned value, 3312_ would be the notation for a Surety value of 3 assigned to principal 1 and principal 2, 1 to date, 2 to place, and no assignment for the memo for a particular citation.

To evaluate the conversion of TMG Surety values to FH Assessments for a given dataset, a TMG migrant would want to think about (a) the meanings that s/he assigns to the TMG Surety values, which may or may not be the same as TMG's meanings, (b) the meanings of the FH Assessment values (and/or the values of GEDCOM's QUAY (?)), (c) the implications of the loss of information in the mapping from 5 different Surety values to 1 Assessment, (d) the implications of the mapping algorithm, and (e) whether Surety is mapped to Assessment in the way desired given (c) and (d).

Given TMG-ers' propensity to customize, I would think that migrants would differ in whether they wanted to retain the FH Assessments as mapped. I would expect that the more a TMG user's Surety value's meanings differ from the FH Assessment values' meanings, the less likely the mapping would be acceptable.

I have assigned my own meanings to the Surety values, so the mapped Assessment values don't work well for my database. I will either remove the Surety values (in a temporary database) before migration or remove the Assessments post-migration.

Barbara

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 28 Feb 2017 14:00

Thank you Barbara that is helpful, and I may summarise it in Knowledge Base > Import from The Master Genealogist (TMG).

FYI: There are some other options to consider.
Cannot do much about the first 3 Surety values, unless the 2nd can be related to a Fact Witness and the Source Citation attached there with the 2nd Surety value.
A Source Citation can be attached to a Place field via the All tab and then the 4th Place Surety applied there.
A Source Citation can be attached to a Note field via the All tab and then the 5th Memo Surety applied there.
So the same Source record can be cited by the Fact, each Fact Witness, Fact Place, and Fact Note and each have its own Assessment, Where within Source, Text From Source, and Citation Note, which is all compatible with the Gedcom standard.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
BarbaraL
Gold
Posts: 11
Joined: 21 Dec 2015 00:37
Family Historian: V6
Location: US

Re: Importing citation details

Post by BarbaraL » 01 Mar 2017 00:57

Thanks for thinking this through, Mike! I have a couple of questions.

Just to further my understanding (I have a long way to go!), why is FH Fact Date different from Fact Place in not being allowed its own Assessment? That is a critical piece of information not to be able to assess.

Wouldn't your options presuppose that each of the 5 TMG Surety values are imported into FH and reside in the GEDCOM?

Since I forgot all about the "-" value, here is the result of my more complete testing of the mapping from the lowest level TMG Surety value to FH Assessment:
3 -> Primary evidence
2 -> Secondary evidence
1 -> Questionable
0 -> Unreliable
- -> Unreliable

Barbara

User avatar
tatewise
Megastar
Posts: 19127
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Importing citation details

Post by tatewise » 01 Mar 2017 09:57

There are many data fields associated with each Fact such as Date, Age, Cause, Address, Phone, Email, Website, etc. They form part of the Gedcom specification that FH adheres to. Of those, only the Fact itself and the Place and Note fields are allowed to have a Source Citation by the specification. You would have to ask the authors, Church of LDS, why that is.

The Gedcom specification does allow custom extensions called user-defined fields, and FH does use them to some extent. However, they rarely migrate easily to other products via Gedcom. Conversely, non-Gedcom data fields from other products, such as those in TMG, rarely migrate easily into FH.

If the 5 TMG Surety values were imported into FH in the Gedcom in say the Citation Note field, then it would make my proposed options easer to implement. As it stands, you would have to review TMG and FH side by side, Citation by Citation. If a suitable Source Citation report could be produce from TMG, then it may be possible for an FH Plugin to be written to implement those options automatically.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

Post Reply