* TMG Imported Citation Detail

[elevator]

Long time lurker here. I have been using FH7 for about a year now (came from TMG) and I absolutely love it. This last year has been spent verifying integrity of the TMG imported data, learning the new program, alongside my regular research, and during this process I came across an import quirk regarding citation details that I just want to confirm is not a problem. TMG citation detail have two fields called "Citation Detail" and "Citation Memo" (see screenshot). I have consistently used "Citation Detail" to record detailed information about a citation (page numbers, ID numbers, references, etc). FH7 imported all of these as "Text from Source" instead of "Where Within" or "Note", as I would have expected (see screenshots). Is is a problem? I mean, as long as the detail is citation specific (which it seems to be), I am not too concerned about where it goes. But it just seemed odd to me because it has nothing to do with "Text from Source" (which I assumed was more of a transcription tool?), rather it represents a reference into the source.

And if it is indeed more appropriately to be moved into the Citation-specific details under "Where within" or "Note", is there an easy way of doing this or would this basically involve a manual re-entry of all citation detail data?

At the end of the day, it isn't a deal breaker for me that it is where it is, I was just wondering if this is expected behavior or if I did something wrong during the import process?

Also, all the imported sources from TMG are "generic sources". I like the whole idea of template sources and have read some posts and articles here about it. I am wondering about one thing though, if I were to say "convert" a generic source to a template source, what would happen to the citations? Would they be orphaned? Are they automatically linked to the new template source? I am still a little confused about the recommended process for switching (if there is one). At this point, I am leaning towards just staying with generic sources as I have been quite diligent in my setup of various source types from TMG. But if there is a relatively straightforward process that will maintain data integrity, I would definitely consider it.

Thanks for any input,
Ken.

Screenshot_3.jpg (26.71 KiB) Viewed 4599 times

Screenshot_4.jpg (104.12 KiB) Viewed 4599 times

Screenshot_5.jpg (53.51 KiB) Viewed 4599 times

[tatewise]

Ken, there are a number of topics to discuss...

FH V7.0.9
This latest version of FH has changed the way TMG Citation specific details are imported.

• Previously “Citation Details” for a source citation, were being imported as ‘Text from Source’ fields. They are now imported to a new ‘Citation Details’ field, in each source citation.

Also:

• The import now creates source templates (that is, Source Template records, with their collection specified as “TMG Import”) to correspond to all source types used in a TMG project. This in turn has made possible a number of improvements to import of sources. All source record fields are now imported, and (thanks to source templates) the field labels in Source records now match those in TMG.
• The TMG templates used for source citations are now converted to Family Historian equivalents.
• Source record ids in Family Historian now match TMG source record ids.

Citation-specific details
I suspect the reason that FH import does not use the 'Where Within Source' field is that the TMG Citation Details allow multiple lines of text but 'Where Within Source' is restricted to a single line, whereas 'Text From Source' does allow multiple lines.

There is a Plugin that moves 'Text From Source' to 'Where Within Source' and where necessary replaces each newline with a tilde (~) character. It also then moves the 'Citation Note' to the 'Text From Source'.
See Importing citation details for the Plugin download. If necessary the Plugin can be adapted to do whatever you want.
It might for instance be able to move 'Text From Source' to the new 'Citation Details' field, but I'm not certain.

Generic v Templated Sources
See FHUG Knowledge Base Sources and Citations in Version 7 (for Upgraders) which deals with your scenario of having Generic Sources that might benefit from converting to Templated Sources.
It should be possible to adapt your Generic Source Citations to Templated Sources and retain the Citation links. There has been some Plugin development to achieve that but it usually needs adapting to cope with your particular Source Citation formats and desired Source Templates, because there is no 'standard' Generic Source format, e.g. your set of generic Source Types might be very different from other users.
See converting existing generic Source records to templated Source records (18713) and the associated postings.

[elevator]

Thank you so much for that very detailed reply. I did do the final import of the data back in January and a lot of citations and other data have been entered since then, so I think I'll shy away from a fresh import. But it is good to hear that CP fixed that particular import issue. I think I will try the plugin you linked to and see if I like the result, if not I guess I'll just keep it where it is and slowly manually move them as I come across them in my research. The main thing is that all the citation detail data is there and not lost.

I have noticed one little issue though regarding the manual movement of this data (and I'll try and post a screenshot later): If I manually move the citation detail from the "Text from Source" to the "Where within source" field, then on the "All" tab in the property box (I guess what looks like a GEDCOM tree, or whatever it's called) shows an empty "Data" element where the "Text from Source" used to be, and also added the "Where within" tag with the same data. I would expect the Data element to completely removed. It seems like if the "Text from Source" element was there, it will linger in the data file even if it's removed. I suspect this is a CP support question though.

Also, thank you for guiding me to those articles about "generic" vs. "template" sources. I suspect I might end up waiting to convert, mainly because I don't feel super confident about my knowledge of the program yet to embark on such a project. My biggest fear is loss of data, specifically loss of data where it goes unnoticed until it is too late to fix.

Again though, thank you for taking the time for such a thorough response.

[elevator]

After looking a little closer at the 7.09 changes you mentioned, I am now wondering if maybe a new import of the data would be in order after all. Looks like there are quite a few improvements, and it looks like it would fix both the Citation Detail issue and allow me to start off with templates from the outset. So while I'd be stuck redoing some work, it looks like it may save me quite some cleanup in the future. Migrating from TMG has been quite a journey for sure!

[tatewise]

If you import from TMG again using FH v7.0.9 then it will create a new Project in which you can experiment safely in the knowledge that your current Project will be unaffected. So you can return to that if necessary, or eventually delete it when no longer needed.

Regarding the residual DATA tag, FH often leaves such entities behind when you transfer text and delete the original.
The deletion process just clears the field but leaves its tag in place to allow you to fill it with other text.
However, if left empty, FH will usually purge such fields automatically when it is closed and reopened.
You can also delete it manually on the All tab by selecting it and using the Delete key or Edit > Delete.

You should have noticed when moving Text From Source to Where Within Source that multi-line text will not copy without removing the newlines.

[elevator]

Thank you, yes, I think that's what I'll do: start a new project with the imported data using 7.09 and play around with that for a bit and revert if it doesn't work out.

I did notice that FH didn't allow newlines in the "Where within source" field. It automatically added the ¶ character if I just did a straight copy-paste, but I went back and changed it to commas between the lines in the few I did as a test. I also tried what you said with the Data-tag, and that does indeed delete it.

Thanks again for letting me brainstorm here, I'm exited to see what the new import looks like.

[elevator]

I have done a few imports now and it is clear that the process is much much improved. I do have some questions though, and not sure if I should post here or start new thread? Anyway, the template sources created by the import do not have any repository fields. The repositories are imported (and appear in the "All" tab) but no field is made to display them in the source template. So it seems like in every source template I'd have to go and manually add a field for repository to display it. Is this expected behavior? Also, TMG allows multiple repositories, and again these import correctly and are viewable in the "All" tab for the source, but even with a Repository field nothing displays. I have included some screenshots below to illustrate. Neither the source nor the citation is showing a repository field, but the "All" tab does. This particular one happens to have two repositories (correctly imported, but not displayed), but the same exists for sources with a single repository. Am I doing something wrong? What should I do to fix this? I had initially though of going into my TMG sources and assure there's only one repository but that doesn't change anything, the fields are not created even for single repository sources.

Thanks again,
Ken.

Screenshot_6.jpg (51.17 KiB) Viewed 4472 times

Screenshot_7.jpg (109.49 KiB) Viewed 4472 times

Screenshot_8.jpg (92.37 KiB) Viewed 4472 times

[tatewise]

Ken, there is nothing fundamentally wrong.
The All tab for any type of Property Box always shows the full GEDCOM database details.
The Main tab for any type of Property Box shows a summary of the most popular details.
Since every user's needs are different, what is 'popular' for one is not necessarily sufficient for another.
So, every Property Box can be customized to show whatever you want (within limits).
Typically, for Templated Sources, the generic Repository fields are not used so they are not shown by default.
Where necessary, a Templated Source 'Repository' field is defined instead.

To customise your Source record Property Box to show the generic Repository fields follow these steps:
In the Property Box use the top right cog Menu > Customize Property Box...
In Tab: top left choose Main (template) to select the Main tab of Templated Sources.
In Available Items on the left choose Repository and add > to Selected Items twice.
That will provide two items both for the 1st Repository , so select the lower one and click View/Edit...
In Edit Item, change the Label to Repository 2 and add [2] to the Data Reference: SOUR.REPO to specify the 2nd instance.
OK out and all Templated Source records will show both Repositories and let them be adjusted via the righthand cog menu.

[elevator]

Thank you once again! That's perfect. That resolves the Repository issue.

I am going to dare another question: as an exercise in trying to learn the program, I read a bunch of articles and I customized my own list of source templates to be in line with my TMG data and also fit my needs. But I am still baffled as to how I could utilize these templates though. I have the templates I want to use, and I have the imported data which uses a mishmash of imported templates. For example, the import may have several different Email templates based on which fields I used. Using Email as an example, I used a template in TMG which is essentially identical to the one in the FH7 essentials collection, but if I was missing a field FH7 would have created several different "variants" of the source template. For example, I could be missing the sender's name, but have their email address, or I could have their name but not their email address. And so on, and in these cases FH7 makes a separate template for each of these "variants". And most other source types from TMG has similar issues creating several variants in FH7. Not a problem per se, but it hurts my OCD. I have been browsing through the articles you mentioned above and also read through every article on the topic I could find and there seems to be no procedure for doing one or more of the following:

• Moving citations linked to one source to another
• Changing template on a source
• Merging two or more sources

It seems like to the extent that it is supported it is through third party plugins that come with very little help. Since templates seems to be the way of the future for CP, it confused me that there seems to be a lack of a step by step procedure for using templates. Or am I just missing it?

This community has been so helpful and I have been able to resolve every single issue I've had, so I am sure there must be a way for me to overcome this last hurdle as well. I have actually even toyed with the possibility of reinputing everything to get a new and fresh start, but this latest import from TMG using version 7.09 really exited me a lot. The import seems 100% complete, all I am left with is the source template issue - where there is a mess of template variants. So if there was a way to move citations to a new template, or to merge templates with a "final version" so to speak, that would be the cherry on top and FH7 would be darn near perfect for me.

[tatewise]

This is only a summary reply and may need further clarification.
Remember, that most of the help & advice is aimed at users starting to use Templated Sources from scratch rather than imported structures that have 'variants' that need tidying.
BTW: Could you tidy those variants into one template in TMG?

• Moving citations linked to one source to another
  This should not be necessary but a Plugin could do that for bulk moves.
  On the All tab, the Source link can be replaced just like any other record to record link, but only one Citation at a time.
• Changing template on a source
  In the same way any link from one record to another can be changed so can the link from a Source record to a Source Template record, but to be successful the Fields used by the Source must match the Template.
• Merging two or more sources
  Any two records can be merged using the Edit > Merge/Compare Records... command.
  If that is performed with Source Templates then all the links from Source records are adjusted as necessary.
  Of course it is only successful if the merged into master Template has a matching superset of Fields with the merged from Template.

Published Plugins in the Plugin Store usually do come with associated help. Some of them have many pages of help.
Those attached to Forum postings are usually explained where they are attached, but are sometimes so unique to one user they need no details.

[ColeValleyGirl]

Change source template (19143) may be relevant -- it includes a plugin.

[tatewise]

FYI: That thread has two versions of the Change source template plugin (the earlier one should have been removed) and also has a Convert generic sources to template plugin. You may find both useful.

[elevator]

Thank you both for your input. I am going to play around with it over the holidays; cleanup some of the fields used in TMG, and see if the import of the sources improve. I have played around with the "Change source template" plugin ColeValleyGirl mentioned and it didn't work for me, but maybe after some further cleanup it will. I also haven't had any luck with the merge source feature, but again maybe this is due to the way the sources are set up with variants having mismatching fields. Even though the fields are mismatched, they are all supposed to be there. In other words, even though one source is missing, say, the sender's name, it is still supposed to be there as a field, even though it is unpopulated for this particular source. I have also tried the "Convert generic sources to template" plugin, but after 7.09 it seems like sources are always imported as templates, and as far as I can see there seems to be no way to change this behavior?

I am very encouraged by the version 7.09 imported data, so if I can work this source issue out, I'm set

Thank you both, I'll be playing with this, and researching all the links and plugins linked to here to see if I can get it working!

[elevator]

OK, so I've worked a few days on this now, and come up with a solution that I think will work for me for now. For me the "merge" feature for the source templates seemed to work the best and be the one that presented the smallest opportunity for me to destroy data. So what I did is that I consolidated all the TMG source templates into a separate collection and merged some 180 templates imported from TMG down to just under 40 templates. For now I will just contain all the sources imported from TMG to these templates and then going forward with new sources I will have a set of proper templates that I will consolidate in their own collection. Once I am confident with the process I may eventually merge all sources down into one collection of templates. So at the end I will have two collections, once called "TMG Import" where all the source templates are prefixed with TMG to easily identify them, and then another collection that will be the one I'll actively use from now on.

But now that this process is complete all sources are there, and the original repository count, citation count, and media counts all match the TMG counts. So this must mean that all data is preserved.

I have attached a screenshot showing some of the source templates merged from the TMG import. If there are any of you FH7 experts out there who see an immediate problem with this approach I'd love to know about it.

Also, on the list of sources under the "All" tab there were occasionally an unknown field or unmapped field (I could not at this moment find an example to post a screenshot). I assume this stems from merging source templates and deleting fields I didn't need or that were duplicates. After verifying this data was already in another field, I simply highlighted the unmapped field and deleted it. That would be the appropriate way to remove unwanted data from the source, correct?

Thanks again,
Ken.

Screenshot_1.jpg (198.65 KiB) Viewed 4266 times

[Medows]

One thing that was not clear here for me that the import created all the source templates and a collection of TMG import but it does not seem to have created the Source Citation master with a collection of TMG import. When I work with Tools and then with Source Citation Definition all I see is the Advanced and Essentials collection. Is the TMG import not supposed to create a master definition or is this supposed to be a manual process of creating master definition in order to use FH7 field definitions? Currently use FH7.20.

lan

[tatewise]

For reasons best known to CP, the TMG Import only creates Project Source Templates and does not create the matching global Source Template Definitions.
Use the Create Source Template Definitions plugin from the Plugin Store to complete the import of the definitions.

[Medows]

Thanks Mike.
Just tested this out and it appears to work OK. I haven't checked the results 100%. My guess for CP not doing this is people have had issues in TMG and TMG source definitions that look similar and maybe changed come across as variants. On top of this I think there maybe an underlying fault in FH7 because I got 4 duplicated templates which have to be sorted first or deleted.
Once again Thanks for your help Mike.
PS. I started using FH at Xmas and I have joined a new group - FHUZG (Family Historian User Zoom Group) set up by Derek Heritage - aimed at helping users and have a zoom meeting every alternate week on Tuesdays and Thursdays. Its very new and only have about 110 users from all over the world. You have a vast knowledge of FH and you answer so many questions - 25000+ and I wondered if you could run a session with us. We have had Nick Walker from Ancestral Sources giving a session and last week we had Peter Calver from LostCousins.
People learn more from Zoom sessions than reading documentation. Sessions are recorded and are private to the group.
Regards
Alan

[tatewise]

Thank you for your compliments and invitation.
However, I am desperately short of time with all my current commitments (not just FHUG), so that would be too much for me.