* Converting Sources: Method 2 to Method 1

Questions regarding use of any Version of Family Historian. Please ensure you have set your Version of Family Historian in your Profile. If your question fits in one of these subject-specific sub-forums, please ask it there.
Post Reply
avatar
gsward
Famous
Posts: 146
Joined: 11 Mar 2011 14:47
Family Historian: V7

Converting Sources: Method 2 to Method 1

Post by gsward »

My specific case is that I have some census sources that have been created using Method 2 ("lumped") that I had created in a previous program that I used (Genbox), and before I knew better. I have ignored them for a long time, but would now really like to tidy them up. The original sources do not have any attached media.

This question has come up from time to time and there are a number of threads that offer partial solutions.

Using Excel VBA to convert from lumped to split sources (15390)
Sorting Source Citations (15939)

Has anyone come up with a useful strategy for this apart from creating new Method 1 sources in AS and finally deleting the original source?

Graham
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

There is not much point in converting from Method 2 to Method 1 unless in doing so you add transcript Text From Source and a Media census image in the process. Also each Method 1 source must be cited by the Individual records of the people in the same Census household, and that cannot be automated from the Method 2 sources.

So the only worthwhile technique is to use AS and finally delete the Method 2 source.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Converting Sources: Method 2 to Method 1

Post by Mark1834 »

The Excel routines were useful for tidying and converting my own tree, which was originally developed in Family Tree Maker, which has a more traditional "database" approach that doesn't work well in FH due to fundamental limitations of GEDCOM. It was based on older routines that I used with FTM to produce more detailed custom reports that were not available in that package.

I am now about to start a similar process for my wife's tree, and I will use the experience gained in processing FH data for Ancestry import to replace the Excel routines with similar ones written in Python. They should have the advantage of being much simpler to code, and avoid the occasional problems within Excel of corruption between what is text and what is number format. They also have another significant advantage of being more portable, working on any system without needing additional commercial software.

Regarding Ancestral Sources, I realise this is probably a minority view on this Forum, but I prefer to enter data manually and copy events and sources using the routines within FH rather than going through AS. I limit the events I create from each source, for example capturing residence and occupation from Census returns or Parish Registers in the transcript rather than as separate events. If you prefer to record everything as an event, AS would be the way to go. As ever, there is no "right" or "wrong" way to do things - it is what works for you in terms of what you want to achieve as your end result.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

On a point of clarity, as a general rule AS does not create Residence facts, at least not from Census records.
AS also does not necessary create Occupation facts.
Yes, the default configuration does encourage that, but it is perfectly feasible to adjust AS so it does not.
The easiest method is to simply not fill in the Occupation (or Residence) fields in the green forms, but just add those details to the Auto Text pane below, which ends up in the Text From Source field of the Source record.
I also recognise that some users prefer to use just the features of FH.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Converting Sources: Method 2 to Method 1

Post by Mark1834 »

These conversion routines are now done, and I have two Python scripts exactly analogous to the previous Excel/VBA versions but vastly superior (more compact, easier to follow, much quicker, and above all, free!).

The first formats the Family Tree Maker export GEDCOM to address various issues raised in the direct import (most commonly removing the extra illegal detail with the CENS, DEAT and BURI tags and convert it to notes). FH does this as well, but even with a fairly modest tree it gave me an 8000 line log file. Pre-processing the GEDCOM file removed all of this and gave me a clean import without any warnings apart from the generic one about mishandling of CONC/CONT continuation lines that is an Ancestry/FTM trait. It also converted media records to FH format, adding titles and dates from parsing the filename.

The second takes this formatted output and splits all my lumped Census, Parish Register, Probate, MIs and various Quaker databases to individual sources with meaningful names derived from the lumped name, citation, and party names/event locations/event date. All citations are redirected to the appropriate split source, and all citation images automatically become source images. The modified GEDCOM this produces also imports cleanly to FH. Naming is not perfect, as it uses the birth name in the database, but it will be a simple matter to work through the split sources and add married names as appropriate (e.g. to burials and probate records).

Each script is only around 200 lines or so of code, which sounds a lot but is tiny compared to some of the more complex plug-ins, which can run to 10,000 lines or even more. They are far too customised to my way of naming and lumping to be worth uploading directly, but if there is anybody else who needs to split sources and is happy to dive into a bit of Python (which is free, and cross-platform), I'd be happy to share my approach. These were written for a FTM export file, but they would also work with the FH GEDCOM database file directly with appropriate minor syntax changes.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

Mark, that sounds like some of those features may be very useful to Ancestry/FTM migrants.

I thought the CONC/CONT issue had been resolved in FH V6.2.6, because Ancestry/FTM now follows the standard, so FH no longer needs the special workaround.

When your script splits the lumped Method 2 Sources to split Method 1 Sources I suspect that for say Census Events it produces a separate Citation and Source record for each member of the same household.
If so, then it makes finding those household member's Individual records rather more difficult.
The Source records for each such household need to be merged to achieve the recommended structure.

Similar scenarios may apply where say a Birth Certificate is cited not only by the Birth Event of the child, but also by the Name fields, and perhaps other facts for the parents. I realise that you may not use such facts, but I mention it for the benefits of those who do.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Converting Sources: Method 2 to Method 1

Post by Mark1834 »

Mike,

Splitting can be customised according to what information is available. For Census entries, I already have the PRO reference as the detailed citation (the citation Where within Source), so this readily creates sufficiently distinct split source names, one per household. I add the location for convenience but not the name of the individual for just the reason you describe above - it would introduce unnecessary fragmentation. A typical Census source name would therefore be something like 1881 UK Census, location (PRO reference).
Similarly for Probate entries, where I already have the name and date as the Where within Source, so the separated source title would be England & Wales Probate Register, 1892, John Smith.
Parish Registers are more difficult, as I have not recorded anything in the Where within Source field, so use the individual name and event details to derive a name such as Baptism Register, Chelsea, Middlesex, 1838, Jane Smith. For most entries this is fine, but it does create problems when a definite birth date in a different year is quoted in the register. A second source is created referring to the birth year, so would need editing manually afterwards. However, this is a very small minority of entries in my experience. It also doesn't combine "batch" family baptisms to one source, but it is personal preference as to whether they are one source or multiples ones. A more significant issue may be the one you allude to, where for example, a baptism register entry is also cited as evidence of father's occupation or family residence. In this case it would create artefact entries referring to the parents, which is clearly incorrect. However, it is important to note that if the register entries are already sufficiently identified in the Where within Source (e.g. by register page number or event date), this can be used directly without the need to create artificial names. There are probably other options as well - Python list structures fit the design of GEDCOM files very well, so manipulation is relatively straight forward.

I think the reality is that there is no realistic way of creating a generalised splitting tool that works "out of the box" under all scenarios - as evidenced by all the discussion on this forum over the years without such a tool appearing!
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

Thank you for that excellent explanation of how various details are used to collate associated groups of citations to one Source.
As you say, it won't necessarily work for everyone, but does provide anyone following this thread with some useful pointers.
If Method 2 Source users have consistently used Where within Source in the Citations, then it does suggest that a Plugin might be feasible to convert to Method 1 Sources.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

Hallo Graham,

Do you share your code to split the sources?
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

I don't think Graham has produced any code.
It is Mark that has created some Python script.
However, as was said earlier, the conversion process is very specific to the format of your 'lumper' Source Citations and what details you want in the 'splitter' Source Citations. Also, the process may be different depending on whether they are Census, Birth Certificate, Marriage Certificate, or Death Certificate sources.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

Ah ok, then it was Mark, sorry…

I have 15 sources (german church book duplicates) from pommern where i used the lumper method with the page number in the citation field.

After some years where i had no time for this hobby i want to continue my work. I tried to add the pictures of the church books to the citations and have the problem with the lumper method. Some sources have 3000 or more citations and i can‘t change them manually.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

What is the problem you are having with adding the pictures to the citations?
That should work OK and may be easier than converting to 'splitter' Sources.
Just take care to add each Media picture to a specific Citation and not the Source record.

If you still decide to convert your Source Citations then we would need much more detail about the Citation fields and how you want the Source records formatted.
e.g.
Does every Citation use the Where within Source field to hold the Source page number?
Does every Citation refer to a different Source page number?
OR
Do several Citations all refer to the same Source page number sometimes?
What details need to be moved from the Citations to the 'splitter' Source records?
How do you want each 'splitter' Source record Title to be formatted?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

Here is an example on how my sources looks like...

The small number of sources are great with the lumper method, but when i add a picture to a single citation with "Where Within Source" only this entry ist changed. The other citations are without the images.

In Family Tree Maker or RootsMagic (i use this software too) this is no problem, only in Family Historian. I love FTM for the FamilySearch and Ancestry integration (i use both of them) but the way to add data in FH is simply the best... The Automatic Source Citations function ist perfect :) and save a lot of time when working on church books.
Attachments
source.png
source.png (141.66 KiB) Viewed 6463 times
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

Unfortunately, your screenshot does not help me much without the column headings.

Please explain the problem with adding pictures in more detail, otherwise, I am left guessing what you want to achieve.

Which "other citations are without the images"?

Do you mean all the ones with the same Page number, or all the ones with the same Source record?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

When i change one entry with "Where Within Source = Seite 11" for example and add here a picture...
the other entries with the same "Where Within Source" page number still remain without the picture.
Attachments
source2.png
source2.png (45.75 KiB) Viewed 6462 times
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

Yes, in FH, each Citation is completely separate, so you have to add the same picture to each one citing Page 11.

So it is now becoming clearer that you want a separate Source record for each Page number.
Then you can add the picture of that Page to the Source record so it applies to all the Citations of that Page.

Would you want each Source record Title to be the same as the current Title but with the Page number added?
e.g.
Kirchenbuchduplikat - Gol ........ Page 11
Kirchenbuchduplikat - Gol ........ Page 102
Kirchenbuchduplikat - Gol ........ Page 103
etc.

Is the only data in your Citations the Where Within Source field?
Are all the Assessment, Entry Date, Text From Source, and Note fields always empty?

With the answers to those questions I may find time to write a Plugin to make the changes.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

that would be beautiful...
Kirchenbuchduplikat - Gol ........ Page 11
Kirchenbuchduplikat - Gol ........ Page 102
Kirchenbuchduplikat - Gol ........ Page 103
this is ok, i will start to name my sources like this.
Is the only data in your Citations the Where Within Source field?
Are all the Assessment, Entry Date, Text From Source, and Note fields always empty?
yes, the only data is in the "Where Within Source" field, the other fields are all empty.
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Converting Sources: Method 2 to Method 1

Post by Mark1834 »

There was further discussion here. This example looks virtually identical to what you want to do, so if you are comfortable installing Python and tweaking the script in the linked thread (with my guidance), it could probably do exactly what you want very quickly.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Converting Sources: Method 2 to Method 1

Post by tatewise »

cschroeder ~ I suggest you don't change anything manually and let the Python or Plugin scripts perform the changes.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Converting Sources: Method 2 to Method 1

Post by Mark1834 »

This discussion is split between two threads, here and FTM to FH - Citation to Source. I'll reply here, as this is where the data are.

As written, the Python script will convert all examples of a nominated lumped source with different Where within Source values to individual split sources with titles of the form Original Source Title:Where within Source and the Where Within Source data repeated in the Publication Information of the new source. All that is required is customising the input and output file names and converting the input file to UTF-8.

I'm not clear which sources are not being split. Can you give a specific example of what additional splitting you are trying to do please?
Mark Draper
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

After running the python script the following entries are not splitted:

- Record (INDI)
- Record (FAM)

other sources for facts like Name, Occupation, Born... are splitted
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Converting Sources: Method 2 to Method 1

Post by Mark1834 »

Clear now thank you. I never use whole record sources, so did not include that in the search. It’s getting late now but I’ll look at it tomorrow.
Mark Draper
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

i think the "automatic source citations" is the reason for this...
avatar
cschroeder
Gold
Posts: 22
Joined: 14 May 2016 20:44
Family Historian: V6.2

Re: Converting Sources: Method 2 to Method 1

Post by cschroeder »

seen differently...

i don't think that "whole record sources" make sense because every fact (names, events...) have their source...
so i could delete the old source without losing information
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Converting Sources: Method 2 to Method 1

Post by Mark1834 »

Answered in FTM to FH - Citation to Source to keep the downloads in the same thread.
Mark Draper
Post Reply