* Give Witness.. then Migrate Census. Importing from TMG

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
Post Reply
avatar
elricks
Platinum
Posts: 31
Joined: 18 Jun 2021 14:26
Family Historian: V7
Location: Melbourne, Australia
Contact:

Give Witness.. then Migrate Census. Importing from TMG

Post by elricks »

Good morning all. I am making progress. I created a small project out of some 500 individuals, and I am working on that subset. Thanks for the suggestion.

I imported this mini project into FH, using the default options.

To reiterate, I have 1 census tag for each census year. 1 or 2 principals, specific roles for 1 son, 1 daughter, or a number of children, and a general witness role for anyone who is in my database but who is something other than a child. I often use free form memos.

I ran Give Witnesses.. plug in. I ticked all boxes except for a few 'research tags' that have witnesses. I will decide what to do about them once I get the Census licked into shape.

I than ran Migrate Census.. plug in. The message at completion was that there were no records to migrate.

As the Individual narrative Report has duplicated entries, I want to make sure I have not made an error using the pair of plug ins.

My question is triple barreled - Have I used the plug ins as designed? If yes, is it expected that there were no records to migrate? AND of course the bit that needs a complicated answer, how do I ensure I only get one entry per person, per census year, whilst retaining ALL the memo content. Both the tag memo and the witness memo.

This is the entry I get in the Individual Narrative Report for a female who was the head of the house in 1911
------------------------------------------
In the census taken on the night of 2nd April 1911, [SG] was the head of the household. The address was in 12 Row 96

|| along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below
Principal Role
Witness Role: femalehead
, along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below Principal Role Witness Role: femalehead
$!&
In the census taken on the night of 2nd April 1911, [SG] was the wife of the head of the household,. The address was in 12 Row 96 .

|| along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below
Principal Role
Witness Role: femalehead
, along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below Principal Role Witness Role: femalehead
---------------------------------

This same person was a granddaughter of a female head of the house in 1881 (therefore I used the stock standard WITNESS role in TMG, with a witness memo that says 'her grandmother'

The Individual Narrative Report has duplicated entry too

-----------------------------------------------
In the census taken on the night of 3rd April 1881, [SG] was the head of the household. The address was in 126 Chapel Road, Necton, Norfolk, England.

[R:00002], her grandchildren, were with her
Witness Role: Witness
her grandmother..
$!&
In the census taken on the night of 3rd April 1881, [SG] was the wife of the head of the household,. The address was in 126 Chapel Road.

[R:00002], her grandchildren, were with her
Witness Role: Witness
her grandmother..
-------------------------------------
As always, if a text exists that has the answers, all you need to do is point me to it. I have not yet thought about the 'codes' that are showing, so I have not tried to resolve that issue. This query is only about the duplicated entries.
User avatar
tatewise
Megastar
Posts: 28335
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by tatewise »

Can we step back a little and establish your objectives, which may change what Plugins to run and their settings.

You have a separate custom Census event type for each census year.
It is recommended to use the GEDCOM standard Census event for every census year, as explained in your posting for TMG conversion of UK census custom tags - new user (19539).
To achieve that objective, it is probably easier to alter the TMG Master Tag Type definitions so that each custom Census event maps to the standard CENS tag instead of the EVEN tag. Otherwise, the Change Any Fact Tag plugin must be used in FH for each custom Census event.

I think you spoke of having Census narrative sentences that mention the Head of household and other family members with their relationship derived from their Witness Role.
If you Give Witnesses Their Own Facts then there will be no Witness Roles and thus those family relationships cannot be included in the Head of household narrative sentences.
So if that is your objective, then the Give Witnesses Their Own Facts plugin should not be applied to Census Witnesses.

As I mentioned before, there is a problem in FH v7.0 importing Facts with 2 Principals who are a Family Couple.
It should create a Family version of the Fact, but creates two Individual Facts instead.
Anyway, Migrate Census Family to Individual Events only works for GEDCOM standard Census (family) CENS tag events and not your custom Census events.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
elricks
Platinum
Posts: 31
Joined: 18 Jun 2021 14:26
Family Historian: V7
Location: Melbourne, Australia
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by elricks »

Thanks Mike, exactly the sort of advice I was hoping for. I will now work within TMG to work out how to change my census events. If that does not end up as the output I need, then I will go on to explore Change Any Fact Tag plugin within FH.

I would like to retain my 'census narrative sentences' if at all possible.

Some examples of my output are

Principle 2
In the census taken on the night of 3rd April 1881, Maria was the wife of the head of the household, John Base. The address was 35 Tyrolean Square, Great Yarmouth, Norfolk, England. Note that census taker wrote the family name as BETTS. I am sure this is the BASE family, due to occupation, and names and birth years of the mother and children. Their children John, Charles, Maria, Frederick and Walter were living with them.

One of the children - Maria mentioned above - ends up with this sentence
She appeared on the census taken on the night of 3rd April 1881, living in 35 Tyrolean Square, Great Yarmouth, Norfolk, England in the household of her parents, John Base and Maria Bell. She was 16 years old.

This output comes from just the ONE TMG 'tag entry' that has 2 principles and 5 'children'

I don't care if the wording is different, but I want to retain the references to (1) the children and any visitors in the household of a principle, and (2) the reference to the head of the house (and their spouse) on any child/visitor's census entry. As I have many thousands of entries I HAVE to be able to do it in bulk. I would need another lifetime to change each tag individually.

Having said that - my reality is this. I HAVE to change from TMG as it does not run under Win10. My database has 4 types of individuals. Mandatory (blood relatives and their spouses of my grandchildren), Essential (ancestors of people who married those blood relatives), Nice to have (blood relatives and other spouses of "Essential") and Other (people who I have recorded into my database who do not fit into the other 3 categories). In other words I do not keep any research data outside my database. I used TMG as my research tool, not as the place I recorded only important data.

(note in passing - those categories are just something I made up this minute, they are not in any way hard coded into my data)

Standing back to look at the differences between what I would like and what I absolutely need, the only NEED I have is to get out of FH reports that make sense, and hopefully interesting reading, to others. ONLY the first 2 categories above would ever be subjects, as the only reports I would prepare are for my target audience (living blood relatives who show even a passing interest in Family History). Naturally this is a much smaller pool of people than the 100,000 individuals I have in my database. Lets say about 17,000

As I will be using FH in the same way as I used TMG (as a research tool), I also NEED to retain all the data I have recorded, but it is actually immaterial to me how badly formatted it looks, how garbled the sentences are and how many times it appears. Until any given person comes up for additional research I don't even see their record, and if I am 're-researching' them, then a revisit of all their existing data is a positive, not a negative.

Therefore if it does come down to not getting my census tags massaged in bulk, I could actually tackle making changes on an individual basis. But hopefully using some carefully crafted FH sentences, so my output is attractive.

Shirley
User avatar
tatewise
Megastar
Posts: 28335
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by tatewise »

I will wait until you have tried adjusting TMG to migrate your custom Census as standard Census events.

One question though. In your example sentence, it says: "Note that census taker wrote the family name as BETTS. I am sure this is the BASE family, due to occupation, and names and birth years of the mother and children."
Where does that unique text come from? Is it in a Memo/Note somewhere? Can you see it in FH?
We need to know such things to advise how to include text like that using the narrative sentence codes.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
elricks
Platinum
Posts: 31
Joined: 18 Jun 2021 14:26
Family Historian: V7
Location: Melbourne, Australia
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by elricks »

Tatewise - I have just checked my TMG tag type list. Each of what I called my custom census tags are mapping to GEDCOM = cens. (see attachment of tag type list for this and other configs)

(note) When I set these up I copied an existing tag, then edited it. Then I copied that copy over and over for each census year.

I assume that means I am good to have another go at running the plug ins but will await your confirmation. I will be using a new set of data, not one I have already fiddled with.

----------

To answer you query - It comes from the memo field. The code for the insertion is <[M].> Whilst I cannot show this exact FH created individual narrative report (I binned that test DB already), I am confident the data was there, but maybe not complete, and certainly duplicated.

I am unsure if the Individual Narrative referred to below was produced BEFORE OR AFTER I ran both plug ins over my sample database, so maybe the info is useless for your purposes. The text from the FH Individual Narrative is a cut/paste from the doc.

The FH Individual Narrative I HAVE got a copy of, which uses the exact same sentence but for a different year, produces this -
----------------------------------------------
In the census taken on the night of 2nd April 1871, [SG] was the head of the household. The address was in 34 Chapel Street, Necton, Norfolk, England.

mistranscribed as JOHN, but original clearly shows JOHNSON
Witness Role: with par +
.
$!&
In the census taken on the night of 2nd April 1871, [SG] was the wife of the head of the household,. The address was in 34 Chapel Street.

mistranscribed as JOHN, but original clearly shows JOHNSON
Witness Role: with par +
------------------------------------------
I have attached a screen dump that shows my tag type config and the 1871 census entry for Louisa. Unfortunately she is only one of the children of the family, not one of the principles.

In 1911 same person was the head of the house and had with her 3 children (entered as children) and a newly adopted son (entered as a witness) The usual sentence is used for that census = the only change is the date.

My TMG sentence for 1911 is - [:CR:]In the census taken on the night of 2nd April 1911, [SG] was the head of the household <|and [POF] was his wife>. The address was [L] < [M]> <Their children [RF:with par +] were living with them><His children [RF:with fath +] were living with him><, [M2]>[:CR:]

The output as I see it inside TMG is -
In the census taken on the night of 2nd April 1911, Louisa was the head of the household. The address was 12 Row 96, Great Yarmouth, Norfolk, England. Her children Hilda, Geoffrey and Ralph were living with her, along with Reginald Arthur Pegden, her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below.

The output as I see it in the FH Individual Narative is -
---------------------------------------------------------------------------------------
In the census taken on the night of 2nd April 1911, [SG] was the head of the household. The address was in 12 Row 96

|| along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below
Principal Role
Witness Role: femalehead
, along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below Principal Role Witness Role: femalehead
$!&
In the census taken on the night of 2nd April 1911, [SG] was the wife of the head of the household,. The address was in 12 Row 96 .

|| along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below
Principal Role
Witness Role: femalehead
, along with [R:00002], her adopted son.Transcript of the original scan I have downloaded, and comments, are in the Citation Detail below Principal Role Witness Role: femalehead4
---------------------------------------------------------------------------------------------------
Attachments
JOHNSON louisa for FH help.jpg
JOHNSON louisa for FH help.jpg (370.02 KiB) Viewed 4161 times
User avatar
tatewise
Megastar
Posts: 28335
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by tatewise »

I believe that with the GEDCOM : CENS setting all your custom Census events should become standard Census events.
Therefore, in the Individual Property Box Facts tab, it should say Census in ... instead of CENS1871 or CENS1871a, etc.
Also, in the FH GEDCOM file, there will only be 1 CENS tag lines and no 2 TYPE CENS1871 tag lines.

I suspect the Memo text such as "mistranscribed as JOHN, but original clearly shows JOHNSON" and "Note that census taker wrote the family name as BETTS. I am sure this is the BASE family, due to occupation, and names and birth years of the mother and children" will appear in the Note box for Census events on the Facts tab.
In Narrative Reports, the default Options setting is to include such Note text after the associated sentence, which is where it appears in your samples.
The FH Sentence Templates allow such Note text to be inserted into the middle of a sentence, which would then be similar to your TMG sentences.

Please refer back to my first reply to this thread where I discuss whether to run those two Plugins at all.

If you want sentences that mention the Role of people with respect to the Head of household then you should NOT run Give Witnesses Their Own Facts as that removes all those Fact Witness Roles.

If you run that Plugin then every Fact Witness Role gets a copy of the Census event.
If the Head of household is also their own Witness with a Role such as femalehead then I think that explains the duplicate sentences; once as Head of household Principal Role and once as Witness Role femalehead.
Those two Roles are shown near the end in the sample sentences you posted.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
DonF
Diamond
Posts: 97
Joined: 07 Dec 2014 00:31
Family Historian: V7
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by DonF »

The sentence results you are seeing in FH are exactly as I would expect, as your TMG sentence structures are similar to mine, although not quite as complex.

The FH v7 facility to convert TMG sentences to FH format has been very poorly implemented, with a complete disregard as to the documented TMG sentence construction techniques. I have lodged 14 bug reports to CP, who rolled them all into one bug item and then promptly closed it.
This says to me that either (a) they have no idea of how to run a proper bug tracking system, or (b) they are not going to bother fixing the current mess. Given we have seen 7 updates so far with no solutions, it seems to be (b).

This leaves you with 2 choices - either spend weeks fixing up the FH-created sentences OR don't use the v7 TMG sentence conversion facility and spend weeks creating proper FH-format sentences yourself.

Don
avatar
elricks
Platinum
Posts: 31
Joined: 18 Jun 2021 14:26
Family Historian: V7
Location: Melbourne, Australia
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by elricks »

Don, Thank you for your reply. Would you be kind enough to expand on my 2 options please. I will state up front that I have no issue at all with spending weeks - or months even - fixing up my boiler plate sentences. It has taken me years, many years, to find and enter the data, and manipulate TMG to produce reports that make sense AND interesting reading. I have a very good understanding of TMG codes and sentences, developed from on the fly learning as I went. Develop a need, then work out how to achieve it. Of course, knowing TMG codes is now next to useless, but the understanding how 'sentences' become a readable 'short story' in output is the same for both applications. It will take me a few days to get my head around the actual FH codes, that's all. As my memos are already worded to facilitate use in boilerplate sentences, I will only have to craft a set of sentences for each census year once.

I say 'reports' but in fact I used my database to publish an ever changing set of families to my web site. This was the place where anyone with an interest in any given person/family in my database got access to my data. Most often found from a google search for a 'person'. I have since closed my web site down, but I have kept up the process of creating off line web sites from my data (using second site). I put these on DVD's and give them to whoever wants them. The last iteration of my sentences were written with 'web publishing' in mind.

My end point is to get to polish up these sentences in FH to a point where I can once again produce off line web sites, using FH. My main reason for doing this is so that my data will be accessible to people who have no ability to use a genealogy database. ie family members as opposed to genealogy researchers. I am not always going to be here to answer questions.

What I am struggling to understand is your reference to 'don't use the v7 TMG sentence conversion facility'. My current conversion steps are to 'import' my project using Project/new project/import from other family tree file. At what stage in this conversion process do I elect NOT to use the TMG sentence conversion facility?

Census facts - In my test project, after running my version of the conversion process, but WITHOUT running any plugin, I end up with (in FH), the same list of people, with the same role names. In the FH census fact the head of the house is the {individual} his wife has the role of Principal 2, and all the others in the house have a role with the same label as allocated in TMG. The person in TMG in this same tag (wife of head) does not end up as an individual with her own census fact.

I am comfortable that, given time, I could craft sentences in FH that fulfill my need for grammatically sound output. My concern going into this was that I would loose my memos and/or recognisable role names. But from what I can see all is intact. My other concern was that there would be no process in FH to craft sentences for output at all. So - both of my fears have been allayed.

Would you be willing to share a sample of your census sentence? I would be very grateful to see how someone else resolved the problem. I am more than happy for you to email me direct if you prefer.
avatar
elricks
Platinum
Posts: 31
Joined: 18 Jun 2021 14:26
Family Historian: V7
Location: Melbourne, Australia
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by elricks »

Mike - thanks for your reply. I am coming around to the idea that I can work with the data that I end up with after doing the conversion without running any plug in. Crafting my previous email to Don, meant that I needed to look closely at and compare my sentences, memos and output between the 2 apps. It looks like I will be able to craft sentences in FH from the data as it has been imported from TMG. This would be my preference anyway. One pass (or more using plug ins if necessary) at conversion, then all work done in FH, has got to be preferable to having to flick back and forth.

I think I may have some specific questions if I get errors, or unexpected output, when crafting my sentences, but I promise I will check the KB and the forum contents first.

There is one thing that you could advise me on. At the moment I have only discovered one way to 'access' my output, and this is quite convoluted. When in an individual record I click on Publish, then choose Narrative Report/Individual Narrative. Clicking OK gives me a report on the screen but I cannot search it, nor can I copy paste any of the text. I have to 'save report as' a txt file, then open the file. I am thinking there MUST be an easier way....

In the person's individual window, if I choose Facts, then click on a fact, the window shows me the sentence. It is in a window too small, and if I click in it, the sentence is replaced by the boilerplate sentence. Clicking out of it gives me back the output sentence. This back and forth is useful, but to get to the place I edit the boilerplate sentence I loose access to the sentences in the individual and in the witness windows. With judicious placement of the windows I can see the Individual AND the witness sentence, but with the edit fact type window open I cannot click on either of the sentences, nor can I make either of the sentence panes bigger so I can see the complete sentence.

My memory is not so good that I can retain my thread of what I should be changing.

Is there something I am missing
avatar
DonF
Diamond
Posts: 97
Joined: 07 Dec 2014 00:31
Family Historian: V7
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by DonF »

Hi - understand your thinking re converting out of TMG, although that then leads to the question - why? But that is something best discussed off-forum, I think.

To answer your question on how NOT to use the FH sentence conversion process, it fairly simple.
Once you've said you want a New Project, then 'Import from other family tree file', you then get a window to browse for your TMG PJC file (as you would know). Having selected that, and clicked Next you get a screen of specific TMG Import Options. The 2nd item is a drop-down list for Event/Witness Sentences, where the default is 'Import, Convert & Log Conversions' - which is probably what you used, being the default.
The other options are to Import & Convert (i.e no Logging) or Import Only or No import. If you play round with those last 2 you can achieve what I meant.

Happy to share TMG sentences but I can't give you my FH solutions as I don't have any - the magnitude of the work involved in converting to FH (probably about 3 months full-time is my estimate) means I stay with TMG. But I notice you're in Melbourne (as am I) so suggest direct mail contact on the details might be easier?

Don
User avatar
tatewise
Megastar
Posts: 28335
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by tatewise »

Shirley, you are not missing anything regarding the editing of Sentence Templates.
Others have complained about the small size of the Sentence box in the Facts tab.

The best workaround I can suggest is to edit the Specific Fact Template as follows:
Click in the Sentence box to reveal the Template boilerplate, then use Ctrl-A, Ctrl-C to copy & paste with Ctrl-V into a plain text editor window such as Windows Notepad.
Edit the boilerplate Template in Notepad and then copy & paste it back into the Sentence box.
You can then immediately see the effect of the revised Template on the sentence.
When you have a successful solution, delete the contents of the Sentence box to restore it to the Default Template.
Finally, in Tools > Fact Types... edit the Fact definition and copy & paste from Notepad into the default Sentence Template.

You mention that Principal 1 gets an Individual Census event but Principal 2 does not.
That is the problem I referred to earlier regarding facts with 2 Principals that should import as a Family fact for the couple but does not and is believed to be a bug as described in Importing from TMG - Family event vs single event (18925).

You mention Second Site, which has been superseded by GedSite also by John Cardinal, with who I work closely.
FH has an Export Gedcom File plugin that has settings for many products including GedSite.
GedSite lets you build CD/DVD 'web pages' similar to Second Site using either FH or GedSite boilerplate Templates.
You need to investigate whether the FH or GedSite templates are more suitable for your purposes.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
DonF
Diamond
Posts: 97
Joined: 07 Dec 2014 00:31
Family Historian: V7
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by DonF »

Can we just stop right here? I've just re-read one of your early posts and you said
Having said that - my reality is this. I HAVE to change from TMG as it does not run under Win10.
That is entirely WRONG. TMG works fine under every Windows version back to about v3.1.
What makes you think this????

Don
User avatar
tatewise
Megastar
Posts: 28335
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Give Witness.. then Migrate Census. Importing from TMG

Post by tatewise »

See the TMG Fact Type import issues (19677) topic that summarises the problem and my experimental tests on importing Census facts that explain the symptoms discussed here.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply