* Witnesses when entering Census data

AS allows faster and more convenient creation of source records for Family Historian.
Post Reply
avatar
mitcheip
Diamond
Posts: 67
Joined: 21 Aug 2014 08:37
Family Historian: V7

Witnesses when entering Census data

Post by mitcheip »

Hi, This is my first time using AS so I am probably doing something wrong.

I am entering a Census event and there 3 people involved, a mother and her two children.

I was expecting AS to create a Census fact for the Mother with 2 witnesses, the 2 children.

Instead AS created a separate Fact for each of the people with no link between the 3 facts?

Is this what is intended/expected?

Looking at some similar census events imported from TMG there is a problem.

If the Mother is entered as Principle 1 in TMG, one of the children as Principle 2 and the 3rd child as a Witness then the direct import creates 3 separated unrelated facts, just like I'm experiencing with AS.

If the Mother is entered as Principle 1 in TMG and the children entered as 2 Witnesses then the direct import creates a single Fact with 2 witnesses, (which is what I would expect)
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Witnesses when entering Census data

Post by jbtapscott »

AS does create a separate fact against each Individual as you have found - the "link" is the Source, where, if you chose the appropriate options in AS, the "Where within source" will have details of all people mentioned on the Census. I am also in the process of moving from TMG, but as I did not use the Witness "approach" within TMG, I am quite happy with the way AS / FH handles this.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
NickWalker
Megastar
Posts: 2597
Joined: 02 Jan 2004 17:39
Family Historian: V7
Location: Lancashire, UK
Contact:

Re: Witnesses when entering Census data

Post by NickWalker »

mitcheip wrote:Hi, This is my first time using AS so I am probably doing something wrong.

I am entering a Census event and there 3 people involved, a mother and her two children.

I was expecting AS to create a Census fact for the Mother with 2 witnesses, the 2 children.

Instead AS created a separate Fact for each of the people with no link between the 3 facts?

Is this what is intended/expected?

Looking at some similar census events imported from TMG there is a problem.

If the Mother is entered as Principle 1 in TMG, one of the children as Principle 2 and the 3rd child as a Witness then the direct import creates 3 separated unrelated facts, just like I'm experiencing with AS.

If the Mother is entered as Principle 1 in TMG and the children entered as 2 Witnesses then the direct import creates a single Fact with 2 witnesses, (which is what I would expect)
All the census facts, occupation facts, birth facts, etc. that AS will generate from a census fact are all linked together by a source.

Witness features were only added to Family Historian in December when version 6 was released but AS (and my previous census entry software) has been used for many years before this. However, I don't really think that using witnesses are the solution for census records. Firstly a witness doesn't have an age recorded so this data will be missing (or just recorded in a note or source). Secondly, the 'principle' may be someone not in your file (e.g. they might be a landlord). Thirdly, a census entry will usually require many occupation attributes to be created and these will need to be added as facts to those people so if you're going to have all these separate occupation facts what advantage is there in not having separate census facts too.

If software like AS wasn't available then entering all this data into your file would be tedious and slow and I can see why you might use witness roles to record the basic details but with AS doing most of this work for you I just don't see a reason to use witness roles for this.

I think witness roles are great for recording people who are 'associated' with the event but are not the principle people involved, so a godparent at a baptism, best-man at a wedding, mourner at a funeral, etc. (and AS uses witness roles to record all these associated people). But in a census record all the people you are recording are equally important and I think this is the reason why witness roles are not the best solution.

Best wishes

Nick
Nick Walker
Ancestral Sources Developer

https://fhug.org.uk/kb/kb-article/ancestral-sources/
avatar
mitcheip
Diamond
Posts: 67
Joined: 21 Aug 2014 08:37
Family Historian: V7

Re: Witnesses when entering Census data

Post by mitcheip »

@Nick,

Thanks for your answer, it explains what I'm seeing.

I think everybody using family history software will record things in a different way, a lot of the time, based on what the software they are using is capable of doing.

Within TMG I used the Witness functionality extensively precisely for the reasons you mention, entering a census record was relatively quick using witnesses and other users had provided standard lists of "Roles" likely to occur in each census year and country making it even quicker.

There are limitation to this method as you mention, age and occupation being the main ones. Having said that I rarely, if ever, did a full transcript of a census because most of the time I wasn't interested, for example, in the occupations of people that strayed from my direct line. Where I wanted to record an occupation I created another event as you mentioned.

All this said FH does now provide a different way to record census detail, you might call it "method 3" in your software and I would suggest a significant proportion of TMG uses would like to see the Witness method as a way of recording the census data.

I really would like to use your software to create Census entries but I would end up with a database that is a mess. I have literally hundreds of census events and they have been imported into FH, by the direct import routine developed by Calico, and they are using the Witness method rather than individual Fact for each person on the census. So if I used your software I would have the majority of the census events in my database using witnesses method and the newer stuff using individual facts method.

So, heres an enhancement request, on behalf of any ex TMG users that have used the Witness Method for recording Census and have/are considering moving to FH, can you add Method 3 to AS? :D
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

An alternative approach, if you are interested, is to convert all your Method 3 Witnesses to use Method 2 individual Census Events.

That could be performed quite easily with a custom Plugin, and not beyond the capabilities of one of the TMG migrants I am sure. That would also resolve the other deficiences that you acknowledge, of lack of Age, etc.

Remember that AS is produced for free (or a small donation) by another FHUG member, and adding a Method 3 that is of no use to him, and might over complicate a program that is already riddled with options, might be to ask too much.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
NickWalker
Megastar
Posts: 2597
Joined: 02 Jan 2004 17:39
Family Historian: V7
Location: Lancashire, UK
Contact:

Re: Witnesses when entering Census data

Post by NickWalker »

Mitcheip said:
... I rarely, if ever, did a full transcript of a census because most of the time I wasn't interested, for example, in the occupations of people that strayed from my direct line. Where I wanted to record an occupation I created another event as you mentioned....
...So, heres an enhancement request, on behalf of any ex TMG users that have used the Witness Method for recording Census and have/are considering moving to FH, can you add Method 3 to AS?...
I'll add it to a list of possible future enhancements though it isn't a 'method 3' (the methods refer to the way sources are used), it would just be an option. To be honest though you'll probably not get much benefit from using AS if you don't want to record all the information that census sources can give you. You may as well just use FH to do it

Mike Tate said:
Remember that AS is produced for free (or a small donation) by another FHUG member
I also accept large donations :lol: Actually I've had a couple of very large donations over the years from people who feel that AS has saved them hundreds of hours of work - it's lovely when that happens and always manages to guilt-trip me into doing further work on AS!
Nick Walker
Ancestral Sources Developer

https://fhug.org.uk/kb/kb-article/ancestral-sources/
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

Like mitcheip, I have hundreds (1000+) of censuses created in TMG using witnesses, a method I like, despite its drawbacks, but am ready to forego. Moreover I can already see that Ancestral Sources is a great tool to assist in census and other data entry, for which I will be using Method 1. My query is more to do with what mitcheip went on to say, which was about running FH with "old" imported TMG census events alongside "new" FH/AS census events. I note that tatewise said that a conversion "could be performed quite easily with a custom Plugin, and not beyond the capabilities of one of the TMG migrants I am sure." Well, it is certainly beyond my capabilities, and I wonder if any TMG refugee or anyone else has created such a plugin?
Nigel Brown - https://vousden.one-name.net
Vousden One-Name Study - https://vousden.one-name.net
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

See how_to:recording_census_records|> Recording Census Records for a summary of the pros and cons of Census Events versus Witnesses.

If there were sufficient interest, I would be able to create the suggested Plugin using techniques from my Export Gedcom File Plugin, that synthesises Witness Individual Events from the Witnessed Event, so I await developments.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

Thanks Mike. Its not exactly that I hope other TMG refugees are having the same "problem" as me, but perhaps there are some out there who would benefit too... Nigel
Nigel Brown - https://vousden.one-name.net
Vousden One-Name Study - https://vousden.one-name.net
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

I have been working on the Plugin, and got a prototype going.

Currently, it lists all the Individual Fact Witness Roles to let you tick the ones for which to synthesise separate Individual Facts for each Witness.

Each synthetic Fact is a duplicate of the principal Fact (including Media and Source Citations) except as follows:
  • The Age field is NOT copied.
  • The Witness details are NOT copied.
  • The Note field is copied, and has the following appended:
    • Labelled name of the Role, e.g. Witness Role: Resident.
    • The specific Witness Note field if any.
Finally, a Result Set is produced for all the changed Principal Facts and new Witness Facts.

Note that Family Facts (such as Marriage & Divorce) are excluded, because it is not possible to synthesise an Individual Fact from a Family Fact for the Witnesses.

Does that sound like what you expected?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

Mike,

This prototype plugin sounds like it will save me - and no doubt other users - a huge amount of time in achieving a single census approach combining "old" TMG and "new" FH census events.

Not copying Ages is something I can live with, or rather, work around. I understand what you say about Family Facts being excluded, again, no problem with that.

I have two questions: firstly, what Individual Fact Witness Roles do you envisage I might NOT want to tick for synthesising separate Individual Facts? And secondly, what sorts of Witness details are not copied? I am sure your answers will help me better understand the process.

So, all in all, your description seems very promising indeed, with few limitations.

I will be pleased to test the plugin on my imported TMG project, because for the time being - not much longer hopefully - I am still able to re-import the same data at any time.

Thank you very much for your work on this. I am looking forward to the end result.

Nigel
Nigel Brown - https://vousden.one-name.net
Vousden One-Name Study - https://vousden.one-name.net
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

Nigel,

The Age field is not copied, because it only applies to the Principal, and not any of the Witnesses.
i.e. For a Census it is the Age of the Principal shown in the Census Return, whereas the Age of each other person will be different, and must be added manually later.

I can only guess what Witness Roles TMG users might have created (the Plugin is intended for everyone); perhaps the Minister or Mourners at a Burial Event, where converting those Witnesses to use Burial Events would not be meaningful because they have not been buried.

The Witness details are those that are being split off to synthesise Witness Facts. If they were copied, then every synthesised Fact would have Witnesses linked to all the other Witnesses. So there would be multiple Facts listed under each Individual; one synthetic Fact, plus one for each Witness link (with the blue arrow on left).

Where the Witness gains a synthetic Fact, the original Witness details are deleted from the Principal Fact. The Principal Fact will retain Witnesses for Roles that were not ticked, and also any Name Only Witnesses.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

Nigel, I have a Give Witnesses Their Own Facts Plugin Version 0.1 for you to try.

I advise that you use File > Backup/Restore > Small Backup (Gedcom only) before using this Plugin on your Project just in case!

Click on the ATTACHMENT below to install the Plugin into FH.

If not sure how to use Plugins see plugins:about|> About Family Historian Plugins.

[EDIT: Attachment deleted as now published in the FH Plugin Store.]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

Mike,
Re: Give Witnesses Their Own Facts.fh_lua 0.1

Download is fine and installation appears to be fine. But on running the plugin there is an immediate error message ("Family Historian has stopped working. Windows is checking for a solution..."). Have tried several times all with the same result. All other plugins working OK.

I am running a PC Win 8.1 x64.

Nigel
Nigel Brown - https://vousden.one-name.net
Vousden One-Name Study - https://vousden.one-name.net
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

That is rather odd, especially as it is quite a small simple Plugin, so please check a few things.

Confirm you are running FH V6.0.4 by using Help > About Family Historian.

Try the Plugin on the Family Historian Sample Project, as I know it works there.

Try downloading & installing the Plugin again, as sometimes there is a download glitch.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

Mike,

Yes, I am running FH version 6.0.4.

Yes, seems OK with Sample Project.

Have downloaded plugin again, and re-installed.

But same problem on my project (imported from TMG).

???

Nigel
Nigel Brown - https://vousden.one-name.net
Vousden One-Name Study - https://vousden.one-name.net
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

OK, so it is something unusual about your imported Project.
So one thing to try is File > Validate to check for major errors.
Then use the UDF List plugin mentioned in how_to:handling_unrecognised_data_fields|> Handling Unrecognised Data Fields to see how many UDF are in your Project.

Also, please try the attached Plugin V0.2, which will replace the previous one.
When run, it should report when it has "Finished finding Fact Witness Roles".
Click OK, and then it should report the parameter text for the user dialogue.

[EDIT: Attachment deleted as now published in the FH Plugin Store.]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

Version 0.2 is certainly better, but not quite there yet.

It runs, and produces the "Finished finding Fact Witness Roles" window, as you say.

However, I can't click OK, because the popup window lists so many entries it disappears off the bottom of the screen (and its a large monitor!), and your OK box at the foot is out of sight i.e. the window needs to be scrollable. At this point the only option is to close the window by clicking on the "X" in the top right, which produces an error again ("Family Historian has stopped working. Windows is checking for a solution...").

So I can't finish the process.

Nigel
Nigel Brown - https://vousden.one-name.net
Vousden One-Name Study - https://vousden.one-name.net
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

I suspected it might be the number of Witness Roles - there must be a lot!!!

Temporarily, I have limited the Plugin to the first 50 just to prove that is the problem.

Do not worry about clicking OK, as clicking the X, or pressing Enter key does the same.

So try the attached variant.

[EDIT: Attachment deleted as now published in the FH Plugin Store.]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

FH Witnesses.png
FH Witnesses.png (7.02 KiB) Viewed 17370 times
You're right, there are a lot.

This time the long list is topped and tailed (top and bottom of window both off screen). But I hit Enter and get an error message, a new one this time. I have copied this part of the screen and attached it (I hope) as "FH Witnesses.png". I guess you will understand it.

Thanks for your time and patience.

Nigel
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

Often the trickiest aspect of Plugins is catering for the vagaries of all users, and designing the user dialogue. In this case the large number of Witness Roles was the stumbling block.

The predefined dialogue I chose to use currently only allows 50 parameters, and Nigel has more than that. But 50 parameters are too many to fit onto one screen anyway, so now the Plugin presents them in batches of up to 25 per page.

Try the attached Plugin Version 0.3, but keep clicking OK on each page, even if no ticks on that page, otherwise the Plugin will abort.

[EDIT: Attachment deleted as now published in the FH Plugin Store.]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
NigelBrown
Diamond
Posts: 68
Joined: 27 Apr 2015 21:12
Family Historian: V6.2
Location: Wolverhampton, UK
Contact:

Re: Witnesses when entering Census data

Post by NigelBrown »

Mike,

It works! And on the whole it appears to work very well.

As you say, in TMG I was working with a lot of roles, although I suspect a lot less than many TMG users.

I have noticed that a few of these have not translated well using this plugin. For example, my great great grandfather Thomas Brown married three times, and was widowed twice. In TMG I gave Tom the "role" of "widower" in each case, and it generated a sentence for Tom - "He was widowed by the death of Elizabeth... on such and such a date", which I think makes for a good narrative. However, after running this plugin, the widow role goes and the sentence generated now is "He died on..." such and such a date. Of course, he did not die at all as a result of the event of his wife's death. So in Tom's case, having run this plugin he now has three deaths, but two are the deaths of his wives, and finally, one is when he did die.

So, I think I have some cleaning up to do, but I think a lot less than re-inputting 1000+ census sources. So my first impressions are that this plugin is a valuable time saver.

More reflectively, it demonstrates the inherent difficulties in any one software importing the data of anothe software. Having tested my TMG project in Legacy, Rootsweb and FH, FH is undoubtedly the strong front runner, and this plugin adds more to the process.

So thank you for sticking with it, and coming up with a good solution to my problem.

Nigel
Nigel Brown - https://vousden.one-name.net
Vousden One-Name Study - https://vousden.one-name.net
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

I am glad the Plugin is working OK.

But the description you give is an example of why care must be taken in choosing which Witness Roles to give their own Facts, because in many cases it is better to leave them as Witnesses.

Earlier you asked: "What Individual Fact Witness Roles do you envisage I might NOT want to tick for synthesising separate Individual Facts?" Well, I think you have answered that question.

You should only choose Witness Roles where effectively they are all equivalent principals, such as Census Witnesses.

If you chose not to use Edit > Undo Plugin Updates after running the Plugin, then I hope you heeded my earlier advice and used File > Backup/Restore > Small Backup. If so, you can easily get back to where you started, and run the Plugin again with different Witness Roles ticked.

Also you do not have to tick all the Witness Roles at once, that you think may benefit having their own Facts. Just do the obvious ones first, and when happy, repeat the Plugin with some others ticked. I presume you know that by double-clicking the Facts in the Result Set you can review each change in the Property Box. Then if any are unsatisfactory, use Edit > Undo Plugin Updates to reverse all the changes.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Witnesses when entering Census data

Post by tatewise »

The Plugin has been published in the FH Plugin Store, and is referenced in how_to:import_from_tmg|> Import from The Master Genealogist (TMG) and how_to:recording_census_records|> Recording Census Records.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply