* Census Entry: Witness or Not?

[edmacke]

Currently when I enter census information, I

• Choose the head of household as the "focus" person
  Add a "Residence" fact
  For that "Residence" fact, I add all of the household members as "Resident" witnesses

Originally, I did this so that I could include witness info in my fact sentence (although, I'm kind of re-thinking that since a large family across multiple decades can result in a tedious, page-long census fact).

But even without the fact sentence, using witnesses seems to have other advantages:

• I can change the sentence for the fact tab and the list will show me who's included (w/o the verbose published sentence)
  Even without the fact tab sentence, I could always pop up the witness dialog to see who's included
  If I included a wrong person, it's easy to remove somebody and add somebody else
  If I have a typo in the resident fact, changing it once changes it for everybody

Recently, though, I started playing with Ancestral Sources, and it does not seem to utilize witnesses for census (resident) data. It seems to simply add a Census (or Residence) fact for each individual person.

The AS documentation does say something about witness being the default mode of entering things, but that doesn't seem to be the case.

So... am I doing something wrong with AS? Or is the fact that AS doesn't use witnesses a clue for me that it's not the preferred way of doing things?

[LornaCraig]

You are not doing anything wrong. Census entries in Ancestral Sources don't support witness roles as such, because Nick Walker (author of AS) decided that they are not really appropriate for census events. He gave his reasons in this thread in the Ancestral Sources Forum: Witnesses when entering Census data (12237)

[tatewise]

See also how_to:recording_census_records|> Recording from a Census Record that explains the pros and cons, and gives other Forum threads on the topic.

If you don't record Census events, as distinct from Residence facts, then it is more difficult to identify which Census records need further research. Some do not consider that a Census Address is proof of Residence as it may only be a temporary Address. Anyway, it is clearer if the Census event holds that Address at the time of the Census.

AS does support Fact Witnesses, but only in the role of witnesses to Birth, Marriage, or Death events where they are NOT the key focus of the event, i.e. they did not get born, married, or died, but witnessed those events.
For Census events, every member of the household took part, so are all the key focus of the event, rather than witnesses.

Note that Individual Summary Reports and Family Group Sheets do NOT include Fact Witnesses so the appearance in a Census is missing from all but the Head of Household's reports.

[edmacke]

Thanks for the quick reply, folks!

I read some of the background provided in the links, and I get the feeling that FH and AS seem to discourage using the witness role for Census.

I agree, a person living in a house isn't really a witness.

But at the same time, they do share certain attributes that are lost when individual events are created. If the census has 10 people in the house, certain data like date, place, address, and source is repeated 10 times - once for each individual's event. Any corrections to that data would need to be done 10 times (or I suppose one time for Places or Address via those Work With Data lists). Using witnesses means the data is for 1 event and any corrections automatically apply to all witnesses.

Another advantage to witnesses is that I can go to any person that's listed as a witness and see all the other witnesses. I don't have to bop around the child list to see if every sibling was correctly accounted for - I could just open up either parent or any child and see the same list in each.

Things to think about, I suppose........

[tatewise]

You are of course correct that the same data is recorded in each householder's Census event.
It was that type of consideration that lead to Place records, but unfortunately Address was not included with that.
However, as you say it is simple to correct the Place and Address via Tools > Work with Data.

What Witnesses don't allow you to record is the Age of each person derived from the Census records.
Also, if you record Occupation facts derived from the Census records they are out of context in most Reports and in Diagrams that do not include Witnesses, as you don't see the associated Census event.

The Lookup Missing Census Facts Plugin and Internet Data Matches both normally focus on the Census event, which is missing for most members of the household if Witnesses are used.

You don't have to 'bop around the child list' providing you put a transcript in the Text From Source of the Census Source record. Just follow the link from person to Source record using the Go To Source Record blue arrow in the yellow Sources For pane. Once there, if you need more details, then click Got To Record top right and run the Tools > Where Used Record Links Plugin.

As always with these options it is a personal choice.
But if you plump for Witnesses, and later change your mind, then the Give Witnesses Their Own Facts Plugin comes to your aid.

[AdrianBruce]

... I get the feeling that FH and AS seem to discourage using the witness role for Census. ...

Unless I am misremembering, it's not particularly that FH discourages the use of witnesses for censuses, rather that witnesses simply weren't present in FH for many years because they're not standard GEDCOM. So one important point is that life can get tedious if you ever need to transfer your stuff with witnesses to a program that doesn't use them.

What you say about needing to make 10 amendments if you need to alter something about a 10 person census form is perfectly correct. That's actually an argument for a multi-person event, which GEDCOM simply doesn't do. That would be different from a single person event with multiple witnesses because that has one main person, whereas a multi-person event doesn't identify any one person as the main one. (That's the Data Analyst in me speaking.)

[tatewise]

Exporting Witnesses to a program that does not support them is not a big deal when using the Export Gedcom File Plugin, because as stated in how_to:recording_census_records#census_witness_roles|> Census Witness Roles it creates synthetic Census events for each of the Witnesses including the Citations. It will do that even for unspecified target programs if the GEDCOM 5.5 option is chosen.

[edmacke]

What you say about needing to make 10 amendments if you need to alter something about a 10 person census form is perfectly correct....that's the Data Analyst in me speaking.

Yeah, I almost mentioned in the original post about the downside of de-normalized data

[edmacke]

...it's not particularly that FH discourages the use of witnesses for censuses...

Fair enough. Maybe it's more just that the per-person census event seems to be encouraged?

As always with these options it is a personal choice.

Yeah, one of the great things about FH is that it seems to give you lots of choices.

For me, my choice to use witnesses (and a single source, e.g. I have exactly one "1940 United States Census", which is "Method 2", I believe) wasn't some well-thought-out evaluation, it's just something I stumbled into and seemed to work fairly well. It also made sense from a data normalization and a maintenance standpoint.

Really, the only thing I didn't like was the tedium of entering witness-based census events in FH.

So I started to look at Ancestral Sources. And AS (which really seems to know what it's doing) doesn't even support witness-based facts, and the default is "Method 1". So I began questioning whether my current choice is really the best approach. I feel like I'm kind fighting the program's natural inclinations, which is generally a bad sign in my experience.

...you put a transcript in the Text From Source of the Census Source...

I'm using ancestry.com, and I don't see any easy way (or any way at all, really) of exporting the census record data to put into the "Text From Source".

I don't think manually typing in census data is viable (i.e. ain't no way I'm doing that! ). Any thoughts?

BTW, is this the "Method 1" approach? I'm still kind of confused on that.

[E Wilcock]

This may not be relevant to family historians.
But my trees have been through successive software over the years and have no consistent method for census entry. I adopted witnesses to save time in TMG at some point and dropped them after moving to fh as the text search is rapid and effective.

I have not used AS for data entry for institutions. But the copy and paste commands in fh allow one to paste the same census entry and citation into the records for multiple, often unrelated people. As recommended to me by Mike, the census address is in the Place field for each person's census entry and is also in the Text from source field in the citation. Again one can copy and paste.

Since this information is identical for each person, it is easily retrieved by a simple fh text search for that address.
Moreover, if one wants to work though the people in the household, names can be selected and put in a temporary named list.

[ColeValleyGirl]

If you use AS and Method 1 (one source per census per household -- or 'Splitting') instead of method 2 (one source per census -- or 'Lumping'), you don't need to manually type in the whole transcript -- you can produce the whole grid automatically once you've entered the relevant data into AS -- you will still need to check it, depending on how rigorous you want to be. For example, I have a John James born in Llanfair Nant Y Gof, Pembrokeshire but some censuses say Llanvair Pem -- so I enter the right place in the data but correct the transcription to match the image. Others would be less fussy

Alternatively (but less usefully) when viewing the image on Ancestry, you could bring up the list of people at the bottom of the image and copy and paste the group you want (plus copy and paste the heading.) The disadvantage of this is that you'll have to fiddle with the formatting.

[tatewise]

The complication is that the Census option of using Fact Witnesses in FH V6 has blurred the pros and cons of Method 1 'splitting' versus Method 2 'lumping' that were much more distinct in earlier FH versions.

Before Fact Witnesses, if you wanted to avoid duplicated Text From Source transcripts and linked Media images, then Method 1 was the only solution, because Method 2 required that every household member's Citation had a copy of the Text From Source and Media.

But with Fact Witnesses, only the principal 'head of household' has a Census event Citation so it holds the only copy of the Text From Source and Media, and links to a generic Method 2 source record.

However, if you also want other facts such as Occupation and Birth or peoples Names to have Citations for the same Census household, then those Citations will duplicate the Text From Source and Media items. Thus Method 1 becomes the better option from the data normalization and maintenance perspective.

[edmacke]

I think I'm going to have to play around with a test file and AS to see how all of this plays out. I'm thinking Method 1 might be the way to go, given ColeValleyGirl's info on how that avoids manual "Text From Source" data entry - nothing about entering "Text from Source" sounds pleasant!

Question: If I use the plugin mentioned above to convert my current witness-based census data to per-person entries, do I end up with "Method 1" or "Method 2" facts?

[edmacke]

Question 2: I currently have my census data recorded in the Residence fact? Any easy way to change those to Census facts? I haven't looked, but could I just do a "change all" in the GEDCOM file from RESI to CENS (or whatever the fact key is)?

[LornaCraig]

Re question 2: It depends whether all your residence facts are actually census facts. Do you have some residence facts that are not related to censuses, so shouldn't be changed? If so you need to filter by date. There is a Change Any Fact Tag plugin that includes a date filter.

[tatewise]

Re Question 1: Earlier you said you 'have exactly one "1940 United States Census", which is "Method 2"'.
The Give Witnesses Their Own Facts Plugin simply replicates the Fact and its Citations, so in your case will still be Method 2. i.e. every 1940 Census/Residence fact will cite the one 1940 United States Census source record.

It sounds as if you would like the Plugin to create a Method 1 Source record with its Text From Source and Media copied & deleted from the principal Citation and perhaps given a Title based on the principal's Name?

But are there any other Facts that might Cite those Method 2 source records for US Census years?
If so, then that would complicate matters, as they would need to be changed in tandem.

Re Question 2: Lorna has said it all.

[edmacke]

Do you have some residence facts that are not related to censuses, so shouldn't be changed?

Well, of course I do! Otherwise it would be too easy

[tatewise]

So you can't "just do a "change all" in the GEDCOM file from RESI to CENS" as you intimated, which suggested that maybe you did have only census RESIdence facts.

That is where the Change Any Fact Tag Plugin helps, because it has a Date filter.

So, sticking with the 1940 US Census theme, you would set the Date filter to 1 Apr 1940 so only RESIdence facts with that date get changed to CENSus facts. If you have only entered the year, then 1940 will do, but runs the risk that some RESIdence facts may get changed by mistake.

You then repeat the process with the Date filter set to 1 Apr 1930, then 1 Jan 1920, then 15 Apr 1910, and so on. BUT practice on a Copy of your Project, or make sure you have a GEDCOM Only Backup.
Although, Edit > Undo Plugin Changes is available before you Close FH, and there are GEDCOM Snapshots to fall back on.

BTW: After they have all been changed to CENSus events, and if the Dates are not all full day/month/year, then AS has a Census Date Check tool to correct them all.

[edmacke]

It sounds as if you would like the Plugin to create a Method 1 Source record with its Text From Source and Media copied & deleted from the principal Citation and perhaps given a Title based on the principal's Name? Are there any other Facts that might Cite those Method 2 source records for US Census years?

Ummm..... I think I need some Advil and/or a scotch....

Let's see if this helps. Currently I'm at Point A, which is:

• Most (all?) of my census data uses witnesses
  All of my census data is stored in Residence facts
  If a citation is specified for Residence fact, it uses "Method 2"
  There are non-Residence facts that also cite the "Method 2" census sources
  None of my current census sources have "Text From Source" or Media

Where I'd like to go is Point B, which is:

• All of census data is stored in Census facts (via Change Any Fact tag plugin??)
  Each witness has his/her own Census fact (via Give Witnesses Their Own Fact plugin??)
  Method 2 sources are converted to Method 1, i.e. each family gets its own "1940 United States Federal Census - [Unique ID of some sort]" source

I'm not that concerned about populating the Method 1 "Text From Source" field. Nice to have, I suppose, but not mandatory

Also, if it makes things easier, I could somehow do a manual clean-up on non-Residence/Census facts (e.g. "Birth") that currently reference Method 2 sources.

Is this possible??

THANKS!!!

(See screenshots that show current census sources, and a report for where the 1940 United States Census is used)

[NickWalker]

What you say about needing to make 10 amendments if you need to alter something about a 10 person census form is perfectly correct....that's the Data Analyst in me speaking.

Yeah, I almost mentioned in the original post about the downside of de-normalized data

I just wanted to jump in here and make a comment on this. I am also a database designer and I've never liked the way that GEDCOM forces a lot of this data to be duplicated - I prefer method 1 because it requires less duplication but ultimately you are quite right: if you make a mistake with a census entry you might need to change the place and address in a number of records. Witness roles didn't exist when Ancestral Sources was originally created and in my opinion don't really provide a good solution for census. As Mike mentioned the witness roles can't include age and you still end up with multiple facts for occupations, etc. so I really don't think they particularly solve any problems as far as census records are concerned.

[tatewise]

Ed, your summary of current Point A and target Point B with supporting screenshots is excellent.
It gives enough details to make comprehensive proposals.

However, may I suggest that target Point B needs a bit more elaboration.
You need to have a clear idea of how you will capture new Census (and BMD) records in the future.
For example, will you be using AS to create the Method 1 Source records and associated Facts and Citations?
If so, then re-capture some of those US Census records that only have one Citation.
( Having done that, you can Delete each associated redundant Residence fact. )

Then you will have a model structure that defines your target Point B more precisely.
It will define for instance, your [Unique ID of some sort] part of Source record Titles.

Then there are two techniques to achieve Point B:

1. Replace each Method 2 Source Citation with a Method 1 Source Citation manually, and then run the Change Any Fact Tag Plugin for each Census year, and finally run the Give Witnesses Their Own Facts Plugin.
   I suggest changing Source Citations manually, because you don't have too many Census households, and the non-Residence/Census facts (e.g. "Birth") must be done manually anyway.
   Trying to automate that part of the process with such relatively few cases would probably take longer.
2. Re-capture all your Census records using AS, and Delete the redundant Residence facts.
   AS automatically offers to replace Citations on non-Residence/Census facts such as Birth and maybe Occupation.
   This produces a consistent & complete set of data, with full Date & Age fields, plus Birth & Occupation details for every household member that technique a) does not, and many users find this review process reveals new insights.

Finally, review and correct any remaining Citations for the old Method 2 Source records.

[edmacke]

I just wanted to jump in here and make a comment on this. I am also a database designer and I've never liked the way that GEDCOM forces a lot of this data to be duplicated...

Hopefully I'm not blaspheming here, but I've always wondered why FH uses GEDCOM as its data source, due to GEDCOM's many odd structures and limitations, and not to mention that it hasn't been updated in 20 years.

If I were designing a genealogy program from scratch, I'd use a true database, maybe something like SQLite (or criminy, even XML / JSON / YAML), to store the data in a way that makes the most sense and is under the program's control - master of your own domain, if you will. Then the GEDCOM files could be created on demand (or on the fly) from DB data using a set of configurable rules.

But that's just me armchair-designing!

[edmacke]

For example, will you be using AS to create the Method 1 Source records and associated Facts and Citations?

Yes, that.

Then there are two techniques to achieve Point B...Re-capture all your Census records using AS, and Delete the redundant Residence facts.

You know, I think I may just go with that.

I thought I had a lot more census entries that I actually do, probably because it took so long without AS that it just seems like I've been doing it forever.

Honestly, re-capturing all my census data with AS would probably take the same amount of time as manually doing all the conversion, and at least with AS I know it would be right when I'm done.

[ColeValleyGirl]

Hopefully I'm not blaspheming here, but I've always wondered why FH uses GEDCOM as its data source, due to GEDCOM's many odd structures and limitations, and not to mention that it hasn't been updated in 20 years.

From https://www.tamurajones.net/ARealDatabase.xhtml

Calico Pie Family Historian

The best known genealogy application to use GEDCOM as its database format is Calico Pie's Family Historian, and Calico Pie's Family Historian is an exception to the rule. Developer Simon Orde knows what he is doing; he knows about databases, but deliberately does not use a database system because he considers them a bad fit for genealogy, and he is right about that, genealogy applications should really be using objectbases instead of databases. Alas, Family Historian does not use an objectbase but a GEDCOM file, and using a GEDCOM file instead of a database brings all the GEDCOM-as-a-database issues with it, but all these issues are handled well. Family Historian supports GEDCOM import and export to deal with differences between its and other GEDCOM dialects. Family Historian loads and saves its GEDCOM files quite fast, often faster than other desktop programs load and save their own database format. Auto-save works well, and does not get in the way, not even with large databases. Family Historian does not only use a temporary file to avoid losing data on a crash, but keeps multiple older copies, so-called snapshots, around, and allows you to switch back to an old version.
In the case of Family Historian, the use of GEDCOM as a database isn't a mistake, but a conscious choice by a developer who knows what he's doing. Family Historian is a fast and stable genealogy application that handles large genealogy databases with ease.

[edmacke]

Helen,

Thanks for that link - an interesting read! Good to know I'm not alone in questioning GEDCOM as a DB.

GEDCOM was designed as a data transfer format...not as a genealogy database itself.

That.

GEDCOM files contain data, but that does not make them databases

That.

...your main genealogy editor should use a real database system.

And that.

...but all these issues [of FH using GEDCOM as its database] are handled well. Family Historian...loads and saves its GEDCOM files quite fast...auto-save works well... [it] keeps multiple older copies...and allows you to switch back to an old version. [GEDCOM was]...a conscious choice by a developer who knows what he's doing.

But also that!!

Make no mistake, I love FH and think it works great. But, I would also go out on a limb and say FH is great because it's clearly built by people who are passionate about creating quality software, not because it's using GEDCOM.

Obviously, Calico Pie can use whatever underlying data structure they want, and they've done a great job so far using GEDCOM. I just wonder how much better it could be if it weren't married to a standard that stagnated 20+ years ago.

OK, that's my 2¢ soapbox. I'll shut up now!