* Lookup Missing Census Facts / BMD Records errors

Writing and using plugins for Version 5 and above.
User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Lookup Missing Census Facts / BMD Records errors

Post by BillH » 24 Apr 2017 02:48

I just noticed tonight that any time I click on any entries in the results list for the 1820, 1830, 1840, and 1850 USA Censuses, I get the following error.
image1.jpg
image1.jpg (50.37 KiB) Viewed 4554 times
I didn't try all possible census years, so there may be others as well. Some seem to be working fine such as 1870 and 1880.

I am checking U.S.A. Census Records 1790-1940 on Ancestry.com.

I'm sure all of these worked fine last time I used the plugin which was sometime last year.

Thanks

User avatar
mjashby
Superstar
Posts: 457
Joined: 23 Oct 2004 10:45
Family Historian: V6.2
Location: Yorkshire

Re: Lookup Missing Census Facts error

Post by mjashby » 24 Apr 2017 08:26

That message usually signal that Ancestry has a (temporary) problem with it's servers, which is not unusual, especially if weekend maintenance is taking place.

They might also be focussing some of that server maintenance work on the reported problems they are experiencing with their new Sync API. As I understand it Family Tree Maker & RootsMagic Beta Testers have been requested to 'heavily test' data uploading, downloading and synchronisation. while the Ancestry and software developers focus in on some problems already identified by testers. That could conceivably impact on 'normal' usage by others if specific data management tasks have to be adjusted.

User avatar
tatewise
Megastar
Posts: 17534
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Lookup Missing Census Facts error

Post by tatewise » 24 Apr 2017 09:23

No, it is because Ancestry have changed the database search codes for USA 1800-1860 Census collections.
It happens from time to time, and short of someone regularly checking all 45 Ancestry Census collections, and the FMP Census collections, plus now the BMD collections for the Lookup BMD Plugin, it is just something we have to live with.

Try the attached Lookup Missing Census Facts Plugin Version 4.1.1 Date 24 Apr 2017 that updates the codes.

If you have time, check as many Census collections as you can.

I may wait a while before updating the Plugin Store in case Ancestry are in the middle of changing any others.

[Attachment deleted]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
mjashby
Superstar
Posts: 457
Joined: 23 Oct 2004 10:45
Family Historian: V6.2
Location: Yorkshire

Re: Lookup Missing Census Facts error

Post by mjashby » 24 Apr 2017 10:04

Mike,

That's interesting, and it wouldn't surprise me if the changes weren't interlinked with the API issues that the Ancestry, FTM and RootMagic are attempting to resolve; so it may well be worthwhile, as you say, to wait and see if there are any wider changes that might impact on the Missing Census Facts Plugin; and/or any others that also interact with the Ancestry Website.

Mervyn

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 24 Apr 2017 16:13

Mike,

Thanks for the quick fix. I tried it on all years for the USA Census and it worked fine.

Thanks!

User avatar
tatewise
Megastar
Posts: 17534
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Lookup Missing Census Facts error

Post by tatewise » 26 Apr 2017 19:16

The attached Lookup Missing Census Facts Plugin Version 4.1.2 Date 26 Apr 2017 adds the FamilySearch.org web site lookups, and makes a few minor improvements to Ancestry and FindMyPast lookups too.

Please try as many options as possible to exercise all combinations, and let me have your feedback.

If all goes well then probably FamilySearch.org can be added to Lookup Missing BMD Records Plugin.

[Attachment deleted]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 26 Apr 2017 20:51

Mike,

I've tried it with Levi Henshaw and ran into a problem. It shows the 1840 USA Census in the results list. When I click on that it takes me to Family Search just fine, but Family Search says it can not find anything.
image1.jpg
image1.jpg (180.63 KiB) Viewed 4451 times
I then just hit enter again on Family Search without changing anything and it finds a census for Levi.
image2.jpg
image2.jpg (202.93 KiB) Viewed 4451 times
One thing I noticed is that the second search seems to have dropped his sex as a criteria on the search. I don't see a place to specify sex in the search. Are we sure this is a valid search criteria for the 1840 census? It seems to work fine on later censuses that I've tested. I tested 1850 - 1940 and they all worked fine.

Thanks

User avatar
tatewise
Megastar
Posts: 17534
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Lookup Missing Census Facts error

Post by tatewise » 26 Apr 2017 21:41

I have been getting similar odd results with Gender, but not having genuine USA ancestors found it more difficult to pinpoint.

If you scroll down the left-hand panel there is a blue Update button that does the same as pressing Enter.
Scroll down further and there is a Filter your results by: pane with Gender at the bottom offering Male, Female, Unknown.

However, I know that FMP is fussy about specifying a Gender filter only when the Census records hold Gender.
So maybe FS has similar constraints, although it appears not to be fussy about any other filters.
I know for FMP that USA Census records do NOT support Gender for 1790, 1800, 1810, 1820, 1830, 1840, 1890 & 1920.
It does support Gender for 1850, 1860, 1870, 1880, 1900, 1910, 1930 & 1940.

For the UK Census those from 1790 to 1831 do NOT support Gender and those from 1841 to 1939 do.
All other countries appear NOT to support Gender, but cannot be sure for Canada as FMP has no such Census records.

If your experiments confirm the above pattern, then I can make the Plugin use the same Gender filter rules for FS as for FMP.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 26 Apr 2017 22:32

Mike,

I see the filter section, but that only appears when you have some results showing. For Levi Henshaw, after hitting enter as mentioned above, I do get a result so I looked and for Gender it shows only "Unknown". Maybe this is because the 1840 census doesn't have gender on it?

In any case, there is no way to specify the gender in the search page on the web site that I can see so it is hard to test whether or not each census will work with gender specified. I know it didn't cause any problems for 1850 - 1940. I don't have that many people that have any results in Family Search or Ancestry for the censuses prior to 1840 so it is hard to test those. In any case, gender isn't of any use in searching in USA censuses prior to 1850 as it was not on the census form so I can't see any reason to pass it in your search for those years. As far as I can see you might as well remove it for 1790 - 1840.

As for 1920, I'm not sure why FMP doesn't like it because gender is on the census form for that year. Family Search didn't seem to have any problems with you passing it to them for the 1920 census, it worked fine for me.

I can't really tell if that is what is causing my problem with no results initially, but it seems like it is worth a try to change the plugin to not pass gender for 1840 and see if that fixes the problem.

Another thing I noticed is that I am getting an error when either the given name or surname is "?". That is what I use when I don't know that portion of the name. So I might have "?, Susan" if all I know is the given name or "Anderson, ?" if all I know is the surname. I know I can go down on the page and blank out the question mark, but I was wondering if it would it be possible and make sense for the plugin to just pass a blank instead of the question mark?

Another issue I'm having is that if someone was born in the colonies prior to 1776, I use American Colonies as the country name. Ancestry has no problem with this, but Family Search always yields no results. For these places I have a standardized name which uses USA as the country name. So... would it be possible and would it make sense for the the plugin use the standardized name if there is one?

Thanks

User avatar
tatewise
Megastar
Posts: 17534
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Lookup Missing Census Facts error

Post by tatewise » 27 Apr 2017 00:20

The FS filters for Gender are rather strange.

You are correct that US Census records that do not hold Gender are categorised with Gender = Unknown.
The others are categorised with Gender = Male & Female & Unknown.

The URL supports two forms of Gender filter.
1)
From FS home page select Search > Records to get to Search Historical Records.
Enter some details on left, and on right Browse all published collections to choose USA Census that has Gender.
That displays a form with Gender (and Race) as an option that uses the format +gender:M in the URL.
It show Sex: Male etc in the results on the right.
It has the Gender filter bottom left that uses the format &gender=M in the URL.
A major constraint is that only one Collection can be chosen for this method.
2)
From FS home page select Search > Records to get to Search Historical Records.
Enter some details on left, and then click blue Search button lower left.
That displays a form without Gender and with a Records and Collections tab.
The Collections filter allows up to 25 Collections to be chosen, which is required for some Plugin categories.
This never displays a Gender option even when a single USA Census that holds Gender is chosen.
But if format +gender:M is in the URL it shows Sex: Male etc in the results on the right.
It has the Gender filter bottom left that uses the format &gender=M in the URL.
This can be retained by ticking Lock filters just above the Update button.
The Update button clears the format +gender:M from the URL.

The Lock filters option cannot be set from the URL.

A complication is that only one Gender can be chosen, so for a Collection that has Gender = Male & Female & Unknown then if you search with Gender = Male then the Unknown records will not match, and that may be how the Individual is recorded.

I am not sure what the best strategy for Gender should be - maybe NOT use the Gender filter at all.

Regarding Names containing ? perhaps I need to experiment with what symbols FS does not like and exclude them.

I am not so sure about the Place names problem.
It is quite tricky to make a special case of FS, and the Plugin is designed to work with FH V5 that does not have Place records, plus other users may not use your Standardised naming strategy. Assuming all the web sites use Place names as they were at the time, then in most cases the Standardised name would be modern and so not match, especially in the UK.
Maybe there needs to be a Plugin option to use Standardised in preference, but it have to apply to all web sites, etc.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 27 Apr 2017 04:14

Mike,

My searches seem to work just fine when I don't specify a gender for 1790 - 1840. I think you can probably just leave it out for these census years for Family Search.

That would be great if you could dispose of any special characters that FS doesn't like such as the "?".

Don't worry about the place name problem. You made some good points and I wouldn't want to mess it up for other folks. For now, I can just change it from "USA" to "American Colonies" after it gets displayed on FS. This isn't the first time that using "American Colonies" has caused me grief. It seems like most web sites just call places USA even if it was before 1776. I may have to just go back to using USA regardless of the date. I'll have to think about it. It rubs me the wrong way to have to do that, but maybe I just need to give in and go with the flow.

If I do that, is there any easy way to change all occurrences of "American Colonies" to "USA" without reeking havoc on the place records? Last time I wanted to do something like this I used Edit > Find and Replace and it created new place records and it was a real pain to clean up the mess. Would doing a text find / replace on the PLAC and _PLAC records in the GEDCOM work OK? It seems like it would, but I'm a little gun shy at this point.

Thanks

User avatar
AdrianBruce
Megastar
Posts: 918
Joined: 09 Aug 2003 21:02
Family Historian: V6.2
Location: South Cheshire
Contact:

Re: Lookup Missing Census Facts error

Post by AdrianBruce » 27 Apr 2017 12:09

BillH wrote:... Another issue I'm having is that if someone was born in the colonies prior to 1776, I use American Colonies as the country name. Ancestry has no problem with this, but Family Search always yields no results. For these places I have a standardized name which uses USA as the country name. So... would it be possible and would it make sense for the the plugin use the standardized name if there is one?

Thanks
Family Search seems to have decided that there was a place called "British Colonial America" - there wasn't of course, it's a description, not a jurisdiction. BCA is the standardised place-name for that part of pre-1776 North America that eventually ended up in the USA. However, looking at https://familysearch.org/research/place ... anced=true, it seems to give "American Colonies" as a synonym / alternate for BCA - and not give USA or variants thereon as synonym / alternate for BCA.

So without knowing which FamilySearch collections fail when AC is entered, I'm slightly at a loss whether BCA might be a better bet than AC, which was going to be my suggestion.

(When I looked at some pre-1776 births in US censuses, they don't appear to have anything like USA in the birthplace - just "Virginia" say. Not sure how that's different from post-1776 on reflection...)
Adrian

User avatar
tatewise
Megastar
Posts: 17534
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Lookup Missing Census Facts error

Post by tatewise » 27 Apr 2017 14:13

Try the attached Lookup Missing Census Facts Plugin Version 4.1.3 Date 27 Apr 2017.

This only uses Gender filtering for Census years that set Gender to Male or Female in the vast majority of records.
Those are only UK Census 1841-1939 and USA Census 1850-1880 & 1900-1940 and Canada Census 1851-1945.
Similar rules are applied to both FMP and FS ~ (Ancestry seems happy with Gender filters for any records).
The anomaly for FMP in USA Census 1920 has been removed. Although its search dialogue does not have a Gender filter, it does support the URL filter..

For FS the Gender filtering uses the +gender:M URL format, which shows as 'Sex: Male' in the results page.
But if the Update button is used, that filtering gets erased, and thereafter the Gender filter lower left must be used.
This has the benefit of revealing how many records are set to Male, Female & Unknown.

Both Ancestry and FS use the characters ? and * as wildcards with associated rules about their usage.
So the Plugin now always removes those two characters from all Personal and Place name filters for all websites.

A quick experiment revealed that a Place name of America filtered the same as USA.
So you could change every American Colonies to America as an alternative to USA.
Use the Search and Replace Plugin to perform the global Place record change as advised in Knowledge Base > Search and Replace ~ Usage Examples > Place Name Changes in FH V6.

Generally FS appears not too fussy about what is in the Personal and Place name filters.
It just lists records in which parts of the relevant fields match parts of the filters.
But there are a few anomalies such as the American Colonies syndrome.

[Attachment deleted]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 27 Apr 2017 16:00

Adrian,

Thanks for the reply.
AdrianBruce wrote:So without knowing which FamilySearch collections fail when AC is entered, I'm slightly at a loss whether BCA might be a better bet than AC, which was going to be my suggestion.
I think I was searching the 1820 census from within the results of the plugin. In any case, if you go to the main search for FS and enter the name Basil Henshaw and for the birth place you put in USA you will find one record in the 1800 census. If you use American Colonies, British Colonial America, or BCA you will get no results. Interestingly America does work. I'll have to think about using that as it really doesn't indicate that it means pre-1776 and a lot of places still call the USA just America even today. One of the reasons that I chose American Colonies is that some of my folks were from Spanish colonies and some from French colonies as well as British colonies.

Thanks

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 27 Apr 2017 16:12

Mike,

This new version did fix the problem for Levi Henshaw in the 1840 census.

Thanks for the idea of using the plugin. That will be easier than using an editor directly on the GEDCOM.

I contacted Family Search about the American Colonies question. They were very vague in their answer about what they accept, but the person did say that I should not use anything in the search that is not on the record I am looking for. For example, it I am searching in the US census I should not specify anything other than the state as the birth place if I am talking about someone born in what is now the USA as that is all that is on the census record. For people born in foreign countries I guess I should just specify the country. This would be hard (if not impossible) to implement in the plugin as people do not all use the fixed column format and may not have the state in the same position in the place name even if they do.

I guess one option might be to just not pass any birth places at all. It also makes me wonder about using birth date, parents names, and other things in those censuses where these do not appear on the record. I'm not sure they really are helping at all in the search.

Thanks

User avatar
tatewise
Megastar
Posts: 17534
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Lookup Missing Census Facts error

Post by tatewise » 27 Apr 2017 19:23

Without having had the benefit of that advice, I have not performed any scientific checks, but as I said, my impression is that providing Names of relatives plus Birth Place & Date helps constrain the results where there is some sort of match and any non-matching parts are disregarded.

I do not really understand how that advice can be applied to all the record collections (not just USA) as other countries do not always have such a regular way of recording Place Names.
It also is a bit chicken and egg. Saying one "should not use anything in the search that is not on the record" assumes one knows in some detail what is on the record before applying the filters, which sounds ridiculous. i.e. You must perform an investigation of the record to understand the type of details it holds before one can extract components of your own data to act as filters. When it comes to names of relatives, that is nonsense, because different records in the same collection hold different relatives; sometimes spouse, sometimes father, sometimes mother, sometimes children, sometimes other, and sometimes any combination.

Perhaps if a few of us can use the Plugin and other FS searches to see if that advice makes sense or is just misleading.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 27 Apr 2017 20:00

Mike,

I agree. Their suggestions work fine if you are filling in the search form for a specific collection like the 1840 census for example. They don't really make a lot of sense when searching multiple collections such as all USA censuses as the plugin does. If the search as submitted by the plugin doesn't find anything or not enough things, I can always just modify the search criteria to be less specific and re-submit it.

I would have thought FS would just ignore the parts it can't use for any particular collection, but after their reply I'm not sure what they are doing.

So, I think the plugin is just fine as it is.

Thanks

avatar
Gowermick
Megastar
Posts: 738
Joined: 13 Oct 2015 07:22
Family Historian: V6.2
Location: Swansea

Re: Lookup Missing Census Facts error

Post by Gowermick » 28 Apr 2017 06:22

Whilst not directly related to the plugin, were you aware that the country filtering in family search is case sensitive?
Applying a country filter, I typed england, rather than selecting England from the drop-down list, and I got zero results! Normality only returned when I went back to England
Mike Loney

Using FH 6.2.7, with CC 6.7.37 Windows 10 Home, LibreOffice (x64), Firefox(x64) & Thunderbird
Website http://www.loney.tribalpages.com
http://www.mickloney.tribalpages.com

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 28 Apr 2017 16:11

Mike,

That is strange. I don't see the same results. I tried a few searches and I get the same number of results if I use England vs england, USA vs usa, Norway vs norway, etc. I don't see the drop down lists that you are referring to. How to I get those?

User avatar
tatewise
Megastar
Posts: 17534
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Lookup Missing Census Facts error

Post by tatewise » 28 Apr 2017 16:31

I plan to add some more web site specific guidance to the Plugin Help & Advice regarding the manual refinement of the search filters after using the predefined filters from the Plugin lookup web page. The advice will be specific for each of the three web site products: Ancestry, FindMyPast & FamilySearch. It will advise how to edit filters to increase or reduce the number of matching records.

I have updated the Lookup Missing BMD Records Plugin to also use FS, but I am mystified by the BMD collections provided, and perhaps more regular uses can elucidate.
I have used Browse all published collections and from Collections at the bottom selected Birth, Marriage, & Death.
Then I select a Place such as Canada and then Canada from the sub-list to reveal 5 collections that have few records.
Similarly, I select United States of America and then United States from sub-list to reveal 9 collections that also have few records except for US Social Security Death Index and US Genealogy Bank Obituaries.
Similarly, I select United Kingdom & Ireland and most of the collections have few records.
There is no Scotland Deaths and Burials collection although there is for England and for Wales and even Isle of Man.
There seems to be no comprehensive collection or search option for say all USA Births & Baptisms or all UK Marriages like there is in Ancestry and FMP.

How do you guys cope with this situation?

P.S: Yes Mike, where exactly are you applying the Country filter? The one I found would only allow England. The BMD Place filters I tried are case insensitive.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 28 Apr 2017 16:49

Mike,

I never use the section in the FS results called "Filter your results by:". I just click on Search, fill in the search form, hit enter, and browse through the results.

Most birth, marriage, and death records are in state level collections for the US. I don't know if there are any larger groupings of collections that include all the state level and country level collections.

Hopefully someone who searches individual collections will be able to help.

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 28 Apr 2017 16:53

tatewise wrote:P.S: Yes Mike, where exactly are you applying the Country filter? The one I found would only allow England. The BMD Place filters I tried are case insensitive.
Mike (Gowermick),

I think I'm confused. Is it when filling in the search criteria prior to the search that you get different results for England vs. england or is it when using the search filters after the search has been preformed? In my earlier reply I was talking about using England vs england in the search criteria. I think Mike (tatewise) is talking about in the filters after the search has been preformed. I never use these.
Last edited by BillH on 28 Apr 2017 17:03, edited 1 time in total.

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 28 Apr 2017 17:03

Mike (tatewise),

You probably already know this so this is probably not new to you.

If you click on Search on the main FS site, you get the main search entry form. On the right there is a link to Browse all Published Collections. You will see that there are 1283 Birth, Marriage, and Death collections. If you click on that, you will see a list of all of them. If you click on a geographical area you can narrow it down. For example, there are 454 BMD collections for the USA. If you click on United States of America (454) you get the list of all of the USA BMD collections. I don't see a name of a "group collection" that would include all of these, but the URL for the list is https://familysearch.org/search/collect ... TED_STATES if that helps at all.

avatar
Gowermick
Megastar
Posts: 738
Joined: 13 Oct 2015 07:22
Family Historian: V6.2
Location: Swansea

Re: Lookup Missing Census Facts error

Post by Gowermick » 28 Apr 2017 17:47

BillH,
I was on the Refine Your Search panel, where just above the Update blue button, is the Restrict records by and the Country drop-down list.

I had typed england, rather than selecting England from the drop-down list (error on my part), but it resulted in no results being returned.

I tried just now, and the problem was not replicated, as soon as I tabbed away from the country box, england was adjusted to England :oops:
Mike Loney

Using FH 6.2.7, with CC 6.7.37 Windows 10 Home, LibreOffice (x64), Firefox(x64) & Thunderbird
Website http://www.loney.tribalpages.com
http://www.mickloney.tribalpages.com

User avatar
BillH
Megastar
Posts: 1443
Joined: 31 May 2010 03:40
Family Historian: V6.2
Location: Washington State, USA

Re: Lookup Missing Census Facts error

Post by BillH » 28 Apr 2017 18:36

Mike,

Interesting. I didn't see it because I have never used the Restrict records by: section of the form before. Didn't realize it was there. I'll have to start using that!

I was just typing it in the section called Search with a life event: > Birthplace.

Thanks

Post Reply