* Problem with Lookup Missing Census Facts

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Yes, try this proof of concept Lookup Missing Census Facts Plugin Version 4.2.3 Date 19 Feb 2018 with some new features.
[ Attachment deleted as superseded by Plugin Store version. ]
  1. In addition to EMIG & IMMI it now also monitors BIRT & DEAT events with any form of non-blank Date and Place.
    Also it monitors every Fact with a simple Date defining just Day & Month & Year and a Place.
    It checks the Place of such facts just before & after a Census Date to determine if the person is at home or abroad.
    So it does now detect where a person is born abroad and omits the Census records as requested by Bill & John.
    BTW: It would be easy to add any other events to the list of those monitored with any form of non-blank Date.
    e.g. BAPM, CHR, BURI, CREM, OCCU, RESI, and perhaps MARR?
    Remember this only applies when Use their Facts to decide if each person was at home or abroad is ticked.
  2. Previously, missing Census records where a web site search is not available were shown with just the year & n/a in black.
    Now the full Census Date & Locality is shown without n/a but in grey, i.e. greyed out.
    This allows missing Census records to be properly reported even when there is no online web search available.
    e.g. 2 Apr 1911 Scotland and 19 Jun 1921 UK and potentially any country such as Norway and Germany.
    BTW: Currently this is only fully implemented for UK Census dates, but easily added to the rest.
  3. A new option to Exclude any missing Census Records that have no online genealogy web site search available ?
    This excludes any row where 'greyed out' missing Census Records appear in every active web search column.
    i.e. No online search is available for any of the currently chosen lookup web sites.
    e.g. If FindMyPast is disabled then 29 Sep 1939 Register would be excluded from search page.
  4. Previously, every Census Date was included in the checks, regardless of lookup web sites chosen, etc.
    Now, any Census Date excluded by the new option above, is not checked, making the Plugin more efficient.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

The plugin seems to be working great for excluding those folks that never emigrated to the USA. I'll keep plugging away and check some more, but at first glance it seems to be working.

You said "So it does now detect where a person is born abroad and omits the Census records as requested by Bill & John."

I'm assuming you really mean born abroad and never emigrated to or immigrated into the country in question. Is that right?

Thanks!
Last edited by BillH on 19 Feb 2018 23:55, edited 6 times in total.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Yes, correct.
It now also seems to correctly detect multiple migrations through various countries.
This should even work if there are no explicit EMIG/IMMI events, as long as there are enough Facts with Dates & Places in each of the countries.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

Thanks for all the hard work on this. It really cuts down on the size of the report and makes it much easier to find real missing censuses.

Thanks!
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

I am now considering polygamy, and married surnames, etc.
On reviewing the way female surnames are determined, I'm not sure the most common convention is being applied.
Before marriage the lookup search uses the woman's maiden surname.
When married it uses her married surname.
Currently, when a women is widowed, divorced, etc, the lookup search reverts to her maiden surname.
BUT is it more normal for a woman to retain her married surname after being widowed, divorced, etc?
Especially in the years covering online searchable Census records from about 1800 - 1940.

I am proposing that widowed, divorced, etc, women retain their married surname in the lookup searches.
Until that is they perhaps marry again, when their new married name will apply.

To ensure monogamous and polygamous wives have the correct partner relies on monogamous wives having a Divorced, Divorce Filed, Annulled, or Separated family event, or their partner dies.
Does that sound reasonable?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
jbtapscott
Megastar
Posts: 513
Joined: 19 Nov 2014 17:52
Family Historian: V7
Location: Corfu, Greece
Contact:

Re: Problem with Lookup Missing Census Facts

Post by jbtapscott »

Certainly my experiences to date are that Widows have retained their married Surname - this includes records on my tree back to the 1851 UK Census. The married surname has also been used when the widow has re-married (at least, that is my experience with GRO records).

In the case of Divorces though, I have a mix - some women have retained their married surname while others have reverted to their maiden surname.
Brent Tapscott ~ researching the Tapscott and Wallace family history
Tapscott & Wallace family tree
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Thanks Brent, that confirms my latest asumptions. The Plugin cannot know which surname a woman used in Census records, but it is helpful if it matches the majority of cases. The user must edit the web search filter for other cases.

Earlier, when talking about Norwegian Census records, Bill said
The only national searches were for 1801, 1865, 1875, 1891, 1900, and 1910 that are available so far. Some of them were over several days, but isn't that like the USA where some enumerators took weeks to finish.
Regardless of how long enumerators took to finish their records, the USA, UK, and other national Census events are conventionally considered to have taken place on one unique Date. For the purposes of this Plugin, it would be easiest if the same applied to Norwegian Census events, and the first of the alternative Dates used. Would that pose a problem with your data? You might have to adjust some Census Events to use the conventional unique Date. I had thought that perhaps the Day could be omitted and just the Month & Year used to define the Norwegian Census Dates, but some use 31 Dec & 1 Jan of following year :roll: so that won't work.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

For my templates in AS I used 1 Feb 1801, 31 Dec 1865, and 31 Dec 1875. These were the dates I found on several sites which documented the enumerator instructions.

I used the date 31 Dec 1900, but have since learned that it should have been 3 Dec 1900. I haven't gone back and changed the AS template yet, it is on my to do list. Maybe you could go ahead and change my template in the Knowledge Base for me?

I haven't created any other templates for the Norway Census as I've not needed them yet. I believe for 1891 and 1910 the dates were 1 Jan 1891 and 1 Dec 1910.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Bill, I have adjusted your fhugdownloads:contents:template_census_norway_1801-1900|> Ancestral Sources Template ~ Census Norway 1801-1900 to use 3 Dec 1900.

The attached Lookup Missing Census Facts Plugin Version 4.2.4 Date 21 Feb 2018 is close to its final features.
[ Attachment deleted as superseded by Plugin Store version. ]

The new features are :-
  1. Monitoring of Facts is as per Version 4.2.3 as described in 1. on Mon Feb 19.
    The full list of key events is now BIRT, BAPM, CHR, MARR, EMIG, IMMI, DEAT, BURI, CREM.
    Plus, remember, any Fact with a full simple Date and a Place, and of course any CENSus event.
    Is everyone happy with that?
  2. Point 2. on Mon Feb 19 has allowed German and Norwegian national Census Dates to be added, but without any actual online lookup searches.
    Is everyone happy with this? Are any other countries or extra census dates required?
  3. As mentioned yesterday, any widowed, divorced, or separated women now retains their married surname in the lookup searches, until they marry again, as this seems to be the most common convention, especially in those days.
  4. The simple change above caters much better with polygamy, as each wife uses her husband's surname in lookup searches.
    The remaining issue is the lookup search including the spouse's name, and the Spouse Family at Census column.
    The spouse's name and Spouse Family at Census column should only be used if they are likely to be living together.
    If a person is widowed, divorced, or separated, the spouse's name is excluded, and the Spouse Family at Census blank.
    Before, if a person's spouse married again, or had a child by another partner, the first couple were assumed living apart.
    So the spouse's name was excluded, and the Spouse Family at Census column blank.
    But now to cater for polygamy, such Marriage/Child events do NOT have that effect.
    So users may now have to delete the Spouse's name filter to obtain a successful search.
    If necessary, the Plugin could have a Polygamy option that switches that Marriage/Child effect on & off.
    What do you think?
  5. Is there anything else I may have missed?
    I have made a few minor changes to make code more efficient, so please check out all the options.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with Lookup Missing Census Facts

Post by ColeValleyGirl »

Still works as advertised, Mike.

However, have you considered bigamy? :lol:

No, I don't really expect you to handle this (as it's a real edge case) but a woman whose husband married her bigamously usually reverted to her maiden name.
User avatar
johnmorrisoniom
Megastar
Posts: 901
Joined: 18 Dec 2008 07:40
Family Historian: V7
Location: Isle of Man

Re: Problem with Lookup Missing Census Facts

Post by johnmorrisoniom »

Mike,
The plugin now does not seem to allow for the fact that as far as UK census is concerned, the Isle of Man is part of UK (If the use facts option is selected).

Also, would it be possible to have a button to retrieve the last search/results page. I know where to find it, but others may not.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

John, it never did detect the Isle of Man or the Channel Islands.

The British place names it detects are:
England, Wales, Scotland, UK & U.K. for 1841 to 1901 and 1921 & 1951.
It does not detect UK & U.K. for 1911 and 1939 as the England/Wales and Scotland records are separate.

Can you work with any of those?

Do you mean the last web browser page of lookup searches?
If so, with my browser, each search creates a new tab until I choose to delete it.
So recent searches are instantly available.
Or are you talking about someting else?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
johnmorrisoniom
Megastar
Posts: 901
Joined: 18 Dec 2008 07:40
Family Historian: V7
Location: Isle of Man

Re: Problem with Lookup Missing Census Facts

Post by johnmorrisoniom »

It was last web browser page I meant.
I agree each search has it's own tab on chrome until you shut down or change computers.

Is it possible to include Isle of Man in the UK searches? 30% of my individuals are from the Isle of Man, and about 5% of those have emigration events.
The Manx can be found all over the World (Just like the Irish). You probably have a manxie just down the road.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

I'm not sure where the last web page would be retrieved from. Can you give me a clue.
Is the same for every browser?
If not, then I don't see how the Plugin can know which browser to retrieve.

So you would like Isle of Man added alongside England & Wales for the home/abroad detection?
Presumably you don't include England in their Place names.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

Thanks for updating the census template for me. Also thanks for the addition of the Norway and Germany reporting. This will be very helpful.

The list of facts in item 1 looks good to me.

As for item 4, I'm not sure I really understand it, but if the new processing for polygamy can upset the reporting and searching for non-polygamous individuals, then I would vote for having an option to turn it off. If it doesn't impact the reporting and searches for non-polygamous individuals, then I don't see a need for the option (at least in my case). I don't have any polygamous marriages in my data.

Thanks
User avatar
johnmorrisoniom
Megastar
Posts: 901
Joined: 18 Dec 2008 07:40
Family Historian: V7
Location: Isle of Man

Re: Problem with Lookup Missing Census Facts

Post by johnmorrisoniom »

The last browser page always seems to be saved to C:\ProgramData\Calico Pie\Family Historian\Plugin Data on every computer I have.

Yes adding Isle of Man would be excellent.
I have England, Wales, Scotland and Isle of Man set up as different countries in AS. I would not add UK to the Isle of Man entries, because we are not in the UK (Apart from census and VAT)

TIA
John
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

I may have just run into an issue with the new version although it may have been happening before and I didn't notice.

When I run the plugin using myself as the selected person, it shows someone as missing the 1930 USA Census even though they do have one. I've looked at it for awhile now and I can't figure out why.

Here is the census fact for that person.
image1.jpg
image1.jpg (60.41 KiB) Viewed 11593 times
Here is the report line.
image2.jpg
image2.jpg (63.32 KiB) Viewed 11593 times
Here are the options I have specified.
image3.jpg
image3.jpg (96.78 KiB) Viewed 11593 times
Any ideas why that person is showing on the report?

Thanks
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

@John, yes, I see what you mean - had a senior moment blind spot - I'll look at a button for that.

@John, will add Isle of Man and the Channel islands in some form too.

@Bill, sorry, I cannot explain that phenomenon from the details provided.
Please double check the [Record Id] in the search web page versus that of the 1930 Census 9 year old.
What other facts are just before , exactly on, and just after 1 Apr 1930?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

Definitely the same person (my mother :D). Same record ID in both places.

I will send the fact info by private message.

I don't know if it will help, but I had myself selected for the plugin and it was my mother that showed up on the report as the first line in the report.

Thanks

Edit: I downloaded the version of the plugin from the store and she does not appear on the report. She also does not appear using version 4.2.3. With version 4.2.4 she does appear on the report.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Bill, what happens if you untick the Use their Facts to decide if each person was at home or abroad option?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

With that option not ticked, she does not appear in the report.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Thanks, Bill, those details helped solve the problem, which was caused by some strange behaviour in the way the Lua table sort algorithm behaves, and occasionally resulted in the list of facts not being in Date order :roll:
BTW: The Plugin sorts facts in case they were not Date ordered, which can happen after using AS or adding new facts, and has upset the Plugin in the past. The inclusion of extra facts to check in the latest version exacerbated the strange sorting behaviour.

Please try attached Lookup Missing Census Facts Plugin Version 4.2.5 Date 22 Feb 2018.
[ Attachment deleted as superseded by Plugin Store version. ]

It has all the features from earlier, plus:
  1. Isle of Man and Channel Islands are recognised as home place names for the appropriate UK Census records.
    Please check that works for you.
  2. Redisplay the Last Lookup Page button added, but does NOT redisplay the last Result Set.
    Is that OK?
  3. Recognise polygamous relationships ? option added.
    That affects what is shown in Spouse Family at Census column and associated Spouse added to lookup search filters.
    By default each person can only have one Spouse Family and one Spouse at the time of any one Census.
    With this new option ticked, a Wife can have a Spouse Family with the same Husband as another Wife at the same time, and vice versa. However, only one Spouse Family and one Spouse per person is applied by the Plugin to one Census. i.e. If a Husband has two Wives then only one of them will be applied by the Plugin to any one Census. As always, the user can edit the search filters when using their genealogy web site.
  4. Helen said: "A woman whose husband married her bigamously usually reverted to her maiden name."
    That may be, but what FH Event, or any other data, could trigger that change of name in the Plugin?
    I imagine that since the Marriage was illegal there will be no Divorce or Separation event.
    Presumably, until bigamy was proven, both 'wives' would use his surname, just like polygamy above.
    Anyway as always, the user can edit the search filters when using their genealogy web site.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with Lookup Missing Census Facts

Post by ColeValleyGirl »

Mike, I did say I dind't expect you to handle bigamy -- as you rightly say there's no event to signal the change in state.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Understood. I just wanted to make it clear to any others following this thread.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
johnmorrisoniom
Megastar
Posts: 901
Joined: 18 Dec 2008 07:40
Family Historian: V7
Location: Isle of Man

Re: Problem with Lookup Missing Census Facts

Post by johnmorrisoniom »

I Got an Error Mike

Code: Select all

[string "C:\ProgramData\Calico Pie\Family Historian\..."]:4557: invalid order function for sorting
stack traceback:
	[C]: in function 'sort'
	[string "C:\ProgramData\Calico Pie\Family Historian\..."]:4557: in function 'CensusMissing'
	[string "C:\ProgramData\Calico Pie\Family Historian\..."]:4735: in function 'RunLookup'
	[string "C:\ProgramData\Calico Pie\Family Historian\..."]:3308: in function <[string "C:\ProgramData\Calico Pie\Family Historian\..."]:3286>
	(tail call): ?
	[C]: in function 'MainLoop'
	[string "C:\ProgramData\Calico Pie\Family Historian\..."]:2327: in function 'ShowDialogue'
	[string "C:\ProgramData\Calico Pie\Family Historian\..."]:3351: in function 'GUI_MainDialogue'
	[string "C:\ProgramData\Calico Pie\Family Historian\..."]:4825: in main chunk
And yes , Last lookup Browser page not result set is good for me.
Post Reply