* Problem with Lookup Missing Census Facts

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

I also got an error:
image1.jpg
image1.jpg (162.13 KiB) Viewed 13588 times
And the plugin hung at this point:
image2.jpg
image2.jpg (36.09 KiB) Viewed 13588 times
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Sorry folks, that is the same Lua sort function, but is very mysterious, and I cannot get it to fail.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

I have to run off for a couple hours right now, but if it would help I could send you my .ged file when I get back. Or, if you want to put some displays or something into the plugin to help determine what the error is, I could run it when I get back.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Bill, the GEDCOM file would certainly help.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

I just sent the .ged file by email. Hopefully it will help you figure out what is going on.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Please try attached Lookup Missing Census Facts Plugin Version 4.2.6 Date 22 Feb 2018.
[ Attachment deleted as superseded by Plugin Store version. ]

The Date Sort algorithm now seems to be much more robust, after making some significant changes.

Despite what Help says, the Compare dates function treats 9 Oct 1900 and Oct 1900 and 1900 as equal.
That prevents satisfactory sorting, so the Plugin now uses the Day Number of each Date.
BUT that fails for invalid Gregorian Dates such as 10 Oct 1582 or 10 Sep 1752 or any Julian, Hebrew, or French dates, and I have substituted the valid 14 Sep 1752 Gregorian Date as a most probable guess.
This whole area needs cleaning up by Calico Pie.

Bill, that GEDCOM certainly helped as it has so many Fact Date scenarios.
I will delete it as soon as these test are satisfactory.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

The error message is gone and the plugin runs fine now. Also, my mother is no longer showing up on my report. I'll keep playing around with it and let you know if I find anything unusual. At first glance things seem good.

I'm glad the GEDCOM file helped out.
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

I did find an issue of sorts. If you use my GEDCOM and select Sarah Ann Snodgrass (ID=52) the plugin runs fine and the report looks ok, but the progress bar gets to 6% after about 5 seconds and then hangs and doesn't go any further. If I wait the report does get created.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

I can see what is happening there, but need to think of best solution.
It is simply that the progress bar is not progressing but the Plugin is still running.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

MIke,

I see the same kind of thing when selecting quite a few different individuals. Sometimes it doesn't progress at all and sometimes in fits and starts. In any case, no big deal for me. The plugin runs to completion anyway.

Thanks!
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Well that was another learning experience!
The efficiency with which FH identifies related Ancestors, Descendants, or Relatives depends on various factors.
The functions such as =IsAncestorOf(...) and =IsDescendantOf(...) are governed by who is the File Root.

When the Plugin is run choose the File Root as the person whose Ancestors or Descendants you want analysed.
Then run the Plugin again, but choose someone else, and notice how much slower it runs.
Now set that other person as File Root and run the Plugin again with that person, and it runs MUCH faster.
Run again with the very first person chosen, and now it is slow.

Run the Plugin and change Choose the Individuals option to Anyone via the Select Records dialogue.
In the Select Records dialogue highlight the same person as before, choose Add Relatives > Ancestors (or Descendants) and the selection is instant, and the Plugin runs very fast.

I have rewritten the way the Plugin searches for relatives and indicates progress, but I cannot cure the slowness.

On another note, the =DayNumber(...) function objects to certain types of Date and returns nil which upsets Plugins.
If Tools > Preferences > Gregorian year is 1582 it objects to 5th - 14th Oct 1582.
If Tools > Preferences > Gregorian year is 1752 it objects to 3rd - 13th Sep 1752.
If the Date Calendar is anything but Gregorian then it objects.
So I have created a Plugin function to cope with all those cases.

Please try attached Lookup Missing Census Facts Plugin Version 4.2.7 Date 23 Feb 2018.
[ Attachment deleted as superseded by Plugin Store version. ]
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

I tested out the scenarios you suggested and indeed the speed difference is dramatic. I guess from now on I'll use the Add Relatives method to generate the report. The progress bar is working fine for me. The only thing that was kind of weird is that on initial installation of the new version of the plugin, the first run caused FH to crash. I restarted FH and have had no problem since. Not sure what happened there.

In any case, thanks again!
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Bill, during testing, I too had the occasional FH stopped working crash, usually when handling large numbers of records.
I am not sure whether the cause is a resource problem in FH or some unusual data in one of your records.

Now I have to add all the features from this discussion to the Lookup Missing BMD Records Plugin :roll:
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
johnmorrisoniom
Megastar
Posts: 901
Joined: 18 Dec 2008 07:40
Family Historian: V7
Location: Isle of Man

Re: Problem with Lookup Missing Census Facts

Post by johnmorrisoniom »

Hi Mike,
I too noticed the speed when using the add relatives method.
Just a point about the Isle of Man on Familysearch. The Isle of Man is treated as a country in it's own right, probably because a lot of higher ups in LDS have Manx roots.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

John, on FamilySearch I've found Record Collections for Isle of Man but only for BMD, Parish Registers, and Grave Index, but not for Census as far as I can tell.

I'll certainly consider that for the Lookup Missing BMD Records Plugin, but don't think it impacts Lookup Missing Census Facts.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Try attached Lookup Missing Census Facts Plugin Version 4.2.8 Date 25 Feb 2018.
[ Attachment deleted as superseded by Plugin Store version. ]

This adds Ise of Man, Jersey, Guernsey, Channel Islands to the place names for home/abroad assessment.
There are a few minor refinements to make the internal Day Number function more effective when estimating BMD dates.

Try attached Lookup Missing BMD Records Plugin Version 2.2.1 Date 25 Feb 2018.
[ Attachment deleted as superseded by Plugin Store version. ]

This incorporates all the enhancements from the Census Plugin.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Sorry folks, in reporting issues with comparing dates to Calico Pie, I found a more effective way than using Day Number, so here are two updated Plugins that use that new method to sort Facts into order, etc, that caused problems before.

Try attached Lookup Missing Census Facts Plugin Version 4.2.9 Date 26 Feb 2018.
[ Attachment deleted as superseded by Plugin Store version. ]

Try attached Lookup Missing BMD Records Plugin Version 2.2.2 Date 26 Feb 2018.
[ Attachment deleted as superseded by Plugin Store version. ]

BTW: These issues with functions continues in Challenges with Functions (15739).
I've reminded Calico Pie about the slow IsAncestorOf() and IsDescendantOf() functions, and others, reported 2 years ago.
I've provided more details of symptoms, including a Records Window Column for =IsAncestorOf(%INDI%,Record(52,"I")), which on my PC for Bill's Project of 19,000 people, takes over 30 minutes to sort by clicking the column header.
I pointed out that may relationship functions, such as the Select Records dialogue Add Relatives button, run very fast.
Calico Pie confirm that the File Root person has a relationships cache to speed up those functions.
So I suggested they create a cache for the root person when such functions are first called, so that repeated calls are efficient.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

Just to let you know the new version of both plugins seem to be working fine.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

I have had time to revisit these two Lookup Missing... Plugins and add some run time performance enhancements.
Firstly the process of finding Ancestors, Descendants, and All Relatives of a chosen person has been rewritten to search Parent & Spouse Family links instead of calling the slow FH Functions.
Secondly the process of creating the Web Search Page HTML file now uses much fewer FH resources.
Those, together with some other minor changes, help these Plugins cope much better with large Projects.

So try Lookup Missing BMD Records Plugin Version 2.2.3 Date 20 Mar 2018 and let me know how it behaves, especially when the 'chosen person' is NOT the File Root.
[ Attachment deleted as superseded by Plugin Store version. ]

I have not included Lookup Missing Census Facts as the changes are almost identical.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Problem with Lookup Missing Census Facts

Post by Valkrider »

Mike

Significantly faster than before. Works fine with non-root person.

However one little gotcha if you see the attached screenshot. I don't have a MyHeritage account and so used the 'My Heritage Lookup disabled' option but it still prints the header and the link underneath the blank header takes you to a url unknown page in the browser.
Capture.JPG
Capture.JPG (24.16 KiB) Viewed 13029 times
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

Thank you Colin, that oversight will be fixed in the published version.
(It was a minor change I added to Lookup Missing Census Facts that did not get across to Lookup Missing BMD Records.)
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
BillH
Megastar
Posts: 2245
Joined: 31 May 2010 03:40
Family Historian: V7
Location: Washington State, USA

Re: Problem with Lookup Missing Census Facts

Post by BillH »

Mike,

Works fine for me with the exception of the issue Colin already mentioned.

Thanks
User avatar
Tony Jones
Diamond
Posts: 65
Joined: 14 Oct 2016 09:27
Family Historian: V7
Location: Berkshire, England
Contact:

Re: Problem with Lookup Missing Census Facts

Post by Tony Jones »

Likewise - it runs like greased lightning for me
Using an iMac Ventura Intel via Parallels v19 / Win 10 64-bit
User avatar
ColeValleyGirl
Megastar
Posts: 5464
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Problem with Lookup Missing Census Facts

Post by ColeValleyGirl »

Good for me too, Mike.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Problem with Lookup Missing Census Facts

Post by tatewise »

These two Plugins are now in the Plugin Store and the FHUG attached prototypes will soon be deleted.
Lookup Missing BMD Records Plugin Version 2.3 Date 22 Mar 2018
Lookup Missing Census Facts Plugin Version 4.3 Date 22 Mar 2018
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply