* Trying to import Census Reference info for use in FTAnalyzer

Questions about exporting information from FH to other programs and websites
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Sorry Mike I had to pop out for a bit.

The desired format is: RG11, Piece: 2332, Folio: 85, Page: 24


I may have a solution but it requires a lookup into the Source > Note tab, select the note labelled #ORA which may not be the first Note and then make use of the last few lines that provide the source reference we need.

Here's an example
Looking at the attached image you will see a record in the Source > Notes tab.
The entry beginning #ORA contains all the source reference data.
The last few lines contain the Source Reference info we are looking for.
We can see the Class, Piece, Folio and Page in the correct sequence.
So we need to extract those four values and restructure them to meet the desired format; RG11, Piece: 2332, Folio: 85, Page: 24

If this is too much of a task just to help me. I'm quite happy to format the Templated Source > Reference field in order that we just then need a copy routine.
However, If this solution works for you, I will double check that all Sources have the #ORA Note. Which I believe they do.

Thank you so much for your support.
Attachments
45.png
45.png (67.49 KiB) Viewed 2186 times
User avatar
tatewise
Megastar
Posts: 28506
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by tatewise »

I've assumed that both the Reference metafield and Publication Info field need to be updated to the format:
RG11, Piece: 2332, Folio: 85, Page: 24

Try the attached Copy Census Source Reference to Publication Info plugin Version 0.1 Date 11 Jan 2024.

If it can't find a matching ORA Note it copies the Reference metafield to the Publication Info field instead.

If the Publication Info field already had a value then it is not overwritten.

If you want any other plugin behaviour then let me know rather than run the plugin.

The Result Set lists the new Publication Info field values for you to review.
If any look wrong then use Edit > Undo Plugin Updates immediately before closing FH.
Let me know how it goes.
Last edited by tatewise on 16 Jan 2024 21:44, edited 1 time in total.
Reason: Attachment deleted as a better version is attachd later.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Thank you so much Mike very much appreciated. I'll get this tested this morning. It seems you have covered everything.
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Hi Mike, initial testing has thrown up some unexplained issues.

Test prep
Copy FH Database
For All 1881 Census Source records delete the contents of the Templated Source Reference field, and delete using Query > Field Select Publication Information > Clear Contents

Double Check all #ORA Notes contain the info we wish to extract and reformat.

Run Plug for UK 1881
Viewing your result query we can see some success and some missing. The pattern behind the missing Publication info appears to be that the Templated Source > Reference field was not itself populated which then appears to prevent the Publication Info field from becoming populated.
Well that's how it looks at least.
See attached.

Thank you.
Attachments
60.png
60.png (217.73 KiB) Viewed 2145 times
61.png
61.png (230.91 KiB) Viewed 2145 times
62.png
62.png (261.73 KiB) Viewed 2145 times
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

One Request Mike, could the result query also include what is now populated within the Templated Source Reference field please. This makes it easier to check.

Thank you.
User avatar
tatewise
Megastar
Posts: 28506
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by tatewise »

The clue is in my plugin specification:
If it can't find a matching ORA Note it copies the Reference metafield to the Publication Info field instead.

However, you erased all the Reference metafields, so when the ORA Note does not match, the Publication Info field remains empty.
So the root cause is that the ORA Note does not match the plugin criteria which are:
RG11...digits...digits...digits
where ... can be any characters including newline and digits must be one or more digits 1 - 9.

I did not feel the need to include the Reference field in a column because the Publication Info field is always the same value.

What might be more useful is the ORA Note text in a column so it can be reviewed to see why it does not match.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Okay thanks Mike,

What exactly does If it can't find a matching ORA Note it copies the Reference metafield to the Publication Info field instead., mean? What is it trying to match with? I was hoping it would refresh both Reference and Publication Info so they are the same after the update.

I have checked all the #ORA Notes and they all look okay! I'll double check now.
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Okay I think I have found the problem, the successes we see all have the #ORA Note at the top of the Note window. In some sources there are more than just the #ORA Note. So where we see a missing result it is because the #ORA Note is not at the Top. See Attached.

That's the pattern I see...

Thanks Mike
Attachments
63.png
63.png (228.8 KiB) Viewed 2134 times
User avatar
tatewise
Megastar
Posts: 28506
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by tatewise »

The plugin is only looking in the 1st local Note instance.
It never occurred to me that there might be multiple local Notes.

Is it always the case that the ORA Note contains #ORA near the beginning?
If so then the plugin can use that to discover the appropriate local Note to search.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Yes Mike I use a FH 7 Hashtag = #ORA to create the start of the Note and add the ORA extracted data to.

So all will begin with #ORA but may not be the first Note record.
I have tried to columnize the ORA data but it will not do that without further programming.
User avatar
tatewise
Megastar
Posts: 28506
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by tatewise »

Try the attached Copy Census Source Reference to Publication Info plugin Version 0.2 Date 12 Jan 2024.

Where are you trying to columnize the ORA data?
Last edited by tatewise on 16 Jan 2024 21:45, edited 1 time in total.
Reason: Attachment deleted as a better version is attachd later.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

ORA allows the option to extract in a simple horizontal way but it stops short of columnizing the field data.

Thank you Mike, I'll make a new test.
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Hi Mike, generally a better result, but for some reason no Templated Source > Reference field was populated and very oddly the last 6 Source records, the Publication Info is incorrect.

See attached
Attachments
71.png
71.png (279.48 KiB) Viewed 2101 times
70.png
70.png (339.82 KiB) Viewed 2101 times
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Mike, could be pure coincidence but those last 6 Source have a much higher REC ID value.
User avatar
tatewise
Megastar
Posts: 28506
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by tatewise »

I don't think the Record Id have anything to do with it. You presumably added those much later than the others.

I can only assume that the text RG11 occurs earlier in the #ORA Note than the Source.Class line in those cases.
So I have made the plugin only search from the Source.Class text onwards.

Sorry, about not setting the Reference value. It was due to the Reference field being deleted completely and not just deleting its value, which leaves an empty field behind.

I've improved the way it includes the #ORA Note text in the Result Set.

Try the attached Copy Census Source Reference to Publication Info plugin Version 0.3 Date 12 Jan 2024.
Last edited by tatewise on 16 Jan 2024 21:45, edited 1 time in total.
Reason: Attachment deleted as a better version is attachd later.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Thank you very much Mike. I'll retest this when I get home this evening.
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Hi Mike
Tested thoroughly and all working okay. Super work and thank you so much, this saved me a lot of time correcting data fields manually.

Can I please ask for one more adjustment: If the Plug-In is solely for my benefit can you please expand the list of selectable Census Years to be 1841 - 1921, this is to allow me to keep consistency in my database for all 'currently' available Census years.

Thank you - very much appreciate for you help in this matter.
User avatar
tatewise
Megastar
Posts: 28506
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by tatewise »

To cater for all those census years I need to know the formatting involved for each year.
i.e. The ORA format and the required Reference / Publication Info format.

I know many years will be the same (1861 to 1901) but 1841, 1951, 1911 & 1921 are different.

The formatting for 1841 and 1911 must be FTAnalyzer compatible.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Good Morning Mike

The 1841 UK Census #ORA data looks like the attached image. So uses the HO107 designator rather than RG##

Followed by:
Folio
Page
Line

Then the 1911 is simplified to:
Class
Piece
Schedule Number

Where is Class is RG14

These are the only England and Wales Census records currently supported by Lost Cousins, I'll take a look at the remaining England and Wales Census data later today if that's okay. I have things to do this morning.

Thanks
Attachments
80.png
80.png (254.46 KiB) Viewed 1991 times
81.png
81.png (221.51 KiB) Viewed 1991 times
User avatar
tatewise
Megastar
Posts: 28506
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by tatewise »

IMO you need to investigate a bit further.

I think 1841 also requires a Piece and Book number.
In addition, the plugin needs the FTAnalyzer reference format for Piece, Book, Folio, Page, Line, etc.

I think 1911 may need more than Piece and Schedule numbers.
Likewise, the plugin needs the FTAnalyzer reference format for Piece, Schedule, etc.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

I'll make some further testing to be sure FTA accepts the format for 1841 and 1911 Census. Or work out what it should be.
avatar
jelv
Megastar
Posts: 649
Joined: 03 Feb 2020 22:57
Family Historian: V7
Location: Mere, Wiltshire

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by jelv »

tatewise wrote: 13 Jan 2024 11:32 I think 1911 may need more than Piece and Schedule numbers.
LostCousins only uses piece and schedule number for the 1911 census.
John Elvin
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Thank you John. Any chance you could post a screen shot of the format used.
avatar
Ruth
Gold
Posts: 23
Joined: 27 Jun 2018 07:57
Family Historian: V7

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by Ruth »

Mike, there is no line number used in the reference for 1841 census. Lost cousins requires Piece, book, folio, page.
eg HO 107 Piece 749, Book 11, Folio 7, Page 9
avatar
David Potter
Megastar
Posts: 1010
Joined: 22 Jun 2016 15:54
Family Historian: V7
Location: United Kingdom

Re: Trying to import Census Reference info for use in FTAnalyzer

Post by David Potter »

Hi Mike

I can confirm the following format for the 1841 Census works okay with Lost Cousins. See Attached
I'll take a look at the 1911 Census format needs tomorrow.

Have a good evening.
Attachments
90.png
90.png (110.31 KiB) Viewed 1782 times
Post Reply