* Rearrange address and place 1.7 errors

[johnhanson]

I am trying to help someone sort out the places in their large database
66,000+ people
20,000+ families
49,000+ places (address not used)

Running the plugin produces the list of places and presents the mapping screen
I am simply trying to blank place1 on selected records
Change to the input filter is all blue
Click on any entry to deselect everyone before finding the ones I want and FH crashes

[johnhanson]

As additional information
The database has come from TMG via a Gedcom import - I am well aware of the direct import but because of the mess in TMG it was found that Gedcom gave a better import of the data

The person whose database this is put the BMD reference (Volume/page) into first part of the place name
Hence there are nearly 300 entries for Leeds in Yorkshire when one would do

Whilst I can identify them easily in the places tabs of the records window merging them one at a time is not really an option
Using the "Work with data" places is the better option for merging multiple entries find all the entries is a pain with a list this large as it uses find (which only works in the start of the place name rather then using the filter as the places tab does

[tatewise]

Sorry about that John.
It looks like it is something to do with the large number of Place records, because I get the same effect with a Project that has 26,000+ Place records. Smaller Projects with 3,000 Place records work fine.
It should be possible to select the Place records in FH before running the plugin so the Input Filter tab is suitably preset.
That seems to work OK but then the Mapping Options tab does not behave itself and also crashes FH.
I will have to investigate why large numbers of Place records are having that effect.

Even if the plugin worked it does not solve your problem because it never performs any merging, so you would still need to use another method to actually merge the Place records.

However, I think Tools > Work with Data > Places... should offer a solution.
I am assuming that in say the Leeds, Yorkshire case the Part 2 column holds Leeds and the Part 3 column holds Yorkshire.
So click on the Part 2 column header to sort that column and all the Leeds records should cluster together.
That allows them to be easily selected using the Shft key method and the Merge... button used.
After the merge, it is easy to edit the resulting single Place record to remove the Part 1 details.

[johnhanson]

Mike
Thanks for the response and not a surprise I suppose as I thought that was the issue
I hadn't realised that you could feed the plugin that way
Doing it that way generates another issue - which I have seen the solution to anyway - off adding the {xxx} to the end of each
So it looks like the working with places is going to be the best option
Was just trying to find a way to make it all easier!

As an aside what is the easiest way to remove the space that appears at the start of some place parts
As you can see from the image - there are about 300 about the highlighted Yeovil entry

PlaceParts1.jpg (174.5 KiB) Viewed 2697 times

I just wish that FH did some basic editing to follow the basic convention of a single space after a comma

[tatewise]

You are asking about tidying Place comma-separated parts regarding leading space characters.

The usual convention is for the 1st part to have no leading space but to follow every comma with a single space.
e.g.
Town, County, State, Country
That format can be obtained by running the Rearrange Address and Place Parts plugin even without specifying any changes
(after I have fixed the current problem).

But you can choose whatever formatting you prefer.

That can be achieved by using the Search and Replace plugin.
Set the Search Scope to Place Records (_PLAC) and in the Extra Filters tab select only Record Names/Titles.
Enable the LUA Pattern Mode and Show White Space options.
e.g. To remove space characters after each comma search for a comma plus multiple spaces and replace with a comma:
Search: ,·+
Replace: ,
e.g. To remove leading spaces from 1st part use:
Search: ^·+
Replace: with nothing
Note that · represents a space character.

[johnhanson]

Mike
Thanks again for the great help
Looking forward to the "fix"

Have got the end of the letter B and already reduced the number of places to 44k!!

Keep up the great work

[victor]

When it comes to place names use the correct information.
Leeds for example is in West Yorkshire not just Yorkshire. There is Hessle in East Yorkshire as well as West Yorkshire. There is also Scalby in North Yorkshire as well as East Yorkshire.
Never miond about Yorkshire. Whatever the place name anywhere in the world spell it out according to how the detail is spelt in whater country. Just make them all the same setting otherwise you will have quite a few. These can always be corrected using data. place names. No matter how you spell it just make sure the same place always has the correct detail.
With boundary changes make sure you have the correct county. For example an event may have happened in say 1920 in one county. But after the boundary changes that place name may have moved to another county. As an example before the bounday changes in I think 1974 many places were listed in WEst Yorkshire. Some of these are now in South Yorkshire which didn't exist before the changes. The question then would you chose say a 1920 event as happening in West or South Yorkshire?
Victor

[johnhanson]

Victor
I have been researching for over 40 years, have taught family history and have been lecturing on it for over 20 years and regularly talk about what you mention.

Whilst I take your point it has nothing to do with the original question that I raised
I am trying to sort out someone else's FH file

The question was about blanking part 1 of the place, in this case the address, nothing to do with counties and the like

[johnhanson]

Mike
I am beginning to tear my hair out as I am obviously doing something wrong but can't for the life on me see what

I have managed to reduce the number of place names from 49,000 plus to just over 21,000 by blanking the first parts using merging and clear as you suggested but there are still some that have a leading space and don't show in the listing

I tried your suggestion to replace the leading space the way that you suggest but it wants to remove the space before all the comma's!

I have taken screen shots of the Search/Replace plugin screens and the result. all attached

SR1.jpg (145.46 KiB) Viewed 2510 times

SR2.jpg (203.8 KiB) Viewed 2510 times

SR4.jpg (76 KiB) Viewed 2510 times

The last shows that the replace would remove the spaces from the beginning of all the parts

What am I doing wrong!
As a point of interest and in case it is relevant if I search the place tab for 10c then the entries including the Ince one all how up but if I search for the name Ince then the entry does not how

I would appear as well that there are entries in the database that have no trailing space after a comma as in Ince,Lancashire - How can I replace whose

Happy to let you have a copy of the gedcom if you wish - all 68m of it!

[tatewise]

Do NOT tick Fact Place fields as that will cause major problems.
Otherwise, the settings are OK except for the Search and Replace patterns (see later).
The patterns you are using are intended to remove spaces after commas as per the example I gave earlier.

You have selected 7 Place (_PLAC) records. Are they the only ones that have a leading space?
If so, it might be easier to just edit those 7 by hand.

As I said earlier, to remove leading spaces from 1st part use:
Search: ^·+
Replace: with nothing

The ^ locks the pattern to the start of the Place Title and checks for one or more spaces.

To insert a missing space after any comma then use:
Search: ,([^·])
Replace: ,·%1

I think you are talking about Find in the Tools > Work with Data > Places dialogue.
Yes, that only searches at the start of each Place name.
If instead you use the Records Window Places tab then its Filter finds the text anywhere in the Place name.

[johnhanson]

Mike
Thanks - just got in from being at daughters and had a brain wave

Had been having issues trying to get the removal of what appeared to be double spaces working and getting nowhere

So I copied from of them and pasted it into Word"

5h. 465,  Lewes, Sussex, England

image.png (6.93 KiB) Viewed 2448 times

As you can see what appears to be the second space isn't - exactly what it is I am not certain so I have decided that the quickest way to remove them.

Thanks for all your help

[johnhanson]

Helps to re-read before you post late at night after a busy day!

Decided that the easiest way to resolve those mystery ones is to do them manually - thankfully only about 150 or so that I have found so far!

[jelv]

Is that a non break space?

[tatewise]

Try the attached prototype Rearrange Address and Place Parts plugin Version V1.7.1 Date 30 Jul 2023.

There are several challenges with handling such large numbers of Place records.

One issue is that some button-driven functions take a significant amount of time.
But the plugin was not preventing further buttons from being used while the previous function was still running.
It does now inhibit the user interface and turns it pale grey while a lengthy function is in progress.

Some lengthy functions were not interacting with the OS correctly so produced the Not responding error effect.
They also took a long time to update the Filter List user interface. Both those issues are now fixed.

However, there is one feature that I could not fix. The ability to select items from the Filter List has a timing problem when there are a great many items listed. So that feature is now disabled when there are more than 9,999 items.
To select items you must select Place records via the Records Window before running the plugin.

[johnhanson]

Mike
Sorry for the delay been busy with family
Thanks for the update and will try in out in the next couple of days and let you know

[johnhanson]

Mike
Have been using the new version of the plugin with the much reduced list of place list with great success and the ability to pre-select the places to update before calling the plugin is a great help.

Mind at times creating a query to produce the list can be fun!

It is often a case of - is it quicker to update the ones that need changing manually or use the plugin as a group

All appears to be working fine

Finally got round to looking at the new version of the plugin against the original database with its bloated place list and again it appears to work well - the only problem was that the times appears to freeze from time to time while processing the file but will restart again when on the next section

Thanks for all the help and advice as always

[tatewise]

Thank you for that feedback John. Coping with very large Projects in plugins is often a challenge.
The temporary 'freezing' is a likely side-effect of handling such large numbers of Place items.

[Linda Reinfeld]

I have been working with the Rearrange address and place plugin - version 1.7
Should I update to version 1.7.1? I don't see it in the plugin store

I have 56,000+ people and 19,000+ place records, so not quite as many as John Hanson.
It has taken me many iterations of the plug-in to find all the errors - for example, 2 places, one with a capital letter & one without (Massachusetts, massachusetts). I thought you might like to know that the plugin on subsequent running would find new errors - errors that hadn't been identified previously.

The plugin would cause FH to shut down a few times, but it wasn't a problem - I just started it again.

Linda Reinfeld

[tatewise]

Linda, that V1.7.1 prototype is not in the Plugin Store yet.
I was waiting to see if John found any further issues that might need investigating before publishing.

The plugin adds no new functionality except for trying to handle very large numbers of Places without crashing.
You are welcome to try the attachment from a few postings ago.

I am surprised you find the Rearrange Address and Place Parts plugin useful in identifying the kinds of errors you mention.
What features of the plugin were you using?

May I suggest you try the Place Exceptions plugin from the Plugin Store which is more focused on finding such variants.

[Linda Reinfeld]

Mike -

The reason I was using rearrange address and place parts is because I was trying to eliminate the first part of my places as that is always blank. When I went to run the Plugin I got many errors 'Failed to change Place (TEXT) name'. I used that output to fix my data.

I can see that Place Exceptions works better. However I have a question - Is the only way to fix the data to go to tools/work with data/places and merge the places? It is inconvenient to constantly shift from the Place Exception query to the Place List - unfortunately you have to close the place list to go back to the query/report. But I will certainly use the Place Exceptions plug-in.

Thank you for your idea/help

Linda Reinfeld

[tatewise]

I will investigate the 'Failed to change Place (TEXT) name' error when I find time.

Instead of using Tools > Work with Data > Places to merge the Places, select the two records in the plugin Result Set and use the Edit > Merge/Compare Records... command.
In the Select Records To Be Merged dialogue, you may have to use the small up/down arrows top right to move the 'preferred' Place to be first in much the same way as you Select Final Version in the Place List dialogue.

[Linda Reinfeld]

Thanks, Mike. It's much easier to fix/merge with the 'Edit'. Somehow that's a button I don't use very much....

Linda

[tatewise]

Linda, I've had a quick look at the 'Failed to change Place (TEXT) name' error.
I ran the plugin to eliminate the first part of Place records that are blank.
However, I cannot see how the error would have arisen.

Please provide more details of what settings you used that caused the error.
A screenshot of the plugin Mapping Options tab would be useful.

A screenshot of the error message would also be useful.

Did you run the plugin on all entries or did you use the Input Filter feature to select a subset?

[Linda Reinfeld]

Hi Mike -

My data:
Oil City and oil City
Meadville and meadville

What my place list looked like before the plugin:

Screen_Oil_City.png (15.81 KiB) Viewed 1559 times

Place_List.png (21.83 KiB) Viewed 1559 times

Mapping options: only left justify all place parts to Place 1.
I ran the plugin on all my data - no filters.

Plugin_Settings.png (47.51 KiB) Viewed 1559 times

The 2 fail errors:


BTW I am running the 1.7 version of the plugin - not your latest version

Hope this helps

Linda

[Linda Reinfeld]

Mike -

I couldn't attach the fail message to the previous reply. I'm trying again

Fail_Oil_City.png (24.55 KiB) Viewed 1559 times

Fail_Meadville.png (28.29 KiB) Viewed 1559 times

That worked. (I had addd the 3 files by clicking attachments, then wrote some info and then tried clicking attachments again, and nothing attached.)

If you look at the fail message the data looks fine. But if you look at it in the place list you realize the capital letter/small letter problem.

Linda