* Issues with Clean Up Notes

Writing and using plugins for Version 5 and above.
User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Issues with Clean Up Notes

Post by KennethEvans » 25 Nov 2019 19:31

I may have misunderstood what this plug-in does. According to "Getting the Most from Family Historian", a Local Note is
A note that will be stored in the current record

So what does it mean:
This plug in converts record notes which have only one link to "local notes"
In addition, it added blank lines to notes in some cases.

It did not give any indication of what it did if, anything. On comparing the Gedcom files, it appears it only changed things like
1 NOTE Has the same initials as her mother.
1 CHAN
2 DATE 25 NOV 2019
3 TIME 12:41:35
to
NOTE @N97@
both under an individual record (0 @Innn@ INDI).

So is this what it is supposed to do, and, if so, what is the advantage of that?

Additionally, instead of removing blank lines (I didn't have any cases of these), it added a single blank line to several records, which seem to have sources. For example:
2 SOUR @S4@
3 NOTE Albert Welles, President of the American College of Heraldry and Genealogical Registry of New York, History of the Welles Family in England and Normandy (Albert Welles, 67 University Place, New York, 1876).
to
2 SOUR @S4@
3 NOTE
4 CONT
4 CONT Albert Welles, President of the American College of Heraldry and Genealogical Registry of New York, History of the Welles Family in England and Normandy (Albert Welles, 67 University Place, New York, 1876).

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 25 Nov 2019 20:05

It is important to understand the difference between Local Notes and linked/shared Note Records.

Hopefully you are aware of the many types of Record in FH/GEDCOM.
i.e. Individual, Family, Source, Media, Place, etc.
They link to each other using references such as INDI @I2@ and SOUR @S4@

Note Records are another type of stand-alone Record linked to using say NOTE @N97@
Thus they can be shared by several other records in the same way Source Records can be shared.

On the other hand Local Notes are subsidiary fields held within Records attached to the record , or to facts, or to citations, etc.

Many products use Note Records to the exclusion of Local Notes, especially when exporting to GEDCOM.
But such Note Records will often be referenced from only one record.
So it is more effective to migrate such Note Records to the Local Note field they are referenced from.
It is also common for such Note Records to start with a blank line.

The Plugin automates that process of migrating Note Records and optionally removing leading blank lines.

In your examples I would expect NOTE @N97@ to become local NOTE Has the same initials as her mother.
The record 0 @N97@ NOTE Has the same initials as her mother. to be deleted.

I cannot explain why your example 3 NOTE gained 4 CONT blank lines, unless there was a 3 NOTE @N99@ link that has been replaced at that point, and you chose not to remove blank lines.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 28 Nov 2019 18:18

Mike,

I know the difference between @xxx@ references and things that are added in the same place at the next level in the Gedcom file. I didn't know the definitions of the names for these, which seem to be Note Records and Local Notes, respectively. Thanks. That is different than what I saw defined for these terms in the help. But it explains what happened.

As the to fact that the plugin added a blank line: I think that is a bug. It wasn't there before, and shouldn't be there. It seems to happen with SOUR record notes. I just did an Undo, and probably won't use this plugin again. There were no Note Records before, and I probably don't need them. Not having them makes the Gedcom file easier to read. And adding the extras lines is a problem.

BTW I didn't get an email about your response, or I would have responded then. Apparently I have to manually check that I want a response? Normally you are automatically subscribed when you make the original post. Actually, under the Tools menu, it says I am subscribed.

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 28 Nov 2019 20:18

Clicking the Subscribe/Unsubscribe topic option seems to produce messages contrary to the option.
So ticked (which is the default) means no messages despite saying Subscribe topic.
Unticked (which needs setting) means send messages despite saying Unsubscribe topic.
I will report that to the administrator Jane.

Ken could you please post larger snippets of GEDCOM to illustrate the problems so that surrounding lines are shown.
Then the Plugin can be fixed.
What I cannot understand is your first example claims NOTE @N97@ was added, whereas links like that may get removed but never added.
If your GEDCOM had no Note Records beforehand, then the Plugin should do absolutely nothing.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 28 Nov 2019 21:08

Mike,

My first example just showed that the Local Note got replaced with a reference (Record Note), what I now know is supposed to happen. There is no issue. There were lots of these.

I did have Notes beforehand, obviously.

These are more lines around one of the problem cases.

Before
0 @I387@ INDI
1 NAME Lucy /Wells/
2 SOUR @S4@
3 NOTE Albert Welles, President of the American College of Heraldry and Genealogical Registry of New York, History of the Welles Family in England and Normandy (Albert Welles, 67 University Place, New York, 1876).
1 SEX F
1 BIRT
2 DATE 15 JUL 1822
2 PLAC Hanover, OH
1 FAMC @F164@
1 FAMS @F168@
1 NOTE Married by Rev. George Wells.
1 CHAN
2 DATE 25 NOV 2019
3 TIME 12:41:35
After
0 @I387@ INDI
1 NAME Lucy /Wells/
2 SOUR @S4@
3 NOTE
4 CONT
4 CONT Albert Welles, President of the American College of Heraldry and Genealogical Registry of New York, History of the Welles Family in England and Normandy (Albert Welles, 67 University Place, New York, 1876).
1 SEX F
1 BIRT
2 DATE 15 JUL 1822
2 PLAC Hanover, OH
1 FAMC @F164@
1 FAMS @F168@
1 NOTE @N58@
This is the whole part for that INDI. Notice the CONT's.

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 28 Nov 2019 21:25

Sorry Ken, but I believe you have got it the wrong way round.

A Local Note is never meant to be replaced by a link to a Note Record - that is not supposed to happen.
Quite the reverse. A link to a Note Record that only has that one link, should have its text moved to a Local Note and the Note Record deleted.

You example GEDCOM record seems to have got the before and after swapped around.
The Plugin will remove the blank 3 NOTE and 4 CONT lines from the beginning of a Local Note and not add them.
Also the link to 1 NOTE @N58@ should be replaced by 1 NOTE Married by Rev. George Wells. not the opposite.

Unfortunately, you did not include the change 2 DATE 25 NOV 2019 and 3 TIME 12:41:35 in the After example.
That would confirm which changed occurred first.

Can you please revisit your GEDCOM and double-check what you are observing.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 29 Nov 2019 13:11

Regarding your Subscribe/Unsubscribe topic comments, please see Subscribe/Unsubscribe topic (17257) that has resulted in the ticked icon being removed to make the status clearer.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 1450
Joined: 28 Dec 2005 22:02
Family Historian: V6.2
Location: Cirencester, Gloucestershire
Contact:

Re: Issues with Clean Up Notes

Post by ColeValleyGirl » 29 Nov 2019 13:25

Mike, that's only true for the standard theme (MetroLike)

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 29 Nov 2019 13:36

Sorry Helen, but it is true for all except Marina Orange i.e. is true for Metrolike, Metrolike Blue & prosilver.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
ColeValleyGirl
Megastar
Posts: 1450
Joined: 28 Dec 2005 22:02
Family Historian: V6.2
Location: Cirencester, Gloucestershire
Contact:

Re: Issues with Clean Up Notes

Post by ColeValleyGirl » 29 Nov 2019 13:52

I'm pretty sure Jane didn't change it on prosilver (as she's always said she won't customise that theme), so it must always have been that way. So it's always been inconsistent (but the two major themes and one colour variant were at least consistent), and now they're not.

User avatar
Jane
Site Admin
Posts: 7785
Joined: 01 Nov 2002 15:00
Family Historian: V6.2
Location: Somerset, England
Contact:

Re: Issues with Clean Up Notes

Post by Jane » 29 Nov 2019 15:10

ColeValleyGirl wrote:
29 Nov 2019 13:52
I'm pretty sure Jane didn't change it on prosilver (as she's always said she won't customise that theme), so it must always have been that way. So it's always been inconsistent (but the two major themes and one colour variant were at least consistent), and now they're not.
Most of the themes inherit from ProSilver and those Icons came direct from ProSilver, so I changed it there as I did not have time to try and work out how to over-ride them in the metrolike themes.
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 29 Nov 2019 15:42

Mike,

You are right. I have it backwards, and the before and after are switched.

I apparently got it wrong the first time I did a compare, and have been operating in the context of that misinformation. If I had got it right, there would have been no need for this topic.

So the Cleanup Notes plugin did find one line that had a blank and fixed it. I thought I had cleaned all those out (the hard way). I apparently missed that one. They came from the import from Family Tree Maker. There apparently was no CHAN. (This record was added probably in the late 90's.)

So it looks like Cleanup Notes replaces note references with local notes. That seems like a good thing to do.

Guess I'll run it again.

I apologize. This was entirely my mistake. :oops:

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 29 Nov 2019 16:11

That is OK Ken, and explains the lack of 1 CHAN as FTM does not export them.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 29 Nov 2019 17:46

So I have now run Clean Up Notes and have no Note Records.

But I still have some cleanup to do. I have blank lines in some Notes (but not at the beginning). In particular, the Notes for a Source all (except one) end with a CONT (and the other NOTE's don't). In looking at the Source window, this seems to mean there is a CR at the end so the cursor is on the next line. In the Records Window, there is a square box at the end if I click it.

I don't know if this is good or bad. I am not sure where the note for the source is used. I did a Source Summary Report, and the one that does not have a CR looks the same as the others (meaning it must be ignoring the blank line at the end).

In any event, I would like to list all the Notes (which are now local) in my file. Again, I don't readily see a way to do that. Is there one?

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 29 Nov 2019 18:38

Yes, what you need is the Search and Replace Plugin to search for every Note field.

In its Major Options tab leave the Search Scope at All Records & Events/Attributes.
In the Basic Filters untick every option except Note fields so only Note fields are included
In the Search Criteria put dot/fullstop & asterisk (.*) in the Search : box and select LUA Pattern Mode to match any text.
Click the Search ONLY button to search for all Note fields.

In the subsequent dialogue click Report to add each Note to the final report.
When happy it is finding what you want, untick Confirm every item found and click Report once more.
Eventually, the Result Set will list every Note found anywhere. There will a lot :)

If you only want to find Note fields matching certain criteria, and possibly change them, let me know.

In this example the Search Box uses an LUA Pattern to recognise any text.
That is similar to a Regular Expression where in this case dot matches any character and asterisk says there can be any number of characters including none.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 29 Nov 2019 21:03

Thanks, that works.

Except the results have ... instead of showing the whole note, and they only have a blank where the CR is, so it doesn't help with my problem.

I assume I could fix it to just find ones with blank lines. I tried using ^$ as the pattern but got no hits. I have to go right now. I can work on it more if I don't hear from you.

And I did peruse the Help this time. 8-)

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 29 Nov 2019 22:34

^$ won't find blank lines but completely empty Notes.
Use two newline characters to find a blank line (whether Plain Text Mode or LUA Pattern Mode is irrelevant).
Use Show White Space to actually see the newline characters as in Word.

Yes, Note text is truncated at about 150 characters with ellipsis.
Which is reasonable, because Note text is unlimited in length and could not possible be shown in full in all cases.
But if you double-click on the text of the Note it will be opened in its Property Box as discussed before.

Are you trying to blank lines in general, or blank lines at the end of a Note?
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 30 Nov 2019 00:07

^$ won't find blank lines but completely empty Notes.
That surprises me. I guess it depends on how you are storing them internally.

I am trying to find empty lines both in and at the end. If I can just get notes with empty lines, I can double-click to fix them. When I get all notes, clicking them to see if they have blank lines is too much work.

\n\n didn't get any hits. I assume that is what is meant by newline. Neither did \r\r or $$. I didn't see anything in the help about newline.

Even using .* I don't see any difference between Show white space or not. They both show as a space. I assume this means in the results window?

I ended up doing this another way using Search and Replace (with the help of a text editor to find the empty CONT). All the notes with blank lines in the middle also had another common string, which I searched for, and corrected using the results window. The ones with blank lines at the end were all source notes. I cleaned them up from the Source tab of the record window. I was left with two having blank lines at the end. I could not see there was a CR in the Properties | Notes tab, but did see the box at the end in the All tab.

Bottom line is that it is fixed without searching for blank lines using Search and Replace. I still don't know why that didn't work.

And your help was invaluable.

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 30 Nov 2019 00:40

By newline I mean literally that, i.e. press the Return key or the Enter key, just like you would to start a newline in Word.
White space characters are space, tab & newline and shown as . » ¶ when Show White Space is selected.

We needed the LUA Pattern of .* to find all Notes because we did not know what characters to match, so had to use a wildcard pattern to match everything.

To find blank lines we know exactly what characters to match, i.e. literally two newline characters, so that is what we enter.
For many Search and Replace cases the Search and the Replace boxes hold literal plain text characters with Plain Text Mode selected. It is only the more complex/wildcard cases that need LUA Pattern Mode.

To remove blank lines, use Search & Replace button with two newlines in Search and one newline in Replace because that is literally what you want to do - simples!
(It might have to be repeated if you have multiple adjacent blank lines.)
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 30 Nov 2019 01:23

Ok, I'll try that. I do see the newlines in the All tab are represented by ¶.

I understand the .*. I didn't know what you meant by newline. Usually you enter a string of some sort in a Regex search box, not type a CR. I don't know about Lua.

I also had the question if a blank line at the end of a source note is good or bad. In any event I eliminated them. The are no more empty CONT lines in my file.

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 30 Nov 2019 09:51

The LUA Pattern Mode/RegEx features only apply for special wildcard search & replace cases.
Usually Plain Text Mode would apply, and you enter search & replace text exactly as you would in the FH fields.
if you enter newline or tab into a Note using Return/Enter or Tab keys, then do the same in Search and Replace boxes.

The risk with deleting all CONT lines without text is that it would apply to all fields and not just Note fields, unless you took great care to check they were subsidiary to the NOTE tags, i.e. it would affect Address (ADDR), Text From Source (TEXT), and other long text fields.

There is no great harm in blank lines at the end of Note fields, or any other long text fields, but it is good to be consistent.
The once place you may find it worthwhile to have a blank line is at the start of Text From Source fields.
Otherwise its presentation in Reports under the Sources section is untidy and runs into the label.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 01 Dec 2019 03:14

Hi Mike,

Thanks for the information. I went back and redid my changes using the ¶ pattern and also replacing as well as searching. It was much easier this time. This is what I found:

¶¶ only finds blank lines in the middle of a note.

¶$ finds the ones at the end.

Also I didn't really guess that searching for ¶ worked this way, i.e., you type a CR, not enter some special character.

The Hide / Show White Space option applies to what you enter in the Search and Replace boxes. Before, it didn't seem to do anything (because things I was using like .* don't have white space) , and I didn't know whether it meant here or in the Results window. Now it is clear. The help might be more clear about what it does.

It would have been helpful to show boxes □ for the ¶ in the Results Window (as it does elsewhere (e.g. All tab) and does for some Queries).

I note what you said about TEXT, but I don't have any of those.

Finally, where does it store its current settings and Presets. I was working on a test file and saved a Preset. It isn't there when working on my real file, nor were the last settings. I didn't see anything in AppData, %TEMP%, or in the project folder. (The test file didn't have a project, anyway.)

These are just things I noted. Overall, it is a very powerful tool. Before getting your suggestions, I wasn't even beginning to use it effectively.

Thanks.

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 01 Dec 2019 09:58

Like many Plugins it saves its settings in the Plugin Data folder within the Project (alongside the Media folder).
For standalone GEDCOM files it creates such a folder.

To move Presets around see the Knowledge Base > Search and Replace ~ Manage Presets advice also available via the Plugin Help & Advice window on the Manage Presets button.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

User avatar
KennethEvans
Platinum
Posts: 37
Joined: 18 Nov 2019 15:17
Family Historian: V6.2

Re: Issues with Clean Up Notes

Post by KennethEvans » 01 Dec 2019 17:33

Thanks. I should have found the Plugin Data folder, but I apparently missed it.

I also should have read the Help more carefully. I did look at the Manage Preset page. I missed the explanation of the Program List, or more likely, I was not aware plugin settings were project specific, so it might not have registered. At the time I was just trying to figure out how to save it. It wasn't intuitive, and it took me several trial-and-errors to get it on the LHS (with reading the Help).

I also missed the folder it created for my test file. It happened to be sorted to the bottom and off screen in the folder where I had my test.gedcom, even though the folder was sorted by date modified, and test.ged was at the top. So I won't feel guilty about that one. I wasn't expecting it in the first place. I will next time.

So now that I do understand it (at least I think so), it doesn't seem to work. I opened the test file and moved the preset I had created to Program List. I then opened my regular project and it wasn't there. I tried several times and made sure I clicked Finish rather than the X in the top right. It also isn't in the Program Data/.../Plugin Data file.

User avatar
tatewise
Megastar
Posts: 16884
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Issues with Clean Up Notes

Post by tatewise » 01 Dec 2019 20:45

I presume this is the sequence you followed:
  1. With test.ged in Manage Presets used >> Save >> to Program List.
  2. Your preset would then be listed in both Project List and Program List.
  3. Click Finished and Close Plugin and Close FH.
  4. Open regular project and Plugin and open Manage Presets where your preset should be in Program List.
  5. Select preset in Program List and use << Load << to Project List.
  6. Click Finished and that preset should be available in Preset : drop list.
At which step did it go wrong for you?

The Program List is stored in C:\ProgramData\Calico Pie\Family Historian\Plugin Data\Search and Replace.dat.

You could also try the Export and Import alternative.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

Post Reply