* Where Used Record Links v List All Citations for All Sources

[BillH]

Mike,

At one time I believe I got some plugins from you posted in the beta forum. I think one of them was List All Citations for
All Sources. Was this converted for FH 7? If so, do you have a copy I could get? If not, would you be willing to convert it for FH 7?

Thanks,
Bill

[tatewise]

The List All Citations for ... family of Plugins are derived from Jane's List All Citations for a Selected Source that is in the Plugin Store and works in FH V5 V6 V7.
I have not updated any of my derivatives for FH V7 so I don't think they could have been posted in the beta test forum.

Does Where Used Record Links provide a suitable substitute?
That is in the Plugin Store and works in FH V5 V6 V7 and also has a much improved Result Set including the new FH V7 templated Citation fields, which are not listed by Jane's version.

[BillH]

Mike,

I tried to run Where Used Record Links after choosing Sources and then selecting all sources. After about two minutes it got to 1% and then said not responding. After 10 minutes it was till at 1% and not responding. I tried to stop the plugin and I got a pop up that said FH is not responding either end or wait. I ended and FH closed and I had to restart it.

If that is any indication of how slow it will be for all sources, then List All Citations for All Sources is much faster and gave me everything I need. I won't be using templated sources/citations so don't need that info.

If possible I would love to get a copy of List All Citations for All Sources compatible with FH 7.

Thanks,
Bill

[tatewise]

Bill, I believe yours is a very large Project that may be having an impact on Plugins that need in-depth searches.
In FH V7 any Rich Text fields can include internal Citations so this Where Used plugin must search far more than in FH V6.
That includes not just Records and Facts as before but local Notes and Text From Source everywhere.

It also seems that the Lua v5.3 interpreter in FH V7 needs help with memory garbage collection that I have had to add to some other Plugins.

List All Citations for All Sources may have similar problems.
You may not need some options but a published plugin needs to cater for all users or have a user interface to choose options that complicates things.

Does the Where Used Records List plugin run OK on a much smaller selection of Source records?
I know it runs OK on one Source record from your other posting but how many before it hangs?

[BillH]

Mike,

I'm not sure if mine is very large or not. There are 20,175 individuals and 6,791 families. I have 3,288 sources.

The plugin runs fine on smaller numbers of source records. What is kind of weird is if I select more than 100 or so source records, it always starts saying "not responding" after about 20 seconds.

What is also weird is that no matter how many source records I select, the progress indicator always gets to 1% and then never changes, the clock always says 00:00:00, and the progress bar never shows any progress.

As for when it starts to hang it is kind of hard to tell, but I ran some tests. If I select up to about 1575 sources, it takes about 2 minutes or less to display the result set. If I go to 1710 sources it takes 9 - 10 minutes. In those extra 135 sources, there are none that have a high number of citations. I gave up after that.

Thanks,
Bill

[tatewise]

Thank you Bill that gives me something to investigate tomorrow.

[BillH]

Mike,

One more thing. The Stop button doesn't seem to work. I waited 2 or 3 minutes and nothing happened.

Let me know if I can do any testing.

Bill

[johnmorrisoniom]

Mike, I had the same situation re the progress bar (Never moved past 1%)and the plugin looked like it was not responding. Left it for about 15 minutes and it did complete correctly.

[RS3100]

I have a much smaller project in comparison, but I also never see the progress bar move past 1%. It sits at that point for a few seconds until the plugin completes, but no issues with the plugin otherwise. I use it fairly frequently.

[tatewise]

I am working on a solution to these problems and have some radical new techniques that should help.
The lack of progress bar movement was a silly mathematical coding error but handling large Projects is more of a challenge.
Bear with me.

[BillH]

Mike,

No hurry. We are just thankful that you are working on it!

Also, I was just wondering. Since the Subject I used for this topic was originally about a different plugin and the topic has changed focus, maybe you could change the Subject to more accurately reflect what it is about.

Thanks,
Bill

[tatewise]

Well that inspection of the Plugin revealed more than I expected
The progress bar percentage & time as well as the Stop button should now be fixed.
The 'Not Responding' problem seems to be a memory usage issue cured by a memory garbage collection function.
It still occasionally may say 'Not Responding' but if left alone soon seems to continue OK.
I've also devised a far more efficient & faster search algorithm to detect used record links.
The process for detecting duplicate Source Citations was not perfect and is now fixed.

Please try the attached Where Used Record Links plugin Version 1.6.1 Date 29 Mar 2021.
[ EDIT: Attachment deleted as V1.7 is now in the Plugin Store. ]

[BillH]

Mike,

I ran a bunch of tests tonight. I used both the old and new versions of the plugin. I grouped my tests by alphabetic groups of sources... one group is all those starting with A, one is all those starting with A thru C, etc. That explains the weird numbers of sources in my groups.

First the good news. In my testing of the new version I never saw "not responding", the stop button worked, the percentage complete changed, and the time changed.

Now the bad news. The new version is much much slower. Here are the results of my testing.

Old version---

# of sources
Time to complete

90
4 seconds

181
41 seconds

345
51 seconds

457
54 seconds

735
48 seconds

872
47 seconds

1575
1 minute 11 seconds

1710
8 minutes 21 seconds


New version---

# of sources
Time to complete

90
1 minute 34 seconds

181
4 minutes 47 seconds

345
5 minutes 16 seconds

457
6 minutes 53 seconds

735
7 minutes 36 seconds

872
10 minutes 40 seconds

1575
15 minutes 39 seconds

1710
16 minutes 42 seconds

Note the odd timings for the old version where adding more sources resulted in lower times. Also note that I saw the same large increase with the old version going from 1575 to 1710 sources that I did yesterday.

Bill

[tatewise]

How did you get the old published version to run? I thought it always stalled with 'Not Responding'.

Anyway, a few more experiments reveal that the occasional 10ms pause I added, which often prevents 'Not Responding', is actually taking much longer, whereas adding some extra memory garbage collection is more effective. So now the full benefit of the faster search algorithm is fully exploited and my test runs are much much shorter.

So try the attached Where Used Record Links plugin Version 1.6.2 Date 30 Mar 2021.
[ EDIT: Attachment deleted as V1.7 is now in the Plugin Store. ]

[BillH]

Mike,

See my 3rd post in this thread. Even though it was saying "not responding" it was apparently running in the background anyway. I was able to get as far as my 1710 sources test and that took so long I thought it was not going to ever end so I cancelled it. Tests with less sources finished pretty quickly.

I will test the new version later today when I have some free time.

Bill

[BillH]

Mike,

I had a quick chance to run a couple tests with the new version. First I tried 90 sources and that finished in 4 seconds. Then I tried 181 sources and that finished in 6 seconds. Those were so quick I bit the bullet and tried all 3288 sources and that finished in 46 seconds.

It is now blazingly fast.

Also this will work just fine instead of List All Citations for All Sources.

Thanks!
Bill

[tatewise]

Phew! Glad that worked OK. It sounds like your PC is a bit more powerful than mine.
I'll wait to see if the other two guys who posted give it a try before I publish.

[BillH]

Mike,

I have a 2 year old Dell XPS8930 desktop with an Intel i7-8700 CPU @ 3.20GHz with 64 GB of RAM.

Thanks again,
Bill

[NickWalker]

Just 64 gig of Ram? How on earth do you cope?

[ColeValleyGirl]

Nick, It's a struggle but I manage it as well.

[RS3100]

Phew! Glad that worked OK. It sounds like your PC is a bit more powerful than mine.
I'll wait to see if the other two guys who posted give it a try before I publish.

Working fine for me Mike, including the progress display, though I only have just over 1100 sources and 1250 media records to run it against.

[BillH]

Just 64 gig of Ram? How on earth do you cope?

That is what it came with and I love it. Nice and fast and no out of memory problems.

Bill

[tatewise]

@RS3100: Thank you for the feedback, but note it is the total number of entries in all the records in the Project that must be searched by the Plugin to find the where used links that is probably more important than the number of records whose links are being discovered.

[RS3100]

@RS3100: Thank you for the feedback, but note it is the total number of entries in all the records in the Project that must be searched by the Plugin to find the where used links that is probably more important than the number of records whose links are being discovered.

OK, I don't know how to calculate that. But the highest count in the result set from running the plugin is 5233 records when I run it against all my source records, and it completes in less then 3 seconds.

[tatewise]

To obtain the total number of records for each type of record use the File > File Statistics... command.
Getting the total number of items in all the records would need a plugin to count them.