* importing gedcom + media from MyHeritage

[OlivierM]

First of all, I thank you again for your fantastic support

- the files are not visible anymore...
It must be more than just the upper case JPG which should work OK.
There must be some space or other characters involved.
One option is to revert your Project to its pre-plugin state and I will fix the plugin.
Alternatively, you can use the Search and Replace plugin but it must search for more than just JPG and must focus on just the Media Format field otherwise other fields may get corrupted.
Do you need advice on how to set the plugin options?

I verified, but it is really the switch between JPG and jpg that makes the difference.
I would of course appreciate if you modify the plugin, but I feel it would be a good thing for me to better understand the way the Search and Replace plugin works, so I would love your advice on how to set the plugin options in order to replace the "JPG" format by "jpg" without corrupting the data.

- I notice now I have many duplicate title names...
The plugin will detect the problem and will not rename files with duplicate filenames.
It will say "Media file rename failed" and leave the Media record and filename unaltered.
So you need to ensure all Media Titles are unique before running the plugin.
Please run the Check for Possible Duplicated Media plugin attached to viewtopic.php?p=113053#p113053 to check if any of your URL derived files are actually duplicates and the Media records can be merged.

I tried to use the Check for Possible Duplicated Media plugin without accepting the merge, and since I get the attached error message, even after closing and reopening the project:

image.png (29.92 KiB) Viewed 1238 times

[tatewise]

The Search and Replace plugin settings needed are as follows:

Set the Search Scope to Multimedia Records (OBJE) to focus on just Media records.

On the Extra Filters tab ensure only the Multimedia Format & Place Hierarchy fields filter is ticked.

On the Major Options tab select LUA Pattern Mode because I think there are rogue characters in the Format.

Set Search: to .*JPG.* to match any leading or trailing rogue characters as well as JPG.
Set Replace: to jpg to set the lowercase Format.

Click the Search & Replace button and click Replace to confirm each replacement.
If the first few replacements look OK then untick Confirm every item found and click Replace to convert the rest.


Sorry about the plugin error message. It may be caused by an old version of the table data file.
Anyway, try the updated attachment.

[OlivierM]

The Search and Replace plugin settings needed are as follows:

Set the Search Scope to Multimedia Records (OBJE) to focus on just Media records.

On the Extra Filters tab ensure only the Multimedia Format & Place Hierarchy fields filter is ticked.

On the Major Options tab select LUA Pattern Mode because I think there are rogue characters in the Format.

Set Search: to .*JPG.* to match any leading or trailing rogue characters as well as JPG.
Set Replace: to jpg to set the lowercase Format.

Thank you !

It worked as a charm, simply by searching JPG in plain text and with the other settings you recommended.

[OlivierM]

A last question:

I still have about 400 media files where the TITL tag = equal to the file name and has a length of 131 characters like this:

AxZjNhMWUzZDYxNzJhNjMwZjgyMmQ0N2QzNDUxZTNjYzU5OTc5OWY0N2JmZTlkODgxZTYmZT0xNjk3OTU4MDAw_024_768_4651_500685_03430058e1220y661p6920_A

Is there a way to rename in bulk all these files by cutting the first 100 characters and the 18 last characters?

Thank you in advance.

[Valkrider]

Others may come up with an alternative way. Open a copy of your Gedcom file in Notepad++ and use its search and replace function to look for the title and delete the characters at the beginning and then repeat with the characters at the end. Open that gedcom in FH and make sure everything is ok. If it is then repeat on your 'live' gedcom.

[tatewise]

I think it could be done with the Search and Replace plugin and rename the files as well as the Title.
However, the Media would still have meaningless numerical Titles and filenames.

I would advise taking your time and manually give the Media records meaningful Titles.
Yes, it will take a while but only needs to be done once.

Another possible solution relies on those Media being linked to Source records.
In that case, consider the Rename Selected Source Media plugin from the Plugin Store.
That uses the Source record Title as the basis for the Media record Title and filename.

[OlivierM]

I would advise taking your time and manually give the Media records meaningful Titles.
Yes, it will take a while but only needs to be done once.

You are right (as usual) But before giving the media meaningful titles, I want to get rid of these absurd long file names.

[tatewise]

After you have given them new Titles you can run the Rename Selected Media Files plugin as before to use the Titles as the filenames!

[OlivierM]

I think it could be done with the Search and Replace plugin and rename the files as well as the Title.
However, the Media would still have meaningless numerical Titles and filenames.

I temporarily accept the meaningless dimension of these names. But how do I create them ?
I know that the lua string function I must use on the titles with a long name is string.sub(s,101,113) but I do not know how to apply it in the search and replace plugin.

Could you please help me out of this ?

[tatewise]

You cannot apply Lua functions to the Search and Replace plugin and must use Lua patterns as explained in FHUG KB Understanding Lua Patterns.

So to match any N characters you must use the dot . pattern repeated N times.

To capture the 13 characters that are 18 characters from the end of the text use:
Search: .+(.............)..................$
Replace: %1

In the Search Scope, use Select Records to choose just the Media with those long names.

On the Extra Filters tab, as well as the Multimedia Format & Place Hierarchy fields option, also tick on the right the Record Names/Titles > Multimedia (~FILE.TITL) option so that the File link, filename, and Media Title all get changed together.

[OlivierM]

Thank you Mike !

The TITL field is changed, but the file names are not changed: I think I would have to tick the "multimedia linked file fields and media fields " to achieve this. I tried, but it yields poor results because it also replace the directory path (media/ ...) and the suffix (*.jpg).

So I prefer to change the TITL field with the "search and replace "plugin, and then run the "Rename selected media files" plugin.

Again, thank you for your continuous support

[tatewise]

Ah! You are correct. That pattern match won't work for both the Title and the Filename Link.
I was trying to propose the solution too quickly without enough analysis.

[khunkai]

Hi,

I tried making a new project on my FH7, by creating a gedcom file (and the attached folder) with FTB.
Everything went well.
However, as I checked out various ways to do it, I had to be logged inn on the MyHeritage website in order to transfer all the data properly in to the FH7.
If I was not logged in, it would not matter that I already had generated the Gedcom file and the attached folder.

Have I understood it correct?

[tatewise]

I have just experimented with importing the same GEDCOM exported from FTB either logged in to MyHeritage or logged out of My Heritage. I cannot find any differences.

As always when reporting problems, it is very helpful if you can post some specific examples.
Exactly what data is not transferred properly?
Does the Windows > Log Files... Exception Report provide any useful hints?

For me, the two Log Files... Exception Reports were identical.

[khunkai]

Not sure how to find the log files, since I am not that clever with IT.
But I noticed one very weird issue, none of the names were displayed i the tree after I imported the same Gedcom (stored on my downloads folder on the PC).

Will try to do it again, and take some screenshot.

[khunkai]

This is how it showed up, repeatedly times....

[tatewise]

I think when the New Project process finishes it explains where to find the Log Files.
They are found via the Windows > Log Files... command.

Do the Names get imported correctly when you are signed into MyHeritage?

Can you post a similar screenshot from FTB for the same family please.

Can you also describe the File > Export GEDCOM... process in FTB giving the answers to each step.

[khunkai]

Yes all is perfect when Im logged inn,
FTB doesnt have the same layout, but here is a screenshot...

[tatewise]

I requested the File > Export GEDCOM... process in FTB giving the answers to each step.
Did you tick the option to Omit the names and info of all living people as shown below?
Also, the Omit private data option underneath will have a similar effect for people marked as Private.
That will produce many [unnamed person] records for living or private people, but other people will have Names.
Check the Records Window and the Individuals tab. Do some people have Names?
I don't understand why being logged in to MyHeritage makes any difference as all the data is on the PC.

image.png (95.28 KiB) Viewed 772 times

[khunkai]

Hmmmm, working well now.

Been following the instructions step by step.
https://www.fhug.org.uk/kb/kb-article/i ... historian/
And I thought I did that yesterday as well.

I wonder, and again... I am a beginner in this.... could it be because the MyH website and the FTB was not synced, that I never got "healthy" Gedcoms created with the export Gedcom function in the FTB?

I spent hours and hours only trying to solve this yesterday, and now... poofffff... it works.

I will thank you guys for helping out here and for your patience, you are worth Gold for beginners like myself.

[tatewise]

My FTB and MyHeritage are not synced either so I doubt if it is that.
As I said, all the GEDCOM data is on your PC, so the online MyHertitage website has no part to play.

Try repeating the process but tick all the options on the Eport GEDCOM: Pick Content page I posted earlier.
With those Omit... options ticked does the imported FH Project have many [unnamed person] records?
If so, then that is the explanation. Perhaps you are not being consistent with those tick options.

As the FHUG KB page Importing to Family Historian says under Import from MyHeritage (MYH):
Export GEDCOM from MyHeritage Family Tree Builder
"Take care to choose appropriate options in the export dialogue. Include all people, save notes, sources & photo information, but do not omit living people or private data."

[egn]

Another issue with import of media from MyHeritage is that MH uses specific tags to handle assignment and cut outs from media for the icon images:

Code: Select all

2 FORM jpg
2 TITL <name>
2 FILE <path>\<filename>.jpg
2 _PRIM Y
2 _PRIM_CUTOUT Y
2 _POSITION <x1> <y1> <x2> <y2>
2 _PHOTO_RIN MH:<id>
2 _FILESIZE <filesize>

This is very helpful because this avoids a lot of manual work and reduces image storage. This is also supported by other programs like the The Genealogy Reporter.

Is there any chance to get support for this in FH?

A further issue is that MH GEDCOM contains parts that are flagged as errors by FH, which cause exclusion of the following parts of an individual, like images. Especially text contained in sources and general notes are sometimes placed 1:1 in GEDCOM without CONT tags for longer texts with newlines. FH seems to be strict and bails out of processing.

This may be an error of MH, but getting fixes done in MH is a long process. I had found issues in MH before and it took months to get feedback and corrections. FH could be more tolerant here by either taking the text as is till the next valid tag, or just drop the content.

I will open new tickets on the support website for both issues.

[tatewise]

FH does support Media cut outs but since it is a non-standard GEDCOM extension uses different tags.
See any FH Media tab and Edit an image where there is a Link to Face/Detail option for creating cut out frames.
It might be possible for a custom plugin to convert from MH _POSITION tags to FH _AREA tags.

Please post explicit examples of the further issues as it may be possible to suggest workarounds.
Usually, FH is quite good at preserving invalid GEDCOM data in Uncategorised Data Fields (UDF).
See FHUG KB Handling Uncategorised Data Fields which explains how to find them and possible fixes.
It is one of the popular Importing to Family Historian techniques.
However, if MH is exporting invalid GEDCOM then it is not fair to expect FH to handle that in all cases.
You have to resort to editing the MH exported GEDCOM file.

What FH Tools > Preferences > File Load/Save > File Loading Options have you chosen?

[egn]

The main import problems seem to be caused by LF characters in NOTE and CONC tags:

grafik.png (9.97 KiB) Viewed 641 times

This causes the following log messages:

grafik.png (11.12 KiB) Viewed 641 times

I had a look into the GEDCOM specification and found the following:

grafik.png (16.42 KiB) Viewed 641 times

I interpret this that only CR terminates GETCOM records that are continued using CONC tag, but FH seems to take the NL characters as record terminator, too.

Of course, on the other side the grammar also defines all combinations of CR and NL as general record terminators:

Code: Select all

terminator:=
[carriage_return | line_feed | carriage_return + line_feed |
line_feed + carriage_return ]

IMHO the specification gives some room for interpretation here.

I also didn't have an issue with numerous other GEDCOM tools I have used in the past and currently. I.E. the NOTE texts appear without any problem in the reports generated by "The Genealogy Reporter", even in very old versions.

I used the default File Loading Options and I have found no setting to control the use of NL. I am wondering a bit that this hasn't be recognized by other NOTE users before.

What do you think?

As I want to stay with MH and use FH for some reports and diagrams only, missing notes are no real issue. But that as consequence parts of the records are also dropped is annoying. So my solution at the moment is to remove the NOTE records from GEDCOM completely before import into FH.

[tatewise]

I agree the GEDCOM specifications are sometimes a little ambiguous!

How did those LF characters get into the Notes in MH?
I would like to be able to reproduce the effect and see how FH handles them.
I'm sure my MH trees have multi-line Notes that do not have that LF problem when exported to FH.

To exclude Notes from the GEDCOM, use the MH Export GEDCOM: Pick Content dialogue and untick Save Notes.