* Repeated Merging - Duplicate Media

Importing from another genealogy program? This is the place to ask. Questions about Exporting should go in the Exporting sub-forum of the General Usage forum.
Post Reply
avatar
SquashIt
Gold
Posts: 14
Joined: 11 Mar 2022 20:38
Family Historian: V7

Repeated Merging - Duplicate Media

Post by SquashIt »

I imported to FH from a GEDCOM which was exported from FTM. After making changes in FTM I repeated the process many times. Now I notice I have multiple media files attached to individuals (I assume other records as well).

I have used FH to create diagrams, created text schemes and added record flags, but no changes to data.

How do I get out of this mess? Should I start again, but can I copy my diagrams, etc. or should I try and fix the current FH data. Either way how do I stop it happening?

I just noticed also a birth record had not updated in FH which I knew I had changed in FTM. On investigating I found two birth records on the individual. So the merge process does not update changed records?
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Repeated Merging - Duplicate Media

Post by tatewise »

Welcome to the FHUG.

I suspect you are not checking all the tabs in the Merge dialogue, which is a common mistake.
See the FHUG Knowledge Base Merge/Compare File advice.

Without knowing the details of how many duplicates and other problems exist it is difficult to advise the best solution.

The Merge does not automatically combine facts such as Birth. You must use the Merge features to do that.

If you start again then saved Diagram Types and Text Schemes and Flag names will be preserved as they are global.
Saved Family Historian Charts will not be preserved. Flags will not be associated with Individual records.

From a strategic perspective, you must decide which product is going to hold your master database.
You must choose either FH or FTM. It is not feasible to update them both in parallel.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
SquashIt
Gold
Posts: 14
Joined: 11 Mar 2022 20:38
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by SquashIt »

Thanks for the reply.

I am using FTM as my master and want to make FH an exact copy. I don't want to lose saved charts.

From your comments, I assume going forward, I will need to keep detailed notes on what changes have been made in FTM and use the merge dialog to find them. It would be helpful if the dialog had more features to ignore data that matches and only see differences - am I missing something?

I don't understand why the merge does not match up the incoming media records when they have not changed?
User avatar
Mark1834
Megastar
Posts: 2458
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: Repeated Merging - Duplicate Media

Post by Mark1834 »

SquashIt wrote: 11 Mar 2022 22:43 I am using FTM as my master and want to make FH an exact copy.
A cautionary note - most non-trivial genealogy programs (and certainly all of the "big three", FTM, RM and FH) include custom features in their app that are not in the others. It's how the publishers make their product distinct in order to attract your cash.

While most basic data should transfer between applications without corruption, it is extremely unlikely that custom elements such as detailed charts, reports, will transfer seamlessly. You will probably have to settle for the data being as close as possible, but accept presentation differences.

As a generality, FH and RM play together better than FTM, which has more custom features and a "closed" data structure that is not accessible to other apps.
Mark Draper
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Repeated Merging - Duplicate Media

Post by tatewise »

IMO the FH File Merge feature is not designed to provide the kind of synchronisation you are looking for.
So you will have to help it to import the latest data from FTM.

The key to keeping FH Saved Charts and Record Flags intact is to preserve the Individual Record Id.
So it is crucial that Individual records are not deleted and act as the unique key between FH and FTM.

Have a look at File > Split Tree Helper... that offers a lot of data deletion options.
Experiment by using that to delete all records and facts just leaving Individual and Family records.
Then File Merge the latest FTM GEDCOM.
Hopefully, it will synchronize the Individual records using Names and close relatives, and all other data will be imported anew.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
LornaCraig
Megastar
Posts: 3190
Joined: 11 Jan 2005 17:36
Family Historian: V7
Location: Oxfordshire, UK

Re: Repeated Merging - Duplicate Media

Post by LornaCraig »

SquashIt wrote: 11 Mar 2022 22:43 I don't understand why the merge does not match up the incoming media records when they have not changed?
Can you post a screenshot of the Multimedia tab of the Merge dialog? (Preferably with some of the records expanded so that the sub-fields, if any, are visible). This might shed some light on why FH thinks it can't merge the multimedia records.
Lorna
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Repeated Merging - Duplicate Media

Post by tatewise »

FTM does not export Media Records in its GEDCOM but uses Local Media Objects instead (as do many products).
When importing/merging they get auto-converted to Media Records probably with a current Updated date-time stamp.
I suspect that the differing date-time stamps between previous and current records inhibit the auto-merging.
I don't have FTM to experiment with so it is difficult for me to reproduce the symptoms.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
SquashIt
Gold
Posts: 14
Joined: 11 Mar 2022 20:38
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by SquashIt »

For anyone facing this problem, here is what I've done to fix my mess:

- Delete media and source records from FH. Note: you cannot delete multiple records directly, you have to use a named list.
- Find the media folder on your computer in your project and delete the media files. Deleting the records from FH does not appear to delete the files.
- Merge/Compare from your source GEDCOM.
This will restore the project so that only one copy of each media file exists and deduplicate the citation records.

I then created a new temporary project and imported the GEDCOM file into that. I then did a Merge/Compare of my restored project against the temporary project, examining each individual for differences. I discarded all media files. However, after this process I noticed the citations had duplicated again, so maybe also discard source records?

This gets me back to square one, but I don't want to do this as an ongoing process and would like to know why the media files are not matched up.

Here are screenshots of the media tab of the merge dialog:
Screenshot 2022-03-12 091702.jpg
Screenshot 2022-03-12 091702.jpg (33.15 KiB) Viewed 2843 times
Screenshot 2022-03-12 091559.jpg
Screenshot 2022-03-12 091559.jpg (29.68 KiB) Viewed 2843 times
Expanding the + shows the file name which is the same.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Repeated Merging - Duplicate Media

Post by tatewise »

Please try the File > Split Tree Helper... as I suggested because it does allow all Media and Source records to be deleted in a single step, much easier than using Named Lists. It may also resolve the duplicated Facts problem you mentioned.

Yes, you will need to delete the Media files separately, but the Check for Unlinked Media plugin will do that.

When you use the [+] expansion are the date-time stamps the same for both sets of Media?
In your screenshot, I can see one set has 3/12/2022 08:30 but you have omitted the others by cropping on the right.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
SquashIt
Gold
Posts: 14
Joined: 11 Mar 2022 20:38
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by SquashIt »

Thanks for the tips. I hope not to have to do this going forward, if I'm careful about future merges, especially by not adding media.

There are no timestamps on the media files being added in the dialog, even when the list is expanded. The only difference is in the folder structure of where the file is imported from and where it's stored in FH.

Interestingly, if I compare the file properties in File Explorer, the created dates are different. The FH version has the date of the import. Maybe that's why they are considered different? If so, I would have to classify that as a bug, but maybe there is some justification for always importing media that I haven't considered.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Repeated Merging - Duplicate Media

Post by tatewise »

So it looks like the existing Project Media Records do have date-time stamps and the Media Records being merged do not.
That difference is prossibly what prevents them auto-merging.

The Media Files (as opposed to Records) are copied from the FTM file path to the FH file path (if you choose that option).
By definition, when Windows copies files, the Creation Date is always the current date-time.
However, I don't think the file dates are the problem.

If the merged GEDCOM is the one directly exported from FTM then the File links in the Media Records will be the FTM file paths, whereas the existing File links in the existing Media Records will be FH file paths.
That difference will definitely prevent auto-merging.

If you import the FTM GEDCOM into an FH Project before merging then that difference will be eliminated and perhaps then the auto-merge of Media Records will go ahead.

To understand those subtle differences, Lorna asked that you posted Media Records in the Merge dialogue with the [+] expand button clicked to show all the subsidiary fields such as Title and File and Format. Unfortunately, that screenshot has not been forthcoming, yet.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
wildbill
Platinum
Posts: 34
Joined: 16 Nov 2021 18:16
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by wildbill »

I have never used FTM but I have invested data in Rootsmagic 7 and I would say there are many who cannot break ties with their previous data home for various reasons and hopefully FH will overcome those issues. Mass duplication in various areas of Rootsmagic was one reason for buying FH but I also have not completely invested in FH as my master database and I will explain later.

Regarding Media, I can say if a drag an existing Media item to Rootsmagic the program is smart enough to recognize it as already being part of the gallery and not duplicate it. I tried the same in FH and the link which is identical is duplicated, the original Media link having the links and the duplicated one having zero links, that is not good.

In Rootsmagic all my linked media have captions so if I drag in one or several new items they will appear in the gallery with No Caption helping me identify them for linking, as said before existing links will be recognized and not duplicated. What that meant was I could drag a complete folder contents to Rootsmagic Gallery and only those with No Caption would be newly added items needing linking. I had hoped the same from FH where regardless of Caption existence they would flag as New Unlinked Media, that is not what I discovered.

It would take a relatively short piece of programming to cycle through existing links and any newly dragged items to resolve duplication and I would hope that becomes a reality in the future.

The reason I cannot completely ditch Rootsmagic is my investment in Place Details with geocoding for mapping, explanation notes and historical and modern media attachments, I lose that in FH. I have every confidence FH will overcome that shortcoming as Rootsmagic, Legacy and Heredis all offer support of those enhanced sub-division attachments, then my tie will be broken but not until then.

I also realise a sizeable user base wanting Ancestry integration and the like, I sincerely hope FH does not go down that road and the burden of maintaining the Treeshare feature appears to be the ruination of Rootsmagic development. I do use Ancestry for research but prefer to download and link the information myself.
avatar
SquashIt
Gold
Posts: 14
Joined: 11 Mar 2022 20:38
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by SquashIt »

Here are the screenshots with expanded items -
Screenshot 2022-03-12 145341.jpg
Screenshot 2022-03-12 145341.jpg (33.57 KiB) Viewed 2783 times
Screenshot 2022-03-12 145244.jpg
Screenshot 2022-03-12 145244.jpg (36.87 KiB) Viewed 2783 times
Yes, the paths are different as you say, but they would be different on the initial import. Probably a scenario that wasn't envisaged.

I'm reluctant to go the route of importing to a temporary FH database because it adds another step. Easier to just discard the media when merging. I just have to watch out for any new items I do actually want to import.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Repeated Merging - Duplicate Media

Post by tatewise »

Thank you for those screenshots, which clearly indicate why the auto-merge is not happening as the records are different.

Yes, importing the FTM GEDCOM to a Project is an extra step, but so is deleting records before the merge.
The import process would convert the Media record File path to the FH Media\.... format and add Keywords.
Creating a Project prior to a Merge is the recommended method as advised in the FHUG Knowledge Base, and I believe is an alternative way of getting the Media Records to Merge.

@wildbill: Have you raised the problems you mentioned in these Forums to see if there is a known workaround?
I am surprised you have problems with Place as FH definitely supports "geocoding for mapping, explanation notes and historical and modern media attachments".
Please do NOT continue such discussions in this thread but start a new topic instead.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
SquashIt
Gold
Posts: 14
Joined: 11 Mar 2022 20:38
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by SquashIt »

Just to be clear (for anyone reading this later), I don't intend to delete anything going forward prior to a merge, that was only a means to get my FH database clean. I intend to merge as before with the addition of checking the media tab, as tatewise suggested, and discarding any media files that would otherwise be duplicated.

I think that will be less effort than creating a temporary project, merge into that from the FTM GEDCOM, switch to the real FH project, merge again from the temporary project, delete the temporary project which closes the current project, then reopen the real project.

I think it's unfortunate the software does not handle reimporting as one might hope, but we have to work with what we're given.
User avatar
wildbill
Platinum
Posts: 34
Joined: 16 Nov 2021 18:16
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by wildbill »

tatewise wrote: 12 Mar 2022 21:48 Thank you for those screenshots, which clearly indicate why the auto-merge is not happening as the records are different.
My media paths and file names are identical, the only exception being the Rootsmagic originated ones use lowercase "d:\" and newly dragged FH entries are noted as uppercase "D:\". I have amended some entries to uppercase drive name and dragged again but the file link still becomes further duplicated, am I missing an option somewhere?

I tried a Plugin to search for duplicate media and it neither found or resolved anything so I'm very confused and believe FH is missing a clean up utility.

@tatewise, that previous comment referred to Address (Place Details in RM)
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Repeated Merging - Duplicate Media

Post by tatewise »

@wildbill: My comments you quoted were aimed at Squashit. Please follow my advice and start new topics for your problems that are quite different from these merge problems.

@Squashit: FYI: You don't need to create a temporary project before merging the FTM GEDCOM but simply import it directly into a New Project using the Import Gedcom option which is much quicker, and just as you did to create your master project initially.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
wildbill
Platinum
Posts: 34
Joined: 16 Nov 2021 18:16
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by wildbill »

tatewise wrote: 13 Mar 2022 11:06 @wildbill: My comments you quoted were aimed at Squashit. Please follow my advice and start new topics for your problems that are quite different from these merge problems.
I was not hijacking a post, rather attempting to provide further supporting evidence of Media Duplication and what appears to be a short coming in Family Historian. In my work method I keep similar items grouped together and not fragmented over several posts, I have always found that is the best way to fully understand the issue.

In my opinion being dismissive or authoritarian never moves the bus forward, perhaps that was not your intention.

I have tried again and duplication of EXACT same file name/path occurs in Family Historian where it did not in my previous program which had it's own duplication problems. I believe if we accept that position and define it correctly then users stand a chance of seeing a resolution in the future, failing to accept a problem usually has the opposite affect.
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Repeated Merging - Duplicate Media

Post by ColeValleyGirl »

wildbill wrote: 13 Mar 2022 10:16 @tatewise, that previous comment referred to Address (Place Details in RM)
Many FH users adopt a way of working where the address and place details are all held in a place record (and don't use Addresses), as described in Working with Places and Addresses. That allows them to use "geocoding for mapping, explanation notes and historical and modern media attachments" down to building level. (If you want to explore this further, it definitely warrants a separate topic as it's unrelated to the Media Duplication Issues being discussed here.)
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Repeated Merging - Duplicate Media

Post by ColeValleyGirl »

wildbill wrote: 13 Mar 2022 13:44 duplication of EXACT same file name/path occurs in Family Historian
I don't think anyone is arguing that this is OK -- but has anyone logged the issue with with Calico Pie? It may only become apparent when certain ways of working are adopted -- for example bulk migration of media into FH, or repeated merges (the examples in this thread).

Which plugin did you try: Check for Possible Duplicate Media?

(Also, I've just checked on the Wish List and there's an related wish Duplicate check when adding new Individual or other record types although it doesn't discuss the duplicate media issue in any detail.)
User avatar
wildbill
Platinum
Posts: 34
Joined: 16 Nov 2021 18:16
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by wildbill »

Thank you Helen, that wish was for a feature my previous software did badly and nothing to do with media duplication.

I will test this some more, for example my media are not in the Project folder, maybe try export import to see if the duplication gets resolved that way and will report the issue once fully understood.
User avatar
wildbill
Platinum
Posts: 34
Joined: 16 Nov 2021 18:16
Family Historian: V7

Re: Repeated Merging - Duplicate Media

Post by wildbill »

Reported to FH and passed to development for consideration.
Post Reply