* Dealing with Media and Possible errors in an Ancestry Gedcom Import

Importing from or exporting to another genealogy program. This is the place to ask.
avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 15 Aug 2020 20:42

HI Mike
sorry it's late, but I had to get internet back, then deal with 200 chickens, 2 dogs, and various other things. I have now done as you said, I belive to the Letter. As follows.

But think I misunderstood one thing? Re the Untick/Tick MOVE to Notes, thing, didnt seem to stop The address thing. BUt I suspect you meant me to leave the BOX unticked as I imported each of the SUB Geds ?? Whereas I read the notes slightly different. So I went back one stage, and Did the Import BEFORE I Ticked the box to go back to default.
Hope fully that gives the results you thought. As follows

TRY 2 step by Step (2ND ATTEMPT at try2)
SO Started a TRY two, deleted projects already in FH projects file in Documents.

1 Created a folder called Play 2, placed a new copy of the Whole Ged as sent in that folder.( named Latest Griffiths Family Ged)

2 Opened another FH project, ( can be any, so click FH icon and choose) and selected FH Sample. OPENED TOOLS>> PREF’s>> FILE LOAD/SAVE>> UNTICKED Move invalid data to note field ( remember to reset to default when finished)

3 FILE>> PROJECT WINDOW >> NEW PROJECT >> IMPORT A GEDCOM>> DIRECTED TO THE GEDCOM IN No1 above>>NEXT>>

NOTE the Choose Import File Box now seen has under the Gedcom File name box, a ticked box names ‘copy any linked multimedia files into the new project folder(recommended)

Mike, I LEFT this box ticked, as no mention in the instructions, and I assume as will be deleting the media stuff anyway its not an issue?

Continued, word wrapping was correct

4 THEN a Box on screen warning Imported Data contains 662 broken links to Ext Media Files Repair? Selected NO Then warns some issues logged see Log file on windows menu >> OK
FH created a NEW project in the project store named as the Ged in 1 above.
5 Looked in log files, and there are Two, 1st named ‘Process imported Gedcom Extensions on project’, and 2nd named ‘ Exception report for Gedcom import on project.
QUESTION MIKE, in case in the future I might want someone’s complete Ged rather than a line or two
The first folder holds items Like ADDR converted to RESI , MILT to Military , ALIA etc I assume these are corrections handled by FH built in plugins?
File two This holds quite a lot. As part of which follows. IF I needed sort these is that a manual process, or are there ways? I checked some of the items, like I.30 And I.37 and it appears to have moved the ADDR items listed into the NOTES field of each Fact. How would these be corrected easily, without having to know the coding to do so?

l.30 - INFO ONLY: Loaded uncategorised data (non-GEDCOM): "1 ADDR 17 Ash Grove"
l.31 - INFO ONLY: Loaded uncategorised data (non-GEDCOM): "2 DATE Bet. 1968–1978"
l.32 - INFO ONLY: Loaded uncategorised data (non-GEDCOM): "2 PLAC Wells, Somerset"
l.33 - INFO ONLY: Detected & fixed invalid use of EVEN (event) tag: "1 EVEN Napier Road Infants and Junior School"
l.37 - INFO ONLY: Loaded uncategorised data (non-GEDCOM): "1 ADDR 55 Flanders Road"
l.48 - INFO ONLY: Loaded uncategorised data (non-GEDCOM): "2 PLAC Gloucester"
l.49 - INFO ONLY: Detected & fixed invalid use of EVEN (event) tag: "1 EVEN East Ham Grammar School for Boys"
l.53 - INFO ONLY: Detected & fixed invalid use of EVEN (event) tag: "1 EVEN Wells Blue School"

Record Type=Individual. Gedcom Id=P4. Record Number=4.
l.141 - INFO ONLY: Detected & fixed field format error (data moved to Note Field): "1 RESI 20 Overbrook Close"
l.144 - INFO ONLY: Detected & fixed field format error (data moved to Note Field): "1 RESI Old Pike House"

6 Moved on to next item
Selected the person(s) whose lines I wished to export as a separate gedcom. USING the ALL RELATIVES diagram, I selected all I wanted to Export and placed them in 3 named Lists.
CALLED those lists
A DG FAM and Main Lines Only
B DG’s Cousin Susan Gladwin or McCarthy Line ALL

C Alan Hart Line From Lionel Levey

Use File > Import/Export > Export > GEDCOM File command and Select... the branch of Individuals to export

(DONE USING NAMED LIST PERSONS) for each of the three above

Leave other settings at defaults except for setting Media Records: Exclude and click OK.

DONE

Save the exported GEDCOM files with a memorable sub-branch name.

THEN GEDCOMS EXPORTED with SAME name as named list. ( A-C above)
As follows

FILE>> EXPORT>> GEDCOM>> SET Media Records to exclude >> CHOSE named List, >>>ADDED ALL>> DID for all three A-C



QUESTION

7 Checked said Export has all the people I required in this Ged ( as there is a second variant I need, with more people in , hence the Questions in No 4 above). Because with more people in the split GED I assume there will be more errors to correct?

8 SO DONE PROPERLY following the next repeat paragraphs

Sub-branch GEDCOMs

1 Use File > Project Window > New Project > Import a GEDCOM file to import the GEDCOM file exported above.
2 Use Tools > Preferences > File Load/Save and tick Move invalid data into note fields where possible to return to default

(If you need to repeat the whole process then remember to untick that setting.) I suspect the Window > Log Files will only report exceptions for uncategorised data ~ please confirm. SO AS FOLLOWS


I ABSOLUTELY LEFT THE Move Invalid data to Notes Box UNTICKED for both the export of the three named list Geds I wanted to create from the Ancestry Main ged, as I did the export, and as I did the Import of the sub Geds,as Projects using the project window NEW project.
BUT no Log Files at all. So RAN UDF Plugin as requested for all three, and have the three files.

Have appended the main one.

Actually I did this whole part of the process twice, because I didn’t get any log files the first Export/Import as you said do, and believed I had messed up.. Which I hadn’t it seems? ( Bet you find something I did wrong :-( )
Unidentified Data References.txt
(228.57 KiB) Downloaded 12 times

What next please?
Bill

User avatar
tatewise
Megastar
Posts: 18840
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by tatewise » 16 Aug 2020 10:26

I have made some changes to the original instructions I posted on Friday as follows...

It is imperative you use Tools > Preferences > File Load/Save, untick Move invalid data into note fields where possible, and click OK before doing any importing from Ancestry.
It is this first import of the full Ancestry GEDCOM that requires that setting. Sub-branch exports & imports don't care.

You can untick Copy any linked multimedia files into the new project folder (recommended) as there are no media files.

The sub-branch imports do not produce Log Files.

BTW to answer your QUESTION:
The Window > Log Files > Exception report... will vary depending on how the originator of the tree entered the data.
Some can be fixed as we will be doing here using Plugins, others may need manual edits, and some are for information only and have already been auto-corrected. You will need to learn and keep notes for the various cases.
The Window > Log Files > Process imported Gedcom extensions... are not performed by Plugins, but are a special import feature designed to adjust Ancestry (and other products) poor use of GEDCOM 'extensions' such as ADDR, MILT & ALIA.
They have been permanently applied to the full Ancestry imported Project.


How to handle UDF

The UDF List ZIP file you posted provides the details I needed (but other sub-branches may be slightly different).
If any of the instructions below do not match your data then stop and ask for advice.

Please open the attached Move _UNCAT to Address Plugin in FH and ensure it is listed in Tools > Plugins.
It can also be obtained from Knowledge Base > Compendium of Conversion Plugins.
  1. After running UDF List click on the Item Tag column heading to sort them into order.
    Top of that column will be _APID, followed by such as _UNCAT, DATA, PAGE, etc.
    .
  2. All those _APID tags listed first are Ancestry Person ID numbers that you do not need and can be deleted.
    So select every cell in the left hand UDF Item column that holds an _APID number such as 1,6820::0.
    i.e.
    Select the top cell, scroll down to last numerical cell in 1st column, hold down Shft key, and select that cell.
    ( Make sure that cells associated with _UNCAT, etc, are NOT selected. )
    Hit the Delete key on your keyboard and accept warning to delete all those _APID numbers.
    .
  3. Run UDF List plugin again and click on the Item Tag column heading to list _UNCAT at the top.
    Important: Save a copy of this Result Set for future reference and post it here, we will need it later.
    Sometimes Death events have the Cause in the _UNCAT UDF and need a special Plugin, but the ZIP sample you provided does not have any of those.
    .
  4. Run the Move _UNCAT to Address Plugin.
    That moves every 1st column UDF Item value for _UNCAT tags into the Address field of the associated fact.
    Some of those values may hold age or occupation values that will need fixing by hand.
    That is where the saved Result Set from step C above comes into play.
    I suspect many of those Residence (INDI.RESI) facts should really be Census events so may need further fixing.
    I will give you further instructions once you have successfully got this far.
Attachments
Move _UNCAT to Address.fh_lua
(2.34 KiB) Downloaded 14 times
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 16 Aug 2020 19:45

Hi Mike
sorry, never got a notification, and now out of time, will look tomorrow and get back to you

Many thanks
Bill

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 18 Aug 2020 19:16

Mike apologies, but had some 'family ' trauma's Just not had a free moment. Hope to get tomorrow night or Thursday.
I havent given up
thanks Bill

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 20 Aug 2020 23:35

Hi Mike
I had my issues, then was sent a new gedcom as errors in last. Have now Exported 4 geds from the one received, Which should be the versions I need , and have produced and saved a UDF file for each. So Hope to follow your instructions tomorrow. Unless there is anything to add to what I am doing, or is it worth sending you the 4 files? Do they have a value in respect of what you say do next?

I assume that I need open each Ged , one at a time, rerun the UDF tool, then sort as you said in last post? and so on.
thanks
Bill

User avatar
tatewise
Megastar
Posts: 18840
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by tatewise » 21 Aug 2020 09:38

Rather than open each GEDCOM file, I advise you import each one into a new Project using steps 1 to 6 from last Saturday.

Use Tools > Preferences > File Load/Save and untick Move invalid data into note fields where possible beforehand.
You can untick Copy any linked multimedia files into the new project folder (recommended) during the import.
Then export and import the required sub-branches into further Projects as you did last week.

Run the UDF List Plugin on each sub-branch Project and sort the Result Set as discussed before.
Delete the _APID data and move the _UNCAT data as explained in steps A to D last Sunday.
Run the UDF List Plugin again and let me know what UDF still need fixing.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 21 Aug 2020 09:46

Hi Mike
I have already done the first two paragraphs. I split the part Geds out exactly as last week. Then imported each into FH as a project using your advise. After I had imported each gedcom I ran the udf plugin, just to get a list for each as it was at import, which I saved.

So will now open each again, run UDF tool again, and follow your processes ok!
Bill

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 21 Aug 2020 21:23

Hi Mike
sorry for Delay's.
Did as you said, on the first of the four, named ' DG's FAM and Lionel Levey + MAin Only' best get one right frst eh?
SO

Did exactly as above on the first of the four ‘ DG’s FAM and Lionel Levy+ Main lines Only
Saved a copy of file at stage C.
Ran Move Uncat to Address Plugin.

Noted that still under David Griffiths and others have an issue in Facts , Ie Residence Place says East Ham, London, England BUT Address is empty, and the address is in NOTE .
So message and file stage C attached
Do
APID deltd now stage C UDF's.txt
(20.44 KiB) Downloaded 12 times
ne

User avatar
tatewise
Megastar
Posts: 18840
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by tatewise » 21 Aug 2020 22:37

Take a look at step C, UDF List plugin Result Set for Morris James Victor GRIFFITHS and Lionel LEVEY, et al, who have several Residence facts each that should have Address data derived from the UDF Item value in 1st column of Result Set.
In other words, the Residence Information has been handled correctly by the Move _UNCAT to Address Plugin.

I assume that for David Griffiths and others, the Residence fact had its address info in the Note field in Ancestry to start with.
i.e.
The person who entered the data has not been consistent with which fields were used for address info.
Sometimes they entered it in the Information field and sometimes in the Note field in Ancestry.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 22 Aug 2020 21:28

Hi Mike
ok, so how do I get it in the right field, before we move on to whatever you think is the next bit to do? I kep t the file at step C as you said we would need it?
Bill

User avatar
tatewise
Megastar
Posts: 18840
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by tatewise » 22 Aug 2020 23:24

My suspicion is that the data has been entered in such an inconsistent way that there is going to be no easy solution.
You will have to work through the entire data record by record and fact by fact and copy & paste the data as necessary.

Looking at the step C Result Set data, the information that has been moved to the Address field includes Age, Occupation, relationships, etc, etc, as well as Address details.
So that will all have to be moved to the Age field, an Occupation fact, etc.
There is no way that can be automated easily. Sorry!
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 23 Aug 2020 12:58

Hi Mike
I have looked at the file I kept at stage C. There doesnt seem to be 5 clear columns, though the headers suggest that. IE UDF ITEMS--'RECORD' 'RECORD TYPE' 'ITEM TAG' 'ITEM DATA REFERENCE
should they be seperate and identifiable by '' '' as dividers?

A lot of the first column seem to be the addresses mentioned a few times. Where they need to be in the Address field of FH , but are underneath in the notes, Are you saying the easiest or only way now, is identify each bit of information and correct manually , perhaps as and when I notice them?

I suppose that without looking at each one in the list, then cannot be sure where they are, and where they should be?? So no way of doing a mass move.

So later today if time, shall I do the other three, and get to same stage, and pass for comment? I think a couple will have very few errors like this.

I appreciate all your input Mike. I guess if he hadnt done so many 'strange' entries, then I would have relatively clear files by now?

thanks again
Bill

User avatar
tatewise
Megastar
Posts: 18840
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by tatewise » 23 Aug 2020 13:47

It would have better if you had saved the C stage Result Set as a CSV file, but that can be fixed.
Rename the file to be a .csv file instead of a .txt file and accept the warning message.
Then that .csv file will open in a spreadsheet such as Excel or Calc organised in 5 clear columns.
BTW: You can also open the .txt file with Excel or Calc as long as you choose Tab separators.

The values in the 1st column will have been moved into the Address field but only for the specific facts listed.
So lookup Morris James Victor GRIFFITHS and Lionel LEVEY, et al, who have several Residence facts (INDI.RESI).
Those facts will have the 1st column value moved to the Address field.

BTW: You can easily find Individuals such as Lionel LEVEY in the Records Window by using the Filter search at the top to enter Last Name = LEVEY and First Name = Lionel.
One of his Residence facts has Address value 71, Prestbury Road, Age: 43Relation: Head; Bricklayer for East Ham Council
But I suspect this Residence fact should really be a Census event ~ what do you think?
With Address field = 71, Prestbury Road and Age field = 43
An Occupation fact needs creating to hold Bricklayer for East Ham Council

There is no point looking at Residence facts for other people not listed in that Result Set file.
Those facts will have their data in the Note field derived directly from Ancestry.

The snag is that if the data was entered in a haphazard way then it can only be sorted out by hand one item at a time.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 31 Aug 2020 18:43

Hi mike, I have played a bit, But as Genome MAte Pro says ''It's Complicated' Will be in touch if need any more help Thanks. Bill

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 09 Sep 2020 15:46

Hi mike
I added a husband to a gedcom for my 2 x great Gran, and then needed to export a Ged that was Main Line Ancestors only, to add to Genome mate Pro, This also needed to be in UTF8, which uses it to allocate 'groups' and MRCA's. The group names generally are the surnames of the ancestors parents. So Mary An Chapman the younger, in the exported ged, has father Lionel Levey, and Mother Mary Ann Chapman. Every time I imported the ged into GMP, it gives the younger Mary a group of Chapman-Chapman. Obviously wrong, but no matter what I try I can't seem to stop that happening. The ged looks fine in FH, and I tried in a more basic program, also fine. Might you have any ideas please? In case needs something cleaned as per this thread? So I thought to see what GEDCOM Validator made of it... Obviously loads bad tags from the Ancestry Part etc, as per our discussions, but that doesn't matter for things like this, as only the basics needed. What threw me was Gedcom Validator has a big red X, is that a warning? I can't now find much on what the error codes mean!
Ged Validat .png
Ged Validat .png (183.66 KiB) Viewed 577 times

User avatar
tatewise
Megastar
Posts: 18840
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by tatewise » 09 Sep 2020 16:35

Don't worry about the red X. It is only saying UTF-8 encoding is not strictly allowed by the GEDCOM 5.5 standard specification, but it is used by most genealogy products, so no problem. It only affects the way the characters are represented and not how the family data is recorded.

I had a quick look at Genome mate Pro online but could find no definition of what GEDCOM formats it accepts.
Have you used it before?
Why do you think Ancestry compatible GEDCOM are what it accepts?
How did you create the WR Main UTF-8 exp New.ged file?

It looks like a free download and if I can find some time I might be able to install it and try it GEDCOM import.

Perhaps you would be better asking Genome mate Pro on their Facebook User's Group at https://www.facebook.com/groups/816785941743656/
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 09 Sep 2020 18:29

Mike , sorry, we are a little at cross purpose. My fault. In this case, I just needed a MAin Ancestor GEd, the biggest issue is the fact it splits my great grandparents. More than the stuff to so with whats in wrong fields. The ged I need in this case is only names /dates. I havent had time lately because a lot of up an down to Lonodn sorting a 92 year old friends life, to look in detail at the rest. But it seems wrong that GMP doesnt reconise Lionel as one half of a couple?
Bill

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 09 Sep 2020 18:30

Genome says use UTF8 where possible, so I exported one!

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 09 Sep 2020 18:32

Genome mate is free, and it's powerful in what it does, in databasing tens of thousnads of DNA matches and allowing IN commons and triangulation work.

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 09 Sep 2020 18:35

Sorry in parts, but my internet is playing up. Ok I asked genome Mate. I created the UTF8 by splitting my main GED, adding Lionel Leveys line as slit ( main only) from the Ancestry GEdcom, checking all ok in FH , and then exported a UTF8 which I didnt open, as FH converts to UTF16, and uploading to GMP.

User avatar
tatewise
Megastar
Posts: 18840
Joined: 25 May 2010 11:00
Family Historian: V6.2
Location: Torbay, Devon, UK
Contact:

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by tatewise » 09 Sep 2020 19:13

Bill, I am sure you know what you are describing but it makes little sense to me.

For example, what does this mean: "adding Lionel Leveys line as slit ( main only) from the Ancestry GEdcom"?
What is a "slit"?

You need to describe the commands you are using otherwise I cannot follow what you are doing.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry

avatar
Billread
Diamond
Posts: 93
Joined: 21 Jan 2009 08:58
Family Historian: V6.2

Re: Dealing with Media and Possible errors in an Ancestry Gedcom Import

Post by Billread » 10 Sep 2020 11:17

Morning Mike. Because of the internet dropping in and out I did not notice my type errors. Slit was 'split' . I think I got to the bottom of this issue. It was because Mary Ann Chapman , is shown in the Gedcom as that, with Mother as Chapman as well. Then because Lionel Levey as father was found with DNA use, when adding to the Gedcom he is added as Levey, but her surname remains Chapman. So when the information is read into GMP , it reads as Chapman-Chapman, rather than Levey-Chapman. An easy thing to miss. As for the rest. I have between other things been cleaning split geds from ancestry up, but am still left with residues as from before, that it seems what you stated was correct, to live with it, and as I see the bits that couldnt be corrected in what we did, then just manually alter them one by one. Its difficult to find time to go through and do a whole list in one go. Maybe one day someone will find a way! If you have any more ideas let me know, Have a good day. Bill

Post Reply