* Flexible CSV Importer problems

For users to report plugin bugs and request plugin enhancements; and for authors to test new/new versions of plugins, and to discuss plugin development (in the Programming Technicalities sub-forum). If you want advice on choosing or using a plugin, please ask in General Usage or an appropriate sub-forum.
Post Reply
avatar
Ricardus
Gold
Posts: 10
Joined: 05 Aug 2020 15:10
Family Historian: V6.2
Location: Aberdeen

Flexible CSV Importer problems

Post by Ricardus »

I am trying to upload data from Excel to FH (V6) using the Flexible CSV Importer plugin, however after numerous attempts I am still unable to load anything - I always get the message '0 people added'

I created the file in Excel, exported it as a csv and then attempted to upload it to FH using the Plugin. To try and identify the problem I have pared the test data right down to name, sex and place of birth, no sources or anything else. But still, noting uploads to FH, I get the message:

"4 rows with 16 cells of information processed.
0 people added
0 families created"

I attach screenshots that will hopefully show what I have been trying to do

Grateful if anyone could let me know where I am going wrong, I imagine it is something pretty basic! Hopefully once I get one file uploaded properly I can then move forward with this process!
Screenshot 1.png
Screenshot 1.png (47.28 KiB) Viewed 3846 times
Screenshot 2.png
Screenshot 2.png (39.56 KiB) Viewed 3846 times
Screenshot 3.png
Screenshot 3.png (53.81 KiB) Viewed 3846 times
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Flexible CSV Importer problems

Post by tatewise »

Please review the Help page available via the Help button and check the examples there.

The key column you are missing is the REFN identifier that is similar to the Record Id within FH.
Without that the Plugin does not really know which CSV rows belong to which Individual record.
e.g. You might have two rows for one person with two alternate Names and so would have the same REFN identifier.
In most cases, each row has its own unique REFN identifier.

That is also the way that father, mother & spouse relationships are established.
e.g. The S.REFN column holds the REFN identifier for the Spouse of this person.

I am currently reviewing this Plugin, so thinking maybe it needs to report when there is no REFN column.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Flexible CSV Importer problems

Post by Valkrider »

Some time ago I wrote a simple XL to Ged converter not as sophisticated as Mike's but it does work. You do, however, need to put your data into a spreadsheet template. If you want to give that a try it is available at https://orba-design.com/genealogy/xl-to-ged/
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Flexible CSV Importer problems

Post by tatewise »

Note that Colin's spreadsheets have an ID column that performs the same function as the REFN column.
Be aware that his spreadsheets do not support family relationships for parents, children, or spouses.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Flexible CSV Importer problems

Post by Valkrider »

tatewise wrote: 05 May 2021 12:29 Note that Colin's spreadsheets have an ID column that performs the same function as the REFN column.
Be aware that his spreadsheets do not support family relationships for parents, children, or spouses.
As I said @Mike basic import, I haven't done anything with that converter for about 8 years. Lots of people still use it though that is why I have left it online.
avatar
Ricardus
Gold
Posts: 10
Joined: 05 Aug 2020 15:10
Family Historian: V6.2
Location: Aberdeen

Re: Flexible CSV Importer problems

Post by Ricardus »

tatewise wrote: 05 May 2021 11:47 Please review the Help page available via the Help button and check the examples there.

The key column you are missing is the REFN identifier that is similar to the Record Id within FH.
Without that the Plugin does not really know which CSV rows belong to which Individual record.
e.g. You might have two rows for one person with two alternate Names and so would have the same REFN identifier.
In most cases, each row has its own unique REFN identifier.

That is also the way that father, mother & spouse relationships are established.
e.g. The S.REFN column holds the REFN identifier for the Spouse of this person.

I am currently reviewing this Plugin, so thinking maybe it needs to report when there is no REFN column.
Many thanks Mike. I think I have now grasped the concept of the REFN identifier and have now been able to successfully upload csv files to FH. :D

I assume (from subsequently playing around with test data) that once an upload has taken place and IDs and Custom IDs assigned to records, you cannot then run a further upload which will update existing records, you can only create new records. Thus, for example, if I had a csv file of 100 births, to which I assigned the Custom IDs F1-F100, the upload would create a file of individuals in FH with IDs 1-100 and Custom IDs F1-F100. Let's say I subsequently created an Excel and csv file of 100 deaths and determined that item 50 ('Mr X') from this new file was the same individual as F25 from the original births upload. On uploading this new deaths file I therefore assigned Mr X's death information line a Custom ID=F25. The upload would (I assume) create 100 new records with IDs101-200, with Mr X having ID =150 and Custom ID=F25. I would then, I assume, have to merge the two records, ID25 and ID150 (both of which would have Custom ID=F25)? Or have I got this completely wrong?
Valkrider wrote: 05 May 2021 12:01 Some time ago I wrote a simple XL to Ged converter not as sophisticated as Mike's but it does work. You do, however, need to put your data into a spreadsheet template. If you want to give that a try it is available at https://orba-design.com/genealogy/xl-to-ged/
Thank you Colin I had a go at your Excel to Gedcom converter and was successful in loading data to FH. :D I am in the early stages of a one name study, collecting lots of information and trying to figure out the best strategy for getting it into FH; that will take quite a few weeks of testing I think!
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Flexible CSV Importer problems

Post by tatewise »

Yes, your analysis is absolutely correct.

However, it begs the question "Why use CSV?"
If you are collecting research information from various sources then why not enter it directly into FH possibly using AS?

CSV is intended for where you already have tabulated data in some form and want to bulk enter it into FH.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Ricardus
Gold
Posts: 10
Joined: 05 Aug 2020 15:10
Family Historian: V6.2
Location: Aberdeen

Re: Flexible CSV Importer problems

Post by Ricardus »

Thanks Mike
Indeed for my own family tree I do as you say - use Ancestral Sources or key information directly into FH.
But I am also now setting out on a one name study, where I have obtained hundreds/thousands of records such as births, deaths, marriages and burials, much of it downloaded from GRO, FreeBMD, Ancestry and the like, mainly now cleaned up and sitting in Excel. By manipulating the data in Excel I can create columns for the source and the citation in the format I want. I reckon uploading some or all of this data to FH rather than rekeying it could save me a lot of repetitive time at the keyboard! But of course I'm sure it will not be totally straightforward...
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Flexible CSV Importer problems

Post by Valkrider »

I can possibly tweak the converter for you if you want to PM me a sample of your existing spreadsheet.
avatar
Ricardus
Gold
Posts: 10
Joined: 05 Aug 2020 15:10
Family Historian: V6.2
Location: Aberdeen

Re: Flexible CSV Importer problems

Post by Ricardus »

Thanks Colin - will do
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Flexible CSV Importer problems

Post by tatewise »

I've had a look at trying to synchronise multiple CSV imports using the Custom ID and REFN values.
It is quite easy for the plugin to compile a list of Custom ID versus Individual and Family records.
So in a second CSV import, if the REFN match the Custom ID it uses the same Individual and Family records.
That phase is straightforward and I have that working.

The complications arise when trying to synchronise Facts.
If a person already has a Birth event with various Date, Place, Address, etc, fields and the second CSV has a Birth event where some of those fields may be the same, different, missing, or extra, then deciding that another Birth event is needed gets rather difficult. It seems that the plugin must compare all the fields before it can decide. However, the current design works field by field in isolation. So it would need to analyse the entire row of data fields against each of the existing facts before making a decision about whether any new facts are needed. That is because it cannot assume that all the fields for one fact are clustered together, so it does not know that all the fields have been discovered until the end of the row.
That suggests it may have to be two-pass process: 1) compare old with new facts; 2) create the new facts required;
I'm going to have to sleep on it before getting any further.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Flexible CSV Importer problems

Post by tatewise »

I believe I have resolved the synchronization problems.
Try the attached Flexible CSV Importer plugin Version 1.6.3 Date 9 May 2021.
[ Attachment deleted as it is now in the Plugin Store. ]

It has an extra Settings option to Sync any Custom IDs with REFN of each person ? enabled by default.
That not only synchronizes the records but also the facts. So if for example, the same person has rows with the same REFN in any imported CSV file(s) that include the same Fact details they will not get duplicated. However, if the Source details are different for that Fact then they will get added to the synchronized Fact.

I've also adjusted the Directions Step 3 so that the dialogue does not expand when the import completes.

This version also fixes problems when handling Names and adding Facts for relatives on the same row as the Primary person.
e.g. Facts with the prefix S, PF, PM, P_S, etc.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Ricardus
Gold
Posts: 10
Joined: 05 Aug 2020 15:10
Family Historian: V6.2
Location: Aberdeen

Re: Flexible CSV Importer problems

Post by Ricardus »

Mike, Many thanks for making these changes to the plugin.

I have been loading some simple test data today and so far it is looking really good, with the additional functionality of loading new facts to the same record when the same Custom ID is involved. That now gives me the opportunity to carry out multiple csv uploads.

I also played about with loading new values for facts that were already present and the result was the creation of an additional fact, e.g. two births. However, in practice, I would probably not want to do this anyway and would probably change existing fact values directly in FH rather than via csv upload.

I will give it a harder workout over the next few weeks when I load in some larger and more complex datasets and will let you know how I get on.

Richard
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Flexible CSV Importer problems

Post by tatewise »

Thank you for the positive feedback.
The ability to load multiple facts with different values is more applicable to Census and Occupation facts than BMD events.
Although, multiple Marriage events may fall into that category.
You can also add multiple Source Citations for the same fact using that technique.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply