*Work in progress - Import CSV

[Shelley]

I'm hoping some brave soul might test my "Import CSV" plugin. I'm a bit nervous about this. I'm not a programmer, and this is my first plugin for use by anyone but myself.

The idea is to edit the column headings of the CSV to a convention that the plugin understands. You can import almost any Individual or Family field. It will create relationships between the primary person, their spouse, and the parents of both. You can also choose a single source record to add to all entries, but with separate citation details for each.

I've included description of how the columns need to be named in the plugin description.

I intend on trying to add a progress bar. On my PC, import of a fairly simple file of around 3,500 lines takes about 30 seconds.

Not handled at present are addresses and witnesses. Addresses can wait for version 2. Witnesses, maybe version 3.

One question - is there any way to determine if a fact is an event or an attribute just from the tag? I found fhIsEvent and fhIsAttribute, but they take pointers as input. I want to put in an error check on where (eg) occupation and cause of death records are going to end up before any records are actually created.

Also, I have used some code snippets from the site. What's the etiquette about use of these? I assume that the author would include attribution in the snippet if wanted it...? I'm more than happy to give credit where it's due!

Thanks
Shelley

[tatewise]

Hi Shelley,
May I suggest you take a look at existing plugin Build Tree From CSV File in Plugin Store.
Some concepts and functions it uses may be worth considering.

The most significant concept is what details to include on each CSV row, and how to link relations.

The existing Plugin holds details of only one person per CSV row and assigns a unique number.
All relationship columns such as Father, Mother, and Spouses simply use those unique numbers.
This ensures details such as Names are only entered once, and creating relationships is much easier.

I also found it quite easy to add extra columns for other Facts, as explained in Importing from Excel (12455) on Fri Apr 10, 2015.

I believe it should be possible to create an extended version that automatically handles any CSV with column labels based on GEDCOM tags for any Facts.

To answer your question about detecting Attributes, Events, etc, the solution is to use a dictionary table.
e.g.
local dicAttrib = { EDUC=1, OCCU=1, PROP=1, RESI=1, TITL=1, etc... }
if dicAttrib[strTag] then -- tag is an attribute

local dicEvent = { BIRT=1, BAPM=1, CENS=1; MARR=1, DEAT=1, BURI=1, CREM=1, etc... }
if dicEvent[strTag] then -- tag is an event

But you also need to know if it is an INDIvidual Event or a FAMily Event to add it to the correct record.

[Shelley]

Thanks for your reply Mike.

I embarked on this after a conversation with someone who found that the Build Tree From CSV File didn't quite meet her needs. I think that plugin, and this one, have different aims in mind. That one is helpful when trying to bring in information from another database that is already linked up. This is aimed at bringing in data such as, say, database search output that may have relationships within a row, but not not necessarily any linkage between rows. The kind of thing a one-name study might be interested in.

Of course, that said, I know my code isn't perfect and there's a lot I can learn and ideas to be had from other plugins!

My intention here is that the user should have to make only minimal modifications to their spreadsheet (which the many spreadsheetphobes out there would find challenging enough), and no changes to a plugin. The naming convention I mentioned for this plugin uses GEDCOM tags including any custom tag in use. The most difficult part should be working out where to map the fields to, but once you understand the naming convention it's easy to just rename the CSV headings appropriately. At least, that's my intention. I'm happy with how it's working for my files.

I do have a check for if the field can be mapped to a valid tag, including whether it's for an INDI or FAM. The check I'm hoping to do with Events and Attributes isn't essential. Using a dictionary table might have to do, I just won't be able to check custom facts.

[tatewise]

The information you are catering for is something I was unaware of.

Are you saying that your database searches often produce a single spreadsheet row of facts about one person and their immediate family?

[Shelley]

Yes, that's right.

There are lots records that list the person and their spouse, or the person and their parents on one line. For example Australian birth and death indexes typically include:
Year, surname, given name, father's name, mother's name, place of event, registration number.

Scottish indexes are similar. I've also seen transcripts of UK baptism registers with those details presented in much the same way.

[NickiP]

I find it easier when blanket searching a surname in a particular area to add the details to a spreadsheet in much the same way that Shelley has mentioned, particularly if the records being searched aren't online or, if they are, aren't indexed. Spreadsheets can definitely be a useful research tool. Then at a later date its easier to work out who belongs to the specific family that I am researching and who does not.

I wouldn't necessarily want to import all into FH but in the case of One Name Studies I can definitely see the benefit of being able to do this, although it would depend on how it created family units from this information.

[RSellens]

Hi Shelly, have a look at my 'CWCG Build from CSV' which is on the store. I do something very similar with the CSV's produced from the CWCG site. It can extract details from a note field and build families, and events/attributes from columns..

My next version that is still very much a WIP, i have a spreadsheet that lists columns, that i can then use to build 'entry sheets' with the column names, and the plugin then looks for any of these standard names, and builds the relevant events/attributes as needed if the column is present.

Richard

[tatewise]

It sounds like several of you may have a common objective for importing CSV with various columns.

It should be possible to make the Plugin adapt automatically to the column headings supplied, as long as they conform to some predefined format, maybe based on Gedcom tag names.

Accommodating a variety of Facts (BIRT, MARR, OCCU, CENS, DEAT, etc) is relatively easy.

Supporting various methods for specifying family relationships is trickier if you want to handle both the one row per person with identity numbers technique used by Build Tree From CSV File, and the named relations technique used by CWCG Build from CSV and Import CSV.

[Shelley]

Here's another work in progress version of the CSV Import plugin.

Major changes from the previous version posted are:

• Front end that makes it easier to use, including...
• ... the ability to map columns to fields in the plugin, instead of doing it in the CSV file.
• Accepts individual age at the time of a family event (eg age at marriage).
• Can optionally use custom reference numbers to identify the same person across lines.

I want to tidy up and make some wording changes, but I think it's all working as I intended.

Comments welcome.

[Mike]

Hello - I like your plug in and have used it for BMD import to FH - I have a spreadsheet of census data for a one name study I’m doing but can’t figure out how to tell FH the year of the census - any suggestions gratefully received thanks.

[tatewise]

Welcome to the FHUG Mike.

There is only one GEDCOM/FH standard Census (CENS) fact tag, so multiple instances of Census events are differentiated by the Date (CENS.DATE) field. e.g. UK 1891 Census is 5 Apr 1891 and UK 1901 Census is 31 Mar 1901.

Presumably, when adding Birth, Marriage & Death events to FH they are each accompanied by a Date.
The process for adding Census events is essentially the same, and just like any facts you can add more than one per Individual.

[Mike]

Mike - thank you

I think this is what I now need to do
Create an extra column in my S'sheet, headed CENS.DATE
Populate the rows for this column with the date of the census - for example 31 Mar 1901

Mike