* Splitting a Gedcom file prior to reassembly for TNG?

Questions regarding use of any Version of Family Historian. Please ensure you have set your Version of Family Historian in your Profile. If your question fits in one of these subject-specific sub-forums, please ask it there.
Post Reply
avatar
rfj1001
Superstar
Posts: 291
Joined: 14 Dec 2003 18:11
Family Historian: V7
Location: Dorset, England
Contact:

Splitting a Gedcom file prior to reassembly for TNG?

Post by rfj1001 »

My FH gecom file is 30+MB in size and I want to upload it onto my website (ourfamilyhistory.org.uk) for it to be imported into the website software (TNG) using its php scripts.

There are limitations with my ISP on processing a file above 25MB so I was thinking of splitting my gedcom in 2 using FH and then uploading the two parts separately onto my website.

Any advice on how to do it so I end up with an integrated data set online with no duplicates would be appreciated.

Also, is there a better way to get around the ISP limit,

Many thanks.

Robin
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by tatewise »

Have you tried contacting your ISP?
A Google search suggests that there often is a 'hidden' adjustable size limit parameter.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5465
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by ColeValleyGirl »

Is your Gedcom file that big, or your whole project including media?
avatar
rfj1001
Superstar
Posts: 291
Joined: 14 Dec 2003 18:11
Family Historian: V7
Location: Dorset, England
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by rfj1001 »

Tx Mike/Helen,

/MIke./ Yes I spent 90 minutes on the phone with the ISP (Tech Support IONOS). The outcome was that, apparently, there is a 256MB limit file size on their page.ini script - a limitation of the server package I'm on and I'm set at the max.. It is the processing of my gedcom file that is going over the limit. They tried doing the import at their end but were getting the same error messages so I know there are no errors at this end.

/Helen/ . Yes it is the gedcom file size (11000 records); images are a couple of GB more.

I can't be very much over the max file size as everything worked a few weeks ago and I can't have added more than a hundred new records.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by tatewise »

Robin, are the figures in your postings in error somewhere?
Your original posting mentioned 30+MB and 25MB but now you say the limit is 256MB ? :?

From a different perspective, does TNG allow imported GEDCOM to be merged into an existing tree?
The plugins:help:export_gedcom_file:tng_the_next_generation#import_export_gedcom_import|> Import/Export > GEDCOM Import screenshot suggests various options, but I don't use TNG, so it needs a TNG expert to advise.

If a merge is possible, then exporting a paternal; branch and a maternal branch should be workable.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Jane
Site Admin
Posts: 8508
Joined: 01 Nov 2002 15:00
Family Historian: V7
Location: Somerset, England
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by Jane »

Ah the joy's of One and One, I remember why I have always avoided them, so it's a memory limit you are hitting when importing the file rather than an upload issue.

Something like

Fatal error: Allowed memory size of x bytes exhausted (tried to allocate x bytes) in /path/to/php/script

I assume you mean the php.ini has that as a maximum in memory_limit. You can check for yourself if you run phpinfo() from a php script or in TNG go to Setup then Diagnostics and click the phpinfo link

You would be best talking to TNG to see if they have a recommended work around for the problem.
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by tatewise »

After a bit of digging around, here are some ideas.

First, check your GEDCOM is being exported in UTF8 format via the Export Gedcom File Plugin.
That format can be half the size of UTF16 format, and is necessary for TNG anyway.

Compress the .ged file into a ZIP file, which probably will make it small enough to upload.
The https://tng.lythgoes.net/wiki/index.php ... port_Data. wiki page says under FTP your GEDCOM to your hosting site: "Starting with TNG 11, you can import zip files."
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
paultt
Famous
Posts: 116
Joined: 18 Jan 2005 21:59
Family Historian: V7
Location: Hampshire, England
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by paultt »

I have used FH and TNG for many years, but do my updates in a different way, which may give some pointers. TNG must have the 8bit Gedcom and not the 16bit, so make sure that you are saving or exporting the 8bit flavour.
I run xampp 64bit(php 7.2) on my Windows 10 64 bit pc, and firstly import my gedcom into TNG 12.1 running under the xampp.
My gedcom file contains approx 164000 individuals, over 50000 families, over 7000 links to media items, etc. Current 8bit gedcom size is 72,567kb on disc and that imports with no problems, replacing all, not appending or merging.
Once having checked that there were no errors, I then do a TNG Utilities backup to get the zip .bak files. I then ftp these .bak files up to the server on 1&1 and do the TNG 12.1 Utilites restore. The largest .bak files that I ftp are tng_people.bak at 50,548kb ; tng_citations at 27951kb.
I use to get timeouts when doing the restores, but I called 1and1 and it was resolved by them upgrading the memory_limit and upload_max_filesize to 64mb in the php.ini file on their server. I cannot remember what the values for the memory limit were (currently shows -1) ? , as it was a long time ago, but could possibly check.
User avatar
tatewise
Megastar
Posts: 28341
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Splitting a Gedcom file prior to reassembly for TNG?

Post by tatewise »

Another idea has occurred to me.
In the Export Gedcom File Plugin try using the (-) Brief Data mode instead of (+) Full Data mode.
That omits some redundant data that is probably never used in your TNG database such as Named List details.
It might just let it scrape under the limits.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply