* Splitting a Gedcom file prior to reassembly for TNG?
-
- Superstar
- Posts: 291
- Joined: 14 Dec 2003 18:11
- Family Historian: V7
- Location: Dorset, England
- Contact:
Splitting a Gedcom file prior to reassembly for TNG?
My FH gecom file is 30+MB in size and I want to upload it onto my website (ourfamilyhistory.org.uk) for it to be imported into the website software (TNG) using its php scripts.
There are limitations with my ISP on processing a file above 25MB so I was thinking of splitting my gedcom in 2 using FH and then uploading the two parts separately onto my website.
Any advice on how to do it so I end up with an integrated data set online with no duplicates would be appreciated.
Also, is there a better way to get around the ISP limit,
Many thanks.
Robin
There are limitations with my ISP on processing a file above 25MB so I was thinking of splitting my gedcom in 2 using FH and then uploading the two parts separately onto my website.
Any advice on how to do it so I end up with an integrated data set online with no duplicates would be appreciated.
Also, is there a better way to get around the ISP limit,
Many thanks.
Robin
- tatewise
- Megastar
- Posts: 28410
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
Have you tried contacting your ISP?
A Google search suggests that there often is a 'hidden' adjustable size limit parameter.
A Google search suggests that there often is a 'hidden' adjustable size limit parameter.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
- ColeValleyGirl
- Megastar
- Posts: 5499
- Joined: 28 Dec 2005 22:02
- Family Historian: V7
- Location: Cirencester, Gloucestershire
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
Is your Gedcom file that big, or your whole project including media?
Helen Wright
ColeValleyGirl's family history
ColeValleyGirl's family history
-
- Superstar
- Posts: 291
- Joined: 14 Dec 2003 18:11
- Family Historian: V7
- Location: Dorset, England
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
Tx Mike/Helen,
/MIke./ Yes I spent 90 minutes on the phone with the ISP (Tech Support IONOS). The outcome was that, apparently, there is a 256MB limit file size on their page.ini script - a limitation of the server package I'm on and I'm set at the max.. It is the processing of my gedcom file that is going over the limit. They tried doing the import at their end but were getting the same error messages so I know there are no errors at this end.
/Helen/ . Yes it is the gedcom file size (11000 records); images are a couple of GB more.
I can't be very much over the max file size as everything worked a few weeks ago and I can't have added more than a hundred new records.
/MIke./ Yes I spent 90 minutes on the phone with the ISP (Tech Support IONOS). The outcome was that, apparently, there is a 256MB limit file size on their page.ini script - a limitation of the server package I'm on and I'm set at the max.. It is the processing of my gedcom file that is going over the limit. They tried doing the import at their end but were getting the same error messages so I know there are no errors at this end.
/Helen/ . Yes it is the gedcom file size (11000 records); images are a couple of GB more.
I can't be very much over the max file size as everything worked a few weeks ago and I can't have added more than a hundred new records.
- tatewise
- Megastar
- Posts: 28410
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
Robin, are the figures in your postings in error somewhere?
Your original posting mentioned 30+MB and 25MB but now you say the limit is 256MB ?
From a different perspective, does TNG allow imported GEDCOM to be merged into an existing tree?
The plugins:help:export_gedcom_file:tng_the_next_generation#import_export_gedcom_import|> Import/Export > GEDCOM Import screenshot suggests various options, but I don't use TNG, so it needs a TNG expert to advise.
If a merge is possible, then exporting a paternal; branch and a maternal branch should be workable.
Your original posting mentioned 30+MB and 25MB but now you say the limit is 256MB ?
From a different perspective, does TNG allow imported GEDCOM to be merged into an existing tree?
The plugins:help:export_gedcom_file:tng_the_next_generation#import_export_gedcom_import|> Import/Export > GEDCOM Import screenshot suggests various options, but I don't use TNG, so it needs a TNG expert to advise.
If a merge is possible, then exporting a paternal; branch and a maternal branch should be workable.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
- Jane
- Site Admin
- Posts: 8514
- Joined: 01 Nov 2002 15:00
- Family Historian: V7
- Location: Somerset, England
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
Ah the joy's of One and One, I remember why I have always avoided them, so it's a memory limit you are hitting when importing the file rather than an upload issue.
Something like
Fatal error: Allowed memory size of x bytes exhausted (tried to allocate x bytes) in /path/to/php/script
I assume you mean the php.ini has that as a maximum in memory_limit. You can check for yourself if you run phpinfo() from a php script or in TNG go to Setup then Diagnostics and click the phpinfo link
You would be best talking to TNG to see if they have a recommended work around for the problem.
Something like
Fatal error: Allowed memory size of x bytes exhausted (tried to allocate x bytes) in /path/to/php/script
I assume you mean the php.ini has that as a maximum in memory_limit. You can check for yourself if you run phpinfo() from a php script or in TNG go to Setup then Diagnostics and click the phpinfo link
You would be best talking to TNG to see if they have a recommended work around for the problem.
Jane
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."
My Family History : My Photography "Knowledge is knowing that a tomato is a fruit. Wisdom is not putting it in a fruit salad."
- tatewise
- Megastar
- Posts: 28410
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
After a bit of digging around, here are some ideas.
First, check your GEDCOM is being exported in UTF8 format via the Export Gedcom File Plugin.
That format can be half the size of UTF16 format, and is necessary for TNG anyway.
Compress the .ged file into a ZIP file, which probably will make it small enough to upload.
The https://tng.lythgoes.net/wiki/index.php ... port_Data. wiki page says under FTP your GEDCOM to your hosting site: "Starting with TNG 11, you can import zip files."
First, check your GEDCOM is being exported in UTF8 format via the Export Gedcom File Plugin.
That format can be half the size of UTF16 format, and is necessary for TNG anyway.
Compress the .ged file into a ZIP file, which probably will make it small enough to upload.
The https://tng.lythgoes.net/wiki/index.php ... port_Data. wiki page says under FTP your GEDCOM to your hosting site: "Starting with TNG 11, you can import zip files."
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
- paultt
- Famous
- Posts: 116
- Joined: 18 Jan 2005 21:59
- Family Historian: V7
- Location: Hampshire, England
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
I have used FH and TNG for many years, but do my updates in a different way, which may give some pointers. TNG must have the 8bit Gedcom and not the 16bit, so make sure that you are saving or exporting the 8bit flavour.
I run xampp 64bit(php 7.2) on my Windows 10 64 bit pc, and firstly import my gedcom into TNG 12.1 running under the xampp.
My gedcom file contains approx 164000 individuals, over 50000 families, over 7000 links to media items, etc. Current 8bit gedcom size is 72,567kb on disc and that imports with no problems, replacing all, not appending or merging.
Once having checked that there were no errors, I then do a TNG Utilities backup to get the zip .bak files. I then ftp these .bak files up to the server on 1&1 and do the TNG 12.1 Utilites restore. The largest .bak files that I ftp are tng_people.bak at 50,548kb ; tng_citations at 27951kb.
I use to get timeouts when doing the restores, but I called 1and1 and it was resolved by them upgrading the memory_limit and upload_max_filesize to 64mb in the php.ini file on their server. I cannot remember what the values for the memory limit were (currently shows -1) ? , as it was a long time ago, but could possibly check.
I run xampp 64bit(php 7.2) on my Windows 10 64 bit pc, and firstly import my gedcom into TNG 12.1 running under the xampp.
My gedcom file contains approx 164000 individuals, over 50000 families, over 7000 links to media items, etc. Current 8bit gedcom size is 72,567kb on disc and that imports with no problems, replacing all, not appending or merging.
Once having checked that there were no errors, I then do a TNG Utilities backup to get the zip .bak files. I then ftp these .bak files up to the server on 1&1 and do the TNG 12.1 Utilites restore. The largest .bak files that I ftp are tng_people.bak at 50,548kb ; tng_citations at 27951kb.
I use to get timeouts when doing the restores, but I called 1and1 and it was resolved by them upgrading the memory_limit and upload_max_filesize to 64mb in the php.ini file on their server. I cannot remember what the values for the memory limit were (currently shows -1) ? , as it was a long time ago, but could possibly check.
- tatewise
- Megastar
- Posts: 28410
- Joined: 25 May 2010 11:00
- Family Historian: V7
- Location: Torbay, Devon, UK
- Contact:
Re: Splitting a Gedcom file prior to reassembly for TNG?
Another idea has occurred to me.
In the Export Gedcom File Plugin try using the (-) Brief Data mode instead of (+) Full Data mode.
That omits some redundant data that is probably never used in your TNG database such as Named List details.
It might just let it scrape under the limits.
In the Export Gedcom File Plugin try using the (-) Brief Data mode instead of (+) Full Data mode.
That omits some redundant data that is probably never used in your TNG database such as Named List details.
It might just let it scrape under the limits.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry