* Auto import of image metadata

Questions regarding use of any Version of Family Historian. Please ensure you have set your Version of Family Historian in your Profile. If your question fits in one of these subject-specific sub-forums, please ask it there.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Auto import of image metadata

Post by tatewise »

Gordon, yes I forgot Safe Mode is tricky in Windows 8/8.1 so ignore that for the moment.

BUT the advice about INSTALLING FH V6.04 using right-click Run as administrator with your Sophos anti-virus temporarily disabled is worth a try. The Knowledge Base advice is there because other users have found it works. Sophos and FH will not necessarily raise messages during the INSTALL process, but simply mishandle the installation. Note that I am talking about INSTALLATION not just running FH.

However, having searched Google, there does seem to be some 'instability' in the EXIF/IPTC standards, and Jim may have the explanation of why the metadata is imported unreliably.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

Thanks Mike. I'm sorry I missed your original note about running the installation in administrator mode. Since I've already installed FH V6.0.4 in normal mode how do I go back to run that installation in administrator mode?

I can't just run it again surely, presumably I would need to revert back to the base FH version of 6.0.0 first - I still have the 6.0.0 upgrade install file for that (I was on 5.0.9 before), do I just run that upgrade install file again or will that mess things up? This is beginning to sound a bit messy...?
Gordon Harrison - Researching Harrison and McKernan family history.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Auto import of image metadata

Post by tatewise »

No it is not messy, as you say, just install the FH V6.0.4 download again using Run as administrator, preferably with Sophos temporarily off.

However, the fact that the metedata is partially working suggests it is more likely an image data format problem.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

Ok Mike, I have now run the install again to FH v6.0.4 this time in administrator mode and nothing untoward happened. Sophos was switched off while I did that and have now switched it back on again. FH has opened up and so I'll now continue with the tests.

At this stage the tests indicate it is looking like there is a difference between Photoshop and Windows as regards the format in which metadata is stored. Photoshop can read Windows metadata but my tests imply that as soon as you use Photoshop to amend the metadata it reformats all the metadata to IPTC standards. If you don't try and amend the metadata it seems to leave the metadata as is.

The difference between Windows and Photoshop metadata format is what you suggested Jim. However I'm pretty sure there is no difference between Mac and Windows implementation of the IPTC/EXIF standards which is what Photoshop adheres to. So Mac or Windows based photographers can happily exchange images and read or update each others metadata as long as it was created by Photoshop.

I plan to run some more experiments on this to make sure I fully understand what's going on (if I can!) and will hopefully report back here tomorrow. It does seem that there is a solution to my problem of getting metadata automatically imported to FH yet still available to Photoshop which is what I was hoping to achieve, albeit now with a different solution from the one I envisaged. Fingers crossed! Thanks to everyone for chipping in on this, much appreciated.
Gordon Harrison - Researching Harrison and McKernan family history.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Auto import of image metadata

Post by tatewise »

It might a good idea to report your findings to Calico Pie by reference to this thread.
They may be able to investigate why FH internally handles Explorer differently to PhotoShop.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
Keithj
Platinum
Posts: 39
Joined: 09 Jul 2015 00:33
Family Historian: V6
Location: Witnesham, Suffolk
Contact:

Re: Auto import of image metadata

Post by Keithj »

I've not been doing anything with EXIF data in FH6, but I have found elsewhere that looking at the "source" code in the file may reveal what is causing a problem.

If you look at the source of the EXIF data in the photographs, can you see any obvious differences between Windows and Photoshop versions? I had some wacky behaviour with .SHTML files that came down to one particular editor replacing commas with semicolons.

Knowing that, the modification needed for FH6 may be easier.

Just a thought.
Keith Jillings - researching Jillings and the many variants.
User avatar
AdrianBruce
Megastar
Posts: 2090
Joined: 09 Aug 2003 21:02
Family Historian: V7
Location: South Cheshire
Contact:

Re: Auto import of image metadata

Post by AdrianBruce »

I have not tried getting FH to see keywords but....

I'm using XnView as a browser - it shows separate tabs for EXIF and IPTC metadata, implying they are different kettles of fish.

I added a tag in Windows Explorer - and in Explorer it shows up in Tags under Property / Description.

I added two tags in Photoshop Elements to a different file - and in Explorer they show up in Tags under Property / Description - just the same.

Now looking in XnView - the tag added in Explorer shows up as XPKeywords on the EXIF tab. The tags added in Photoshop Elements are not visible on the EXIF tab but are visible on the IPTC tab as Keywords.

What I can't see how to do is update the Explorer added tags in PSE.

So it looks like Explorer is reading both EXIF and IPTC data and putting it in the same place on the Properties display. Thus giving the incorrect impression it's in the same place.

So could the explanation be that FH is reading the EXIF metadata but not the IPTC data, while Windows Explorer is confusing us by not only reading both but showing both in the same place?
Adrian
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Auto import of image metadata

Post by Valkrider »

Adrian

That sounds highly likely to me. They are definitely not the same as you have found.

For my exif editing and viewing I always use PhotoMe which is the most powerful editor that I have found.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Auto import of image metadata

Post by tatewise »

After much searching I found this online thread date 2010:
https://productforums.google.com/forum/ ... LU3yUcZa5M
It seems to be discussing a very similar problem with Picasa and Windows Photo Gallery, and much confusion over IPTC-IIM, IPTC Core, XMP, EXIF, etc, and specifically mentions XPKeywords.

This 2014 thread says similar things about EXIF & IPTC:
http://twigstechtips.blogspot.co.uk/201 ... -from.html
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

After playing around with countless jpegs and various way of entering/editing metadata it seems to me that Calico Pie have chosen to import metadata based on the format chosen by Microsoft to record such data. Entering metadata via Windows Explorer always results in a successful import by FH of the required metadata.

I suspect that Microsoft's implementation of metadata within its OS is not according to currently agreed standards. Don't ask me to prove this bold statement, that's for others with far more expertise than I to do! Adobe are members of the metadata working group and their imaging products are used by professionals and libraries around the world to record metadata in millions of images daily. If Adobe's software, such as Photoshop, Bridge, Lightroom or its Creative Suite were doing something wrong concerning something as crucial as metadata it would be headline news in the imaging world which I'm part of.

If I record the metadata with Photoshop (or Adobe Bridge) then FH does not see it (because I assume its looking elsewhere) and so cannot import it. If I open in Photoshop a jpeg in which the metadata was recorded by Windows Explorer then Photoshop can see that metadata. I assume that's because that Adobe have made allowance for the way that Microsoft have recorded the metadata. However, if I add to or change that metadata with Photoshop and then save the file Family Historian is no longer able to see the metadata. I assume this is because as soon as any attempt is made to edit metadata recorded by Windows Explorer then Photoshop reformats all the metadata according to agreed metadata standards.

All my tests to date support the thoughts I've expressed above. For example, if I take a jpeg in which I recorded the metadata with Photoshop (i.e. metadata which FH cannot see), then use Photoshop to erase all the metadata, then save the file, then use Windows Explorer to record the metadata in that saved file, FH can see the metadata again when I import it. Anyway, for better or worse, that's my theory. Hopefully when I draw Calico Pie's attention to this their experts will home in on the exact problem and fix it. Clearly, whatever the problem, images created/edited in the pre-eminent image editing suite created by Adobe ought not to present difficulties when other software such as FH wants to extract metadata from them.

Therefore, for images destined for Family Historian I will create/edit them in Photoshop as normal but will not touch the metadata fields in Photoshop! I shall subsequently add metadata with Windows Explorer then import the image to Family Historian. Having done that I will make a minor inconsequential change to the metadata (with Photoshop) in the image file so that Photoshop makes the metadata conform to standards for all other subsequent uses. It's not a 100% perfect solution but it is a great deal better than copying and pasting metadata from one program to another - which is what I've been doing until I decided recently to try and automate the entire process.
Gordon Harrison - Researching Harrison and McKernan family history.
User avatar
Valkrider
Megastar
Posts: 1563
Joined: 04 Jun 2012 19:03
Family Historian: V7
Location: Lincolnshire
Contact:

Re: Auto import of image metadata

Post by Valkrider »

Gordon

Give Photome a try it is free and you may prefer it to Windows Explorer > Properties.
User avatar
jimlad68
Megastar
Posts: 921
Joined: 18 May 2014 21:01
Family Historian: V7
Location: Sheffield, Yorkshire, UK (but from Lancashire)
Contact:

Re: Auto import of image metadata

Post by jimlad68 »

FH might use "external" software for image manipulation and to get at the data, if so, knowing that might "be of interest", if not Calico Pie might let you know exactly what fields it is looking for and there is a possibility that Photoshop could be made to populate those fields instead or as well as. Again, I think that would be in the realms of a Photoshop forum.

I would have thought there were some batch programs that might convert the data, indeed Photoshop might be able to.

However, if you are not dealing with lossless formats (png, tiff), every time you save, you may lose some quality, so make sure you keep your originals.
Jim Orrell - researching: see - but probably out of date https://gw.geneanet.org/jimlad68
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

Valkrider wrote:Gordon

Give Photome a try it is free and you may prefer it to Windows Explorer > Properties.
Thanks Colin, I'll look at that. Although I have used Windows Explorer for the purposes of the tests (rather than introduce yet another variable to the equation!) I normally use Directory Opus instead of Windows Explorer. I will have a look at Photome too.
Gordon Harrison - Researching Harrison and McKernan family history.
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

jimlad68 wrote:FH might use "external" software for image manipulation and to get at the data, if so, knowing that might "be of interest", if not Calico Pie might let you know exactly what fields it is looking for and there is a possibility that Photoshop could be made to populate those fields instead or as well as. Again, I think that would be in the realms of a Photoshop forum.

I would have thought there were some batch programs that might convert the data, indeed Photoshop might be able to.

However, if you are not dealing with lossless formats (png, tiff), every time you save, you may lose some quality, so make sure you keep your originals.
Thanks Jim. Although I'm creating .jpegs (from .tiffs) for FH I only save them once. I can change the metadata in .jpegs to whatever I want via Adobe Bridge (part of Photoshop) without opening the .jpeg itself.

At the moment I'll stick where I am rather than try and persuade Adobe to review/change Photoshop's metadata formats. Even if Adobe agreed, which is probably unlikely if they are adhering to the agreed standard for metadata, it would take quite some time as it would need to be agreed with other members of the Metadata Working Group (Canon, Sony, etc.) as well as take into account the needs of OS's other than Windows. I have a solution now which is about 90% good so I'll stick with that and explain the current issue to Calico Pie. I think it is more likely that Calico Pie could adjust FH to recognise the metadata standards as well as the format used by Microsoft and populate the metadata in FH from one or the other format depending on what they find.

For anyone who is interested guidelines for handling metadata are published by the working group at http://www.metadataworkinggroup.org/pdf ... idance.pdf Towards the end of that lengthy document are links to metadata specifications for the various types of metadata as well as specifications of the various file formats, .jpeg, .tiff and .psd. Light bedtime reading - not! :)
Gordon Harrison - Researching Harrison and McKernan family history.
User avatar
AdrianBruce
Megastar
Posts: 2090
Joined: 09 Aug 2003 21:02
Family Historian: V7
Location: South Cheshire
Contact:

Re: Auto import of image metadata

Post by AdrianBruce »

Having thought through what we've found and glanced at the linked PDF about metadata, I have to say that I personally don't believe that Microsoft is doing it wrong and Adobe is doing it right. In any case, whether or not you agree, I think we can produce a more specific request for Calico Pie.

Firstly, EXIF metadata and IPTC metadata are stored separately in the .JPG images. (NB - It seems to me that .PNG has no ability to store IPTC).

So far as I can see, the IPTC metadata is much richer in what I might term non-technical data - I use it myself to store data about where I get images from. The concept of keywords does seem to appear in both EXIF and IPTC metadata.

Should the keywords in EXIF and IPTC be kept in step? The linked PDF seems to say that it should but it is very specific in its vocabulary and does not say must.

If I compare Microsoft Windows Explorer with what seems to be happening with Adobe, then:
- MS reads both EXIF and IPTC keyword metadata;
- Adobe reads both EXIF and IPTC keyword metadata;
- MS writes only to EXIF keyword metadata;
- Adobe writes only to IPTC keyword metadata;

Both products should write to both EXIF and IPTC keyword metadata - instead, both write only to one. However, as this requirement is a should and not a must, both products are compliant.

I think the simple summary is that, as far as keywords go, MS have majored on using the EXIF format and Adobe have majored on using the IPTC format. Adobe's choice is no doubt quite logical given the (I presume) superior nature of the IPTC metadata format for workflow data. But MS is not using "their" interpretation - it's using EXIF.

That being so, I think a better phrasing of any request to Calico Pie would be - Could they please confirm that they are using only the EXIF metadata and could they please enhance FH to access IPTC metada as well? (How would they reconcile the case if both EXIF and IPTC keywords are supplied? Hmmm)
Adrian
User avatar
mjashby
Megastar
Posts: 719
Joined: 23 Oct 2004 10:45
Family Historian: V7
Location: Yorkshire

Re: Auto import of image metadata

Post by mjashby »

Unfortunately, I'm coming to this thread a bit late in the day, particularly as GordonH has already come across the key difference(s) between Photo Imaging metadata as employed in Photoshop (and other image capture/editing/management software); and Windows File Metadata (Tagging), as employed by Family Historian which allows for tagging and organising a much wider range of file formats, including Office Documents, PDFs, sound, video files etc. In the FH Help File, Calico Pie only describes the the recognition of 4 Fields of file metadata that are revealed when you look at Windows 'File Properties/Details' using Windows Explorer/File Manager (or any of the other Windows File Managers), so we probably shouldn't be expecting the software to 'see' anything that does not appear in those 4 fields.

As discovered, FH uses 4 specific Fields from the Windows File Properties, which were created by Microsoft primarily as an aid to searching and filtering a much broader range of Windows file formats. Also, as speculated earlier, Mac OS X (Unix/Linux/FreeBSD ) also has a similar, but not identical, file tagging method. My question here is not why doesn't FH read image metadata but: "Why don't Photoshop developers; and developers of other Image manipulation software respect the OS systems they rely on and provide their users with a straightforward method of recording those well established and well documented Windows/OS X file tags?" Which, I suppose, is bit like the Better GEDCOM debate: "Yes, we all agree their should be a standard for data transfer, providing we can have the freedom to select which bits of the standard we don't like and don't need to comply with!"

So, my take would be: Yes, it's probably inconvenient for photographers that Family Historian doesn't directly read the carefully applied metadata (tags) in photographic images, but to change that method, simply because Image manipulation software doesn't allow for key events of Windows File metadata to be recorded, may mean either:

a) losing the ability to tag and organise (in a compatible way) many other file types that are widely used by Family Historian users; or
b) developing a much more complex method of reading and blending together multiple metadata recording systems.

Mervyn
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

Many thanks Adrian and Mervyn for your thoughts on this. Yes Adrian, I accept your point that it may be that Microsoft are not doing anything wrong so I'll not express the issue in that way.

Probably the easiest solution to this issue is that if IPTC metadata is present FH should use that, else use the EXIF data.

I've done a number of tests creating metadata with Explorer, creating metadata with Photoshop/Bridge, creating images with MS Paint and Adobe Photoshop, some with metadata, some without, then importing these images to FH and these tests indicate that the above change to FH would work in all cases. However, its up to Calico Pie how they want to proceed and I will only suggest this as a possible solution.

Do you have any thoughts on this way forward before I contact Calico Pie?
Gordon Harrison - Researching Harrison and McKernan family history.
User avatar
AdrianBruce
Megastar
Posts: 2090
Joined: 09 Aug 2003 21:02
Family Historian: V7
Location: South Cheshire
Contact:

Re: Auto import of image metadata

Post by AdrianBruce »

Gordon - I'd buy into your suggestion.

That PDF doesn't make it clear to me (which may be my fault!) what to do if keywords appear in both EXIF and IPTC metadata, though it does go through some of the other items that appear in both. I think there was a tendency to say, use IPTC in preference to EXIF - which is what you suggest.
Adrian
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

The more I dig into the more I discover.

The attached image shows how the various EXIF metadata fields are currently used by Family Historian to populate the FH media record with metadata - see highlighted fields in red.

It also shows how EXIF and IPTC fields are populated with metadata depending on whether data is entered via Windows Explorer or via Photoshop.

One surprise resulting from this analysis is that Windows Explorer actually does populate the IPTC fields as well as the EXIF fields. Photoshop populates only the IPTC fields with relevant data with the exception of the Description field which it also adds to the set of EXIF fields.

This analysis would lead one to the conclusion that FH could simply always look at IPTC for the data it needs. There is however (as might be expected!) a snag.

For some reason Windows Explorer duplicates the Title in the IPTC fields placing it in both the Object Name and Caption-Abstract fields. My reading of the metadata specs referred to earlier in this thread suggests this may be an error. My reading is that the specs suggest that the Caption Abstract field should contain the description and when you look at what Photoshop does it does place the description in there.

Therefore the proposed solution of using IPTC data if present else EXIF would not always lead to the correct result. You could end up in FH with the title being duplicated in both the Title and Note fields of the Media Record.

So, the latest idea is to use such EXIF fields as are present, and where not to use the equivalent IPTC field. Alternatively I just explain the issue to Calico Pie and let them figure out what to do.

Apologies for the length of this thread, I had not intended it to turn out like this. Originally I thought I was just going to complain about a bug in FH!
Attachments
Table of Metadata Equivalences
Table of Metadata Equivalences
MetaDataEquivalences.jpg (77 KiB) Viewed 14830 times
Gordon Harrison - Researching Harrison and McKernan family history.
User avatar
AdrianBruce
Megastar
Posts: 2090
Joined: 09 Aug 2003 21:02
Family Historian: V7
Location: South Cheshire
Contact:

Re: Auto import of image metadata

Post by AdrianBruce »

Hmmm - I wonder if preferring EXIF to IPTC, where both are populated, is necessarily the right way to go? I just feel that the vast majority of people with metadata in image files, will have put it there using specialist images editors and not Windows Explorer. And the specialist editors are more likely to use IPTC items.

Two wholly unsupported assertions there, of course!
Adrian
User avatar
davidm_uk
Megastar
Posts: 740
Joined: 20 Mar 2004 12:33
Family Historian: V7
Location: St Albans, Hertfordshire, UK

Re: Auto import of image metadata

Post by davidm_uk »

I suspect that the"vast majority" of people will just have the metadata recorded by their camera, or what they downloaded from a web site, or produced on a scanner, maybe with a bit of cropping from an image editor.

I suspect that the people into editing metadata (or even knowing about it) are in the minority - but I'm not suggesting that it's not important :)
David Miller - researching Miller, Hare, Walker, Bright (mostly Herts, Beds, Dorset and London)
User avatar
jimlad68
Megastar
Posts: 921
Joined: 18 May 2014 21:01
Family Historian: V7
Location: Sheffield, Yorkshire, UK (but from Lancashire)
Contact:

Re: Auto import of image metadata

Post by jimlad68 »

perhaps an option for one or the other or both

very interesting thread - "better a bad/outdated but accepted standard, than a myriad of good ones", mmm, that reminds me of GEDCOM.
Jim Orrell - researching: see - but probably out of date https://gw.geneanet.org/jimlad68
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

AdrianBruce wrote: I just feel that the vast majority of people with metadata in image files, will have put it there using specialist images editors and not Windows Explorer. And the specialist editors are more likely to use IPTC items.
I agree Adrian. However, I would imagine that Calico Pie would be reluctant to cast adrift users who have been happily using Windows Explorer and then subsequently have to deal with much grumpiness from them! Hmmmm. I wonder how many people on FHUG use Windows Explorer to enter metadata?

Of course we could just dump the issue in Calico Pie's lap and let them fret over it - but it would be nice to be able to propose a possible way of keeping everyone happy. I'll leave it to stew for a couple of days and see if anyone has any other ideas....
Gordon Harrison - Researching Harrison and McKernan family history.
avatar
GordonH
Diamond
Posts: 84
Joined: 27 Jan 2010 13:12
Family Historian: V6
Location: Contin, Scotland
Contact:

Re: Auto import of image metadata

Post by GordonH »

jimlad68 wrote:perhaps an option for one or the other or both
Sounds like a good idea to me Jim! Let FH users tell FH how they are entering metadata, i.e. tell FH in Media Preferences to use EXIF or IPTC, user's choice, if one way doesn't work for them they need to choose the other option. Is that what you meant?
Gordon Harrison - Researching Harrison and McKernan family history.
User avatar
tatewise
Megastar
Posts: 28333
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: Auto import of image metadata

Post by tatewise »

An even more specific method is for FH to provide a table of popular EXIF and IPTC metadata tags mapped to Media fields.
Then the user can choose which EXIF/IPTC metadata transfers to which Media fields.

There may need to be some precedence settings needed where several tags can transfer to the same Media field.

Interestingly, the FH Help page on the subject uses metadata tag names that don't entirely match any names in Gordon's attachment.

So there needs to be some Help on the possible tag names that may appear in different utilities.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
Post Reply