* [Wish List Item 625] 'Free Form' Citation Entry capability

For Wish List Requests that have either (a) been progressed to the Wish List; or (b) been classified as duplicates, or as redundant because the requirement is already satisfied within FH and/or plugins; or (c) closed because it wasn't possible to arrive at a clear specification of the request within 15 months of it being raised.
avatar
Gary_G
Megastar
Posts: 888
Joined: 24 Mar 2023 19:05
Family Historian: V7

Re: 'Free Form' Citation Entry capability

Post by Gary_G »

ColeValleyGirl;

The question of whether Free Form citations would meet the needs of lumpers is an odd one, since by definition each is unique. That is; they are implicitly splitter-style citations. Given the diverse opinions on what constitutes lumping/splitting, I'd need to see an example of where you feel such citations could be, or would ever be, lumped.

Actually; as noted before, altering the behaviour of existing citation-entry methods would most definitely cause issues for DEAs and Ancestral Sources developers. Both of these application-groups tend to be used with some of the more common formats of citations that would benefit from bulk data-entry. Free Form citations, as previously noted, are best suited to one-off citations. So; there is likely very little potential for overlap. As such; it makes perfect sense to not try to change the existing input methods, but add a third method. GEDCOM exports should be able to handle exporting and importing the basic three citation elements, just as they always have.

With respect to enable metafield token substitution in generic templates...
As I've already noted, I think that there would be a fair amount of work (and retesting) resulting from that approach. Existing, already-working code really should not be touched. I should also note that a user would then not be able to simply cut-and-paste text from notes etc. and would have to hand enter the tokens to match what they wanted to see.
Gary Gauthier
Hunting History in the Wild!
User avatar
Mark1834
Megastar
Posts: 2599
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: 'Free Form' Citation Entry capability

Post by Mark1834 »

ColeValleyGirl wrote: 08 Nov 2023 15:41 Extra metafields on existing sources will, I think be less disruptive, but the authors of Ancestral Sources and the export plugins are both in this conversation so will have a view.
I'm agnostic on this one. The Ancestry Sync plugin doesn't include sources - export is straightforward enough at present, but managing syncing across three different apps would be "challenging", as they all store the data slightly differently, and is peripheral to the plugin's core purpose.
Mark Draper
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

Gary_G wrote: 08 Nov 2023 16:14 ColeValleyGirl;

The question of whether Free Form citations would meet the needs of lumpers is an odd one, since by definition each is unique. That is; they are implicitly splitter-style citations. Given the diverse opinions on what constitutes lumping/splitting, I'd need to see an example of where you feel such citations could be, or would ever be, lumped.
My 'burned parish registers' is an example. I have several facts (baptism, marriage, burial, residence, occupation, parentage etc.) which rely on that source. I could hand-craft a citation for each fact as I created them manually (or copy an instance of the hand-crafted citation and edit it to include the details specific to each fact). But what if I changed my mind about the structure of the citations because I realised I'd got something egregiously wrong? Or just that there was a better way of doing it? That's an awful lot of individual sources to edit... about 20, IIRC.

Although I'm by large a splitter, in this instance I'd much rather create a single (lumped) source, with some citation metafields to identify the entry of interest, and Freeform citation details that substituted those citation metafield values where I required. I could then re-vamp my freeform details in a single place.
With respect to enable metafield token substitution in generic templates...
Not sure what you mean by 'Generic templates'?

If you're referring to Generic sources, they don't have metafields, so there'd be no work involved in ignoring any that were specified. :) And passing tokens through from the Source Definition unsubstituted might be useful to indicate how a Freeform Citation for a Generic Source should be updated to be specific to a particular fact.

And metafields are all optional in Templated Sources, so the same arguments apply.
As I've already noted, I think that there would be a fair amount of work (and retesting) resulting from that approach. Existing, already-working code really should not be touched.
I think we have to rely on CP to worry about their own development practices. If we don't want them to touch their existing code base we should wave goodbye to the Wish List and any future releases.
I should also note that a user would then not be able to simply cut-and-paste text from notes etc. and would have to hand enter the tokens to match what they wanted to see.
I very pedantically and exactly said:
to enable metafield token substitution (if present) in the freeform details
No tokens? No substitution. Copy-pasta to your hearts content.
Actually; as noted before, altering the behaviour of existing citation-entry methods would most definitely cause issues for DEAs and Ancestral Sources developers. Given that both of these groups do not handle all types of citations, but rather tend to be used with some of the more common citations.
DEAs don't have to be restricted to a subset of source types; I have written some that handle both Generic sources and a subset of Templated Sources. And I have one under development that is completely agnostic on Templated Source type, as well as handling Generic sources.

More importantly, DEAs rely wholly on FH to create the Source/Citations, so adding the Freeform capability to the existing Source Types is not an issue at all, whereas introducing a new Source Type (now I think about it) is just an irritant.

Nick Walker will have to comment on how much AS would be affected.
Free Form citations, as previously noted, are best suited to the unique types of citations. As such; it makes perfect sense to not try to change the existing input methods, but add a third method.
I explained my disagreement a few paras back.
GEDCOM exports should sell be able to handle exporting and importing the basic three citation elements, just as they always have.
I'll leave this to Mike T and Mark D as I'm not sure how they derive the Citation elements currently.
User avatar
tatewise
Megastar
Posts: 28712
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: 'Free Form' Citation Entry capability

Post by tatewise »

Compared to the other 'complexities' in the Export Gedcom File plugin, I doubt if any Free Form Citation solution would be a significant problem.
What would be very helpful is a function in FH that returned a formatted Footnote/Bibliography/Short Footnote for a chosen Source Citation. Then it would not matter what type of Source was involved as FH would resolve all that.
The footnote, short footnote, and bibliography text (22439) thread was heading to a Wish List Request for just such a function but it has been locked.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

tatewise wrote: 08 Nov 2023 18:59 What would be very helpful is a function in FH that returned a formatted Footnote/Bibliography/Short Footnote for a chosen Source Citation. Then it would not matter what type of Source was involved as FH would resolve all that.
ColeValleyGirl wrote: 08 Nov 2023 14:47 CP could even use the hidden fields if they wished to hold the formatted Footnotes etc. for all Sources and Citations, which would help make them more easily available to plugin authors.
The footnote, short footnote, and bibliography text (22439) thread was heading to a Wish List Request for just such a function but it has been locked.
Nothing to stop anyone actually raising a Wish List Request and referencing that topic, whether it's closed or not. Go ahead and raise it Mike.
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

Returning to the original request, what about:

Proposal: The ability to create and cite a Source that has manually entered or pasted rich-text Bibliography, Footnote and Short Footnotes, including metafield substitution at source or citation level if metafield tokens are included.

Background and Benefits: One or more genealogy programs already permit the user to manually enter, or cut-and-paste, rich-format text to define the Bibliography, Footnote and Short Footnote; this is often referred to as a Freeform Citation.

Having this capability in FH7 would be particularly useful for users of Evidence Explained style, due to the unpredictable structure of the citations and the importance of formatting in that methodology. It also could facilitate the direct import of free-form citations from other programs such as RootsMagic.

Allowing metafield substitution would provide extra flexibility.
avatar
Gary_G
Megastar
Posts: 888
Joined: 24 Mar 2023 19:05
Family Historian: V7

Re: 'Free Form' Citation Entry capability

Post by Gary_G »

ColeValleyGirl;

Yes. Your proposed wording is quite reasonable and embodies user requirements, not design constraints.
I'd vote for it as worded.
Gary Gauthier
Hunting History in the Wild!
User avatar
NickWalker
Megastar
Posts: 2668
Joined: 02 Jan 2004 17:39
Family Historian: V7
Location: Lancashire, UK
Contact:

Re: 'Free Form' Citation Entry capability

Post by NickWalker »

Was just posting this and then spotted Helen has now added some proposed wording for the wishlist item while I was writing this - might as well post it anyway...

I think this is what Helen has been suggesting, but I would have thought the easiest solution (looking at this as a programmer) would be for a 'free form' source to simply be a templated source without any fields (which I guess is already possible), coupled with the ability to be able to manually edit the footnotes and bibliography in a similar way that titles can be manually edited currently. A lumper version could have citation fields specified as required. I think this kind of thing would have the potential to be considered for development as it is just a further development of the existing mechanisms, whereas a more complex entirely separate type of source with its own mechanisms would be more complex and probably unlikely to get beyond the wish-list. I don't think I'd have too much trouble adapting AS to allow the footnotes and bibliography to be entered manually.
Nick Walker
Ancestral Sources Developer

https://fhug.org.uk/kb/kb-article/ancestral-sources/
User avatar
NickWalker
Megastar
Posts: 2668
Joined: 02 Jan 2004 17:39
Family Historian: V7
Location: Lancashire, UK
Contact:

Re: 'Free Form' Citation Entry capability

Post by NickWalker »

ColeValleyGirl wrote: 08 Nov 2023 20:05 Returning to the original request, what about:

Proposal: The ability to ...
Yes this looks like a good proposal to me.
Nick Walker
Ancestral Sources Developer

https://fhug.org.uk/kb/kb-article/ancestral-sources/
User avatar
Mark1834
Megastar
Posts: 2599
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: 'Free Form' Citation Entry capability

Post by Mark1834 »

Looks good with one caveat - I wouldn’t use the term ‘tokens’ in the Proposal. You’ve slipped into IT-speak, so it may not be clear to general readers what you mean.
Mark Draper
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

Mark1834 wrote: 08 Nov 2023 22:02 I wouldn’t use the term ‘tokens’ in the Proposal
Good catch -- thanks, Mark. Does metafields pass muster either, and can you suggest alternatives? My brain is doing that thing where it's stuck on particular words :roll:

I shall add some verbiage to illustrate why simple templates won't suffice, for the benefit of people coming to it on the Wish List without seeing this discussion first.

And I shall strip out the other proposals into their own topic, to make things clearer for Calico Pie when they come here via the Wish List.
User avatar
Mark1834
Megastar
Posts: 2599
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: 'Free Form' Citation Entry capability

Post by Mark1834 »

I think meta fields is only used in the plugin help, so won't mean anything to most users. The normal FH UI seems to just call them "fields", so I would stick with that, or a simple qualification such as "data field" or "source data field", "citation data field" etc, according to context. As long as it is clear, the simpler the better.

It's probably fair to assume it is only users with a working knowledge of templates who will vote on this one, so we can use the language that describes template properties and their definitions.
Mark Draper
User avatar
NickWalker
Megastar
Posts: 2668
Joined: 02 Jan 2004 17:39
Family Historian: V7
Location: Lancashire, UK
Contact:

Re: 'Free Form' Citation Entry capability

Post by NickWalker »

I'd call them source template data fields rather than meta fields. And 'source template field references' may be more meaningful than meta field tokens.
Nick Walker
Ancestral Sources Developer

https://fhug.org.uk/kb/kb-article/ancestral-sources/
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

Another attempt:

-------------------------------------------

Proposal: The ability to create and cite a Source that has manually entered or pasted rich-text Bibliography, Footnote and Short Footnotes, including source template field substitution at source or citation level if field references are included.

Background and Benefits: One or more genealogy programs already permit the user to manually enter, or cut-and-paste, rich-format text to define the Bibliography, Footnote and Short Footnote; this is often referred to as a Freeform Citation.

Having this capability in FH7 would be particularly useful for users of Evidence Explained style, due to the unpredictable structure of the citations and the importance of formatting in that methodology. It also could facilitate the direct import of free-form citations from other programs such as RootsMagic.

Allowing source template field substitution (especially at citation level) would provide extra flexibility to make the feature useful for both splitters and lumpers.
User avatar
tatewise
Megastar
Posts: 28712
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: 'Free Form' Citation Entry capability

Post by tatewise »

FYI:
The Help page Understanding Data References has a section near the end named DATA REFERENCES AND METAFIELDS that says:
Source records linked to Source Template records can have fields which are defined within the Source Template - see Sources and Source Templates. Fields of this kind are called metafields. Citations to Source records that are linked to Source Template records can also have metafields. All metafields have the same tag: '_FIELD'. You can refer to the first metafield within a source record like this: %SOUR._FIELD%. Metafields can have an index like any other field, so you can refer to the second metafield like this: %SOUR._FIELD[2]%. Data references for metafields within citations, if used within a Footnote or Short Footnote format (Source Template format), require the contextual data reference 'CUR~CITN'. So, for example, you can use %CUR~CITN._FIELD% to refer to the first metafield in a citation.
... and so on.

The Help page Source Template Formats has a section near the end named How to Use Data References or Functions within Formats that says:
The table below shows examples of data references that could be used in formats, and the contexts in which they would be valid. Notice that two examples (the second and fourth) use shortcuts to metafields. To learn more about metafields, and how you can reference them using data reference shortcuts, see the section Data References and Metafields in Understanding Data References.
... the following table uses metafield several times.

An FH Help Search for metafield will find those two Help pages.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

tatewise wrote: 09 Nov 2023 10:35 FYI:
The Help page Understanding Data References has a section near the end named DATA REFERENCES AND METAFIELDS
Yes, but that's in the 'Here be dragons' section otherwise known as Advanced Topics and most ordinary users never venture there. I think it's better to stick with terminology used in the main help sections on Sources and Citations.
User avatar
Mark1834
Megastar
Posts: 2599
Joined: 27 Oct 2017 19:33
Family Historian: V7
Location: South Cheshire, UK

Re: 'Free Form' Citation Entry capability

Post by Mark1834 »

Happy with Helen’s wording. If we want general users to participate in the Wish List process, it has to be written in general user language, not expert technobabble.
Mark Draper
avatar
Woodg
Famous
Posts: 135
Joined: 08 Oct 2019 09:28
Family Historian: V7
Location: Orange, Australia

Re: 'Free Form' Citation Entry capability

Post by Woodg »

Mark1834 wrote: 09 Nov 2023 11:07 Happy with Helen’s wording. If we want general users to participate in the Wish List process, it has to be written in general user language, not expert technobabble.
Absolutely!
User avatar
tatewise
Megastar
Posts: 28712
Joined: 25 May 2010 11:00
Family Historian: V7
Location: Torbay, Devon, UK
Contact:

Re: 'Free Form' Citation Entry capability

Post by tatewise »

ColeValleyGirl wrote: 09 Nov 2023 10:52 Yes, but that's in the 'Here be dragons' section otherwise known as Advanced Topics and most ordinary users never venture there. I think it's better to stick with terminology used in the main help sections on Sources and Citations.
FYI: The Help page Source Template Formats which mentions metafields is not in the Advanced Topics section.

I was just correcting the misunderstanding that meta fields is only used in the plugin help.

I'm happy to go with the lastest wording.
Mike Tate ~ researching the Tate and Scott family history ~ tatewise ancestry
avatar
Gary_G
Megastar
Posts: 888
Joined: 24 Mar 2023 19:05
Family Historian: V7

Re: 'Free Form' Citation Entry capability

Post by Gary_G »

Sometimes the trail of comments gets me a bit turned around.
I believe that CoalValleyGirl's last proposal was;
Proposal: The ability to create and cite a Source that has manually entered or pasted rich-text Bibliography, Footnote and Short Footnotes, including source template field substitution at source or citation level if field references are included.

Background and Benefits: One or more genealogy programs already permit the user to manually enter, or cut-and-paste, rich-format text to define the Bibliography, Footnote and Short Footnote; this is often referred to as a Freeform Citation.

Having this capability in FH7 would be particularly useful for users of Evidence Explained style, due to the unpredictable structure of the citations and the importance of formatting in that methodology. It also could facilitate the direct import of free-form citations from other programs such as RootsMagic.

Allowing source template field substitution (especially at citation level) would provide extra flexibility to make the feature useful for both splitters and lumpers.
I trust that is the "last wording" to which you were referring and everyone is happy with that.
Gary Gauthier
Hunting History in the Wild!
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

I'll leave this for a week so that anyone late to the party can raise objections or suggest improvements.
User avatar
AdrianBruce
Megastar
Posts: 2158
Joined: 09 Aug 2003 21:02
Family Historian: V7
Location: South Cheshire
Contact:

Re: 'Free Form' Citation Entry capability

Post by AdrianBruce »

... particularly useful for users of Evidence Explained style, due to the unpredictable structure of the citations and the importance of formatting in that methodology ...
My only thought is that for someone with my previously restricted understanding of EE styles, they might fall into the same trap that I did and be "unable to compute" the idea of EE citations being unpredictable. How about expanding
the unpredictable structure of the citations
to
the unpredictable structure of the citations where the researcher follows the overall principles of Evidence Explained for source types not listed in that work
This suggestion relates only to clarifying the justification. (Not totally sure where to put the bit about "the importance of formatting")
Adrian
User avatar
fhtess65
Megastar
Posts: 695
Joined: 15 Feb 2018 21:34
Family Historian: V7
Location: British Columbia, Canada
Contact:

Re: 'Free Form' Citation Entry capability

Post by fhtess65 »

I'm certainly happy with it. While generally the templates I create and use (either in AS or FH) work for me, but occasionally, the ability to free form in the manner described would be very helpful.
ColeValleyGirl wrote: 10 Nov 2023 09:32 I'll leave this for a week so that anyone late to the party can raise objections or suggest improvements.
---
Teresa Basińska Eckford
Librarian & family historian
http://writingmypast.wordpress.com
Researching: Spong, Ferdinando, Taylor, Lawley, Sinkins, Montgomery; Basiński, Hilferding, Ratowski, Paszkiewicz
avatar
Gary_G
Megastar
Posts: 888
Joined: 24 Mar 2023 19:05
Family Historian: V7

Re: 'Free Form' Citation Entry capability

Post by Gary_G »

While trying to implement some templates in FH7 according to the "Source Template Format" rules. It occurred to me that simply pasting Rich-Text into the template definition or into the meta fields of a defined template might severley upset how templates are evaluated. How would FH7 know what to interpret as RTF and what to interpret according to the source template rules? Requiring escaping of any offending RTF characters would prevent the correct interpretation of RTF text. Failure to do so might impact the application of the source template format rules. In addition; editing RTF text would prevent simply cut-and-pasting of the text. I wonder if the difficulty trying to address the simultaneous application of two sets of text-formatting "rules" was why other platforms developed Free-Form citation as a separate feature.
Gary Gauthier
Hunting History in the Wild!
User avatar
ColeValleyGirl
Megastar
Posts: 5580
Joined: 28 Dec 2005 22:02
Family Historian: V7
Location: Cirencester, Gloucestershire
Contact:

Re: 'Free Form' Citation Entry capability

Post by ColeValleyGirl »

Gary_G wrote: 14 Nov 2023 13:36 While trying to implement some templates in FH7 according to the "Source Template Format" rules. It occurred to me that simply pasting Rich-Text into the template definition or into the meta fields of a defined template might severley upset how templates are evaluated. How would FH7 know what to interpret as RTF and what to interpret according to the source template rules?
Mark D and I have said from the start that pasting rich text into a metafield is fraught with problems and therefore not a good idea. Any Freeform Citation capability should only allow pasting into the footnotes, bibliography etc.
Locked