* [Wish List Item 625] 'Free Form' Citation Entry capability

[Gary_G]

While trying to implement some templates in FH7 according to the "Source Template Format" rules. It occurred to me that simply pasting Rich-Text into the template definition or into the meta fields of a defined template might severley upset how templates are evaluated. How would FH7 know what to interpret as RTF and what to interpret according to the source template rules?

Mark D and I have said from the start that pasting rich text into a metafield is fraught with problems and therefore not a good idea. Any Freeform Citation capability should only allow pasting into the footnotes, bibliography etc.

Even pasting into just the footnotes, bibliography etc. and not within meta fields would have the same effect, since the content of the whole field is evaluated according to the "Source Template Format" rules.

[ColeValleyGirl]

Even pasting into just the footnotes, bibliography etc. and not within meta fields would have the same effect, since the content of the whole field is evaluated according to the "Source Template Format" rules.

Only if the rules don't change, or have an exception added.

[Gary_G]

I decided to see exactly how FH7 imports an RM9 Free-Form citation.

As-imported from RM9 Free-Form template data

Screenshot 2023-11-14 at 9.25.24 AM.png (108.19 KiB) Viewed 433 times

As-imported from RM9 Free-Form source template definition

Screenshot 2023-11-14 at 9.13.25 AM.png (93.48 KiB) Viewed 433 times

It seems that FH7 treats the Bibliography, Footnote and Short Footnote as meta fields. Because meta fields don't currently permit styles within the field itself, such content appears to be stripped of any style info. The current proposal would retain this behaviour. It seems that this behaviour will present a problem with importing Free-Form sources from apps like RM9 via the direct import mechanism of FH7.

[Gary_G]

Even pasting into just the footnotes, bibliography etc. and not within meta fields would have the same effect, since the content of the whole field is evaluated according to the "Source Template Format" rules.

Only if the rules don't change, or have an exception added.

Helen;

If an exception is needed, then that wording needs to be formulated and added to the proposed Wishlist entry.

As noted in one of my most recent posts, not being able to have rich-text in meta fields will likely be an issue for imports of Free-Form citations from apps like RM9.

In short; I think there needs to be a few "real-world" use cases worked through and the proposal updated before submission to the actual wish list.

[ColeValleyGirl]

Gary,

I think importing 'Free Form' citations for other product is something that will have to be looked at by CP for each product for which they have import code; we do need to include the ability to import 'Free Form citations' in the proposal.

Can you give me a worked example of how the Source Template Format rule might affect cutting-and-pasting in the Footnote, etc.? What things might not work?

[tatewise]

As well as the other Metafield Types (Text, Name, Place, URL, etc) could there also be a Format type?
Those Format Metafields would support the Template Format style codes.
The style codes would be preserved when that type of Metafield is referenced in a Template Format.
i.e. Footnote, Bibliography, Short Footnote Metafields would be defined with the Format type.

[Gary_G]

Helen;

One can't give a "worked" example when the current implementation doesn't accept any RTF input in a meta-field field. I have already posted images of what I imported from RM9 and it is clear that FH7 imports the content into meta-fields without any style data. We know FH7 applies the formatting rules when it expands the meta-fields. You've already stated that having RTF in meta-fields would be a problem.

As for importing being considered on a case-by-case basis...
What is hoped for is more than just the ability to enter brand "new" citations in FH7. FH7 already directly imports from RM9 (as an example) and just a few others. It's not a big set of cases. I doubt CP will entertain implementing an enhancement that might "break" the existing capability or raise the issue of why imports aren't handled fully.

[Gary_G]

As well as the other Metafield Types (Text, Name, Place, URL, etc) could there also be a Format type?
Those Format Metafields would support the Template Format style codes.
The style codes would be preserved when that type of Metafield is referenced in a Template Format.
i.e. Footnote, Bibliography, Short Footnote Metafields would be defined with the Format type.

Mike;
I was also thinking about this possibility. If CP could tell FH not to format the content of certain data-types, that might be a robust solution. It might not be called "free-form", but rather "user-defined". That would allow for the following possibility.

One could have RTF (or other) data inserted into meta fields via a user-defined function discussed in another wish-list item? I can see that if that Wishlist item were implemented, one could easily find oneself in a situation in which the plugin output results in text that needs to be ignored by the meta-field formatting routines.

The use of functions, in this way, might handle imports from various sources without "upsetting the apple-cart".

[tatewise]

I deliberately avoided the term Rich Text because the full set of Rich Text styles is not able to be supported.
Only the Template Format style codes in a single line of text would be feasible with maybe BOOK and TITLE styles.
i.e. Just the Format style codes which is why I called it a Format type of Metafield and not Rich Text.
Rich Text would imply tables, links, colours, bullets, and multiple lines would be supported.

[Gary_G]

I deliberately avoided the term Rich Text because the full set of Rich Text styles is not able to be supported.
Only the Template Format style codes in a single line of text would be feasible with maybe BOOK and TITLE styles.
i.e. Just the Format style codes which is why I called it a Format type of Metafield and not Rich Text.
Rich Text would imply tables, links, colours, bullets, and multiple lines would be supported.

Mike;
I understand what you mean, but RM9 doesn't understand links, colours, bullets etc as valid data in it's meta fields. The chance of any exotic character constructs is almost certainly zero. I suspect that other apps are much the same. FH7 has a much richer set of data-types. If you don't have it, I could try to get you a copy of its template language that defines what a field potentially contains.

[Gary_G]

Mike;

As previously posted, RM9 Free-Form citations are imported as a set of metafields; Footnote, Short Footnote and Bibliography.

Both the RM9 template definition text and metafield text can have the following formatting codes. Formatting codes allow you to tell RootsMagic how you want the font to look in the report. Formatting codes are contained inside of a left angle bracket "<" and right angle bracket ">". They are used in pairs - the first tells RootsMagic to turn on the formatting, and the second, with a "/" after the left angle bracket (</ >) to turn off the formatting. For example:

<i>[Title]</i>
The <i> and </i> formatting codes tell RootsMagic to display the title in italics font.

Field
Formatting Codes
<i> ... </i>
Fact sentence and Source templates, fields, and notes
Displays the text in italics

<b> ... </b>
Fact sentence and Source templates, fields, and notes
Displays the text in bold

<u> ... </u>
Fact sentence and Source templates, fields, and notes
Displays the text in underline

<sc> ... </sc>
Fact sentence and Source templates, fields, and notes
Displays the text in SMALL CAPS (upper case letters remain unchanged, lower case letters are changed to upper case, but in a smaller font)

^...
Fact sentence and Source templates, fields, and notes
Displays the text in superscript

_...
Fact sentence and Source templates, fields, and notes
Displays the text in subscript


These formatting codes are a similar set to those in FH7. In RM9 the metafield formatting codes are stored in the metafield data and the template formatting codes are stored in the template definition. As an example of the metafield code storage, here is a snippet from a test GEDCOM in which part of the Footnote text was italicized;

0 @S1@ SOUR
1 ABBR Free-Form Citation Source Name
1 REFN Source Reference Number
1 TITL Free-Form Citation Source Footnote
1 _SUBQ Free-Form Citation Source Short Footnote. [<i>This is italicized metafi
2 CONC eld text</i>].
1 _BIBL Free-Form Citation Source Bibliography
1 TEXT Source Text
1 NOTE Source Comment

When FH7 imports RM9 citations, I suspect that FH7 filters out the RM9 metafield data formatting codes, because FH7 does not currently permit them in a metafield.

Perhaps the solution is as simple as providing the list of formatting codes to CP and requesting that FH7 not filter those codes from the metafield data, but process the ones it can as it does elsewhere. [Note: I don't have some of the other programs from which FH7 can import data, so we need to look at their formatting code list as well and include any additional ones that FH7 can process.

[tatewise]

You have confirmed my suspicions that we are NOT dealing FH Rich Text so should not refer to it as such.
Some of the style codes you posted from RM9 happen to be the same as Rich Text style codes but also the same as Template Format style codes and also HTML style codes, i.e. <i> </i>, <b> </b>, <u> </u>.
Other RM9 style codes are NOT valid Rich Text or Template Format style codes and it is not clear how they should be handled, i.e. <sc> </sc> are similar to <cap> </cap> in FH, but   are not in FH at all.

As well as pasting such metafields from RM9, other products should be considered, and the possibility of entering those metafields by hand where it must be clearly documented what style codes are allowed. Any implication that Rich Text is allowed must be avoided. I say again, that is why I chose Format as the Type and avoided Rich Text.

[Gary_G]

Mike;

I don't know how you determined that I was trying to say the codes were RTF. I showed exactly what formatting codes were used by RM9 to illustrate that they were similar to FH7. I was not trying to imply that it was RTF, nor did I ever use that term in my last post.

I was also trying to show that RM9 uses only a slight superset of those used by FH7 with a specific differences in the way they are processed. For the common set of formatting codes, it appears that FH7 does not seem to show italics in RM9 imports, solely because it doesn't process even the common set of codes within a metafield. That seems an easily resolved issue.

In addition, I did explicitly note that other programs may have a different set of formatting codes and was clear that we need to understand what other programs use as well.

The key point is that the proposed Wishlist entry needs to be amended to explicitly state what set of formatting codes it is requesting be permitted within both the template definition and within metafields.

[tatewise]

Sorry Gary, certainly in the latest few postings you have used the term formatting codes, but earlier yesterday you (and others) often used the terms RTF and Rich Text metafields. I wanted to dispel the idea that metafields should support Rich Text when they need to be significantly different. Not only would they potentially support a different set of style formatting codes (as you have stated) but those codes might also be visible (as they are in Source Template Formats) whereas in Rich Text notes they are usually invisible.

[Gary_G]

Mike;
It is normal for a reasonable person's opinion on a topic to change over the course of a thread. So it's best to not to put too much weight on previous posts when newer ones seem to clearly indicate a change in opinion. That said; I do like the idea of periodically stating the current situation. It helps everyone to focus on the current version of the proposal.

I must admit that I'm torn between having visible and invisible formatting codes.
On the one hand, visible formatting codes are easy to edit, because I can see what needs to be added or deleted. It is also the way FH7 currently seems to show them in the source template edit windows (and elsewhere). On the other hand, the template edit fields currently only show two lines, so one might end up scrolling a bit to edit everything. If CP were to implement the existing Wishlist item about enlarging the template windows, it would certainly make that concern disappear.

Wrt. my data imports to FH7... It appears (as noted) that RM9 stores text containing the raw formatting codes, so FH7 imports should not be an issue. However; from what I can tell, any RM9 preview that one could cut-and-paste appears to be in a word processor format (unfortunately it's likely RTF). So; directly cutting-and-pasting would result in the need to remove the pasted codes and retag the text with the FH7-style formatting codes. All this would mean is that I must try to import as much as I can, because it will be less work. If I need direct cutting-and-pasting from RM9, I want to keep it to a minimum and it will just take a slight bit of extra effort. I'd love to hear from users of programs other than RM9 about what their programs use, so that everyone is aware of how it might affect them.

[ColeValleyGirl]

Sorry Gary, certainly in the latest few postings you have used the term formatting codes, but earlier yesterday you (and others) often used the terms RTF and Rich Text metafields. I wanted to dispel the idea that metafields should support Rich Text when they need to be significantly different. Not only would they potentially support a different set of style formatting codes (as you have stated) but those codes might also be visible (as they are in Source Template Formats) whereas in Rich Text notes they are usually invisible.

I want to dispel the idea that existing metafields should support formatting at all -- we are straying into implementation details.

I also want to dispel the assumption that any cutting and pasting would be done into the existing Footnote etc. format field -- that's another implementation detail. I could (if I wanted to suggest implementation) envisage a tick box next to each format field that said: 'Override' which displayed a different field (without the Format part of the label), and some indication that the 'standard' format has been overridden).

Getting back to basics requirements rather than implementation, what we want is the ability to cut and paste or otherwise enter formatted text that overrides any automatically created Footnote or Bibliography:

Proposal: The ability to create and cite a Source that has manually entered or pasted rich-text Bibliography, Footnote and Short Footnotes, including source template field substitution at source or citation level if field references are included.

My assumption when I wrote that was that Source Template Fields would remain as unformatted text, and any necessary formatting would be done in the 'freeform' Footnotes etc. using visible Source Template Style codes (building on users experience of the current UI, and avoiding any need to introduce a different 'wyswig' editor alongside the existing FH Rich Text editor.)

This means:

• manually creating a freeform footnote etc. will involve entering plain text, FH Template Format style codes, and Source Template Field references.
• a paste operation will transform already-formatted text (from whatever source it is derived from -- another programme, a website, a word processing document) into plain text and FH Template Format Style codes.
• importing a freeform footnote etc. will transform a set of style codes used in the source to the corresponding FH Template Format Style codes.

What happens if you try to import or paste unsupported formatting for footnote, bibliography etc? Realistically, I suggest CP can only discard the unsupported formatting, but I'd personally leave that to them to work through.

Do we need a list of additional formatting options that we would like to see supported for compatibility with other programmes/citation styles? We started discussing that here Additional Formatting Options for Bibliography, Footnotes etc. (22453). Or do we just assume that including import capability will bring any extra formats automatically?

=======================================================================

Draft reworded proposal:

Proposal: The ability to create and cite a Source that has manually entered or pasted formatted Bibliography, Footnote and Short Footnotes, including Source Template Field substitution at source or citation level if field references are included.

These Footnotes etc. should consist of plain text, FH Template Format style codes, and optional Source Template Field references for consistency with the current Footnotes etc. It would be desirable to allow a larger edit field for these Footnotes etc. as they can get quite complex.

It should also be possible to import such Footnotes etc. from RootsMagic and other import sources that support them.

Background and Benefits: One or more genealogy programs already permit the user to manually enter, or cut-and-paste, formatted text to define the Bibliography, Footnote and Short Footnote; this is often referred to as a Freeform Citation.

Having this capability in FH7 would be particularly useful for users of Evidence Explained style, due to the unpredictable structure of the citations and the importance of formatting in that methodology. It would also improve the direct import capability programs such as RootsMagic.

Allowing source template field substitution (especially at citation level) would provide extra flexibility to make the feature useful for both splitters and lumpers.

[Gary_G]

Helen;

Wrt. ...

I want to dispel the idea that existing metafields should support formatting at all -- we are straying into implementation details.

I definitely can't support this interpretation of free-form for the following reason.

FH7 currently imports RM9 free-form citations as exactly three key metafields per source and the imported data for these fields often contains formatting codes. Those codes are the exactly the same as the existing FH7 formatting codes (with the exception of RM9 supporting similar codes for super and subscripts). Currently FH7 strips all but plain-text on importing into the noted metafields. This results in all imported free-form citations being incorrectly formatted upon import into FH7. Your interpretation of the proposed wishlist entry would result in a user having to completely retype all imported free-form citations just to enter the text outside the related metafield and then tag it there. The ONLY way I can see your interpretation being acceptable is if CP altered the current import of free-form citations so as to place the text of the Bibliography, Footnote and Short Footnote directly into the associated citation element fields rather than in metafields that are inserted into these citation fields. This also presupposes that FH7 does not strip FH7-recognized formatting codes from the imported text.

Actually; I wonder if the issue of free-form templates would not be completely solved by CP altering the way it imports free-form data to reflect the alternative suggested above.

[Gary_G]

Mike;

Can you comment on whether the current template definition process requires at least one metafield per Bibliography, Footnote or Short Footnote? I seem to remember trying to see what I could do with just formatted text in the template elements (no metafields) and remember that the template instantiation didn't display any of the entered text. It seemed as if the upper-level formatting rules required at least one metafield to work. That could be an issue in implementing purely free-form templates.

[ColeValleyGirl]

Helen;

Wrt. ...

I want to dispel the idea that existing metafields should support formatting at all -- we are straying into implementation details.

I definitely can't support this interpretation of free-form for the following reason.

Gary, I think we're busy misunderstanding each other.

My expectation is that the existing Footnote etc. FORMAT metafields and Source Template Fields are left untouched, so that users with no interest in the feature can happily ignore it.

Freeform citations would most probably (if I second-guess CP) leverage the three metafields you refer to and -- by allowing them to include Template Style Format codes -- allow all the methods of input I described (one of which is an enhanced version of the import process which preserves or transforms the formatting codes).

When outputting the Footnote etc, FH would check for 'enabled' freeform metafields and override the default format if found.

However, this is all implementation, and my revised proposal does not reference it. There should definitely be no need for any rework post-import (unless imported styles are not recognised).

[tatewise]

Gary, it seems that Source Template Formats do not need any metafield references.
The only oddity is that any plain text colon (:) or period (.) gets followed by a space as shown below.
Enclosing URL in < angle brackets > is meant to eliminate the spaces but eliminates the entire text.

image.png (32.71 KiB) Viewed 235 times

[Gary_G]

Helen;

My point was that perhaps we already have free-form capability, but the import routine is removing valid codes from the imported data, because of the way the import is performed. So; perhaps we may be asking for the wrong thing.

The three meta-fields I referenced are not predefined by FH7. They are essentially whatever they would be called in the automatically created RM9 free-form template that was imported from RM9 (and would also be visible in the RM9 GEDCOM file). From what I understand from a test of tagging non-metadata fields in the FH7 citation formatting fields, one can already apply formatting codes to tag standalone text and any metafield. The only thing CP appears to need to do is NOT to first place the completely filtered imported citation data into metafields, but instead take the raw imported text + formatting codes and place that directly into the citation formatting fields (after having filtered any unrecognized FH-7 format codes). That would leave any standard formatting codes in place and not require ANY change (ie. no switches needed) in how the user currently handles things.

The only needed change seems to be in the way the import works, not a new capability to inject format codes. We already seem to have the latter.

[Gary_G]

Gary, it seems that Source Template Formats do not need any metafield references.
The only oddity is that any plain text colon (:) or period (.) gets followed by a space as shown below.
Enclosing URL in < angle brackets > is meant to eliminate the spaces but eliminates the entire text.

Thanks, Mike;
I think that if the import routine and the behaviour you noted were to be tweaked, the free-format issue might become irrelevant.

[ColeValleyGirl]

Gary, we lack the ability to copy and paste formatted text, as well as lacking the ability to preserve formatting on import. Both of the may need additional Style codes.

I can envisage somebody wanting to cut and paste formatted source information from e.g.Ancestry or Zotero as either the whole or part of e.g a Footnote.

We can - agreed - create formatted citations as complex as we wish manually without reference to source template fields.

[ColeValleyGirl]

PS I strongly disagree with including Style codes in all metafields. It mixes meaning and presentation and makes life unnecessarily hard for plugin authors. It also will potentially lead new users into cul de sacs with inconsistent styling and where undoing a stylistic decision is a lot of work.

Style codes should be reserved for specific metafields - the existing Footnote etc Format metafields and any additional metafields required to support Freeform citations, for example.

If we do suggest that importing and pasting should take place into the existing Footnote etc. Format metafields, there ought to be a machanism to rest them to 'default' values.

[Gary_G]

Thanks for bearing with me, Helen.

I "think" I now see what's causing the confusion. Your point appears to be that once the template is instantiated, FH7 treats input field content as literal text and does not currently evaluate any formatting codes in the field data. So; even if FH7 just placed validated imported data in the instantiated template's input fields, the text would still need to be run through the evaluation routine to correctly display a formatted citation. Of course, the same would be true for any manually entered or cut-and-pasted text. [And... FH7 still needs to make sure it doesn't strip out valid imported codes as well. ] Must admit, though, I still don't really see the need for any user "switch", but that is something CP can address.

The key for me was that your wording was intended to apply to the behaviour of an instantiated template, not a template definition.

Perhaps this distinction could be explicitly made in the wishlist request?