* [Wish List 581] DOCX and/or ODT Report Generation Capability (was Hyperlinked footnote references in reports)

[Gary_G]

Writing programs that one might use for producing a final report often can recognize hyperlinked references to footnotes. Scrivener is an example of one such program. It is used by quite a few people to write their family stories based on the output from a genealogical program.

Having already included "Heading" format tags, Family Historian is halfway to a perfect pairing with such a program. Adding hyperlinked references to footnotes would make Family Historian the "go to" program for managing ones front-end research, because there would be a clear path for final processing and publish of the FH output reports into a variety of formats.

[ColeValleyGirl]

This would be a useful feature to have.

[fhtess65]

I agree - it definitely would be very useful.

This would be a useful feature to have.

[ColeValleyGirl]

Gary, what output formats do ypu envisage this being implemented for?

[ColeValleyGirl]

Might this be satisfied by a markdown output option?

[Gary_G]

While Markdown is accepted by Scrivener, I'm not sure that it would work as well as MS DOCX or ODT format. I'm pretty sure that Scrivener parses the DOCX input to organize the sections in accordance with the standard Header 1... Header X definitions of the imported document and will also extract any tagged references and footnotes. I believe it does the same for ODT, as the ODT contains the same formatting info. I don't believe that the same is done for Markdown, because of the method of tagging that is used to indicate formats. The Markdown standard is also not as "tightly" controled and it may be difficult to keep Scrivener and FH7 "in sync". I notice that FH7 can currently save a report in RTF format, so I will also have to see what Scrivener does when importing that format.

So; I will have to make a test files in the four formats to verify which results in the best support.

Consider that if the RTF engine code is already present in FH7 AND it will work for importing all the key formatting and footnote and hyperlink elements into Scrivener, then CP may find that more to their liking. Also; DOCX, ODT and RTF are supported by most word-processors, so one of those formats could make the output more widely usable.

As you can see, the question of output format requires a bit of investigation and discussion. We'll see...

[ColeValleyGirl]

Thanks, Gary.

From memory (and I'd need to track this down) here's something odd about the way that RTF handles footnotes which means it doesn't work as well as other options, but let's see what your experiment shows.

[Gary_G]

Helen;

Your memory is in line with my own. RTF was not really viable. Markdown was also not usable, because the importing of structure was keyed to MS-default header tags. Tag-names were VERY important, as RM9 has a habit of using arbitrary tag-names for some of the structure.

When trying the import test for RM9 reports, I remember that I had to use the available RM9 DOCX report (no other was available at the time), convert it to ODT, standardize the header and footnote tag-names, then import it. Once I did that, the document structure and embedded references worked as they should. The most important thing is that Scrivener wanted the MS-default header and footnote tagnames used and somehow was more happy with the ODT version to be able to import the references. Straight DOCX didn't work so well.

[ColeValleyGirl]

Gary,

Just musing, so please don't take this as an opinion set-in-stone.

Do we need to separate a 'requirement for hyperlinked footnote references' from a requirement for 'hyperlinked footnote references that work with Scrivener'?

In addition, we already have related wish list entries in the Reports category for:

Export to Markdown

Save Report As options for extra file types (DOCX, ODT, etc)

Hyperlinks in Reports and Books (this doesn't mention footnotes)

Do any of these provide the functionality required? (You've already said Markdown won't work for you, but that ODT would).

If so, it might be best to add a vote to the existing item, and a comment stressing the importance of hyperlinked footnote references.

Alternatively, we can raise an additional item similar to Hyperlinks in Reports and Books and note the other related wish list entries.

What do you think?

[Gary_G]

Helen;

I think that the ue of the word, "hyperlinks", may be causing some confusion. The capability that allows one to go back-and-forth between a footnote number in document text and the footnote itself is implemented in a particular way in an MSWord document. This bi-directional linking behaviour has to be implemented just as in MSWord AND use the default MSWord tag-names in order to be interpreted properly during an export to Scrivener. This functionality is not quite the same as a typical "hyperlink" to a website, although it appears similar on the surface.

Now; this does not mean that one cannot export from FH7 to Scrivener without the noted footnote capability, provided that the exported document uses the default MSWord tag-names. Those tag-names are what Scrivener uses to auto-split the document into sections, sub-sections and footnotes. However; I believe that the footnotes may not work quite as expected, if they have not been implemented correctly in the source document.

As for true hyperlinks; they should be able to be embedded anywhere within the body of the document text AND within footnote text.. This would be very useful to those who use valid URLs within their document and/or citation text to provide quick access to an online document.

As you can see; these are two distinct things and need to be addressed separately.

[ColeValleyGirl]

Gary,

Yes, worth clarifying that. However, if

The capability that allows one to go back-and-forth between a footnote number in document text and the footnote itself

is the crux of this request, is it covered by the existing wish for Save Report As options for extra file types (DOCX, ODT, etc), or covered by it with a note that this capability in particular is required?

[Gary_G]

Helen;

is the crux of this request, is it covered by the existing wish for Save Report As options for extra file types (DOCX, ODT, etc), or covered by it with a note that this capability in particular is required?

My explanation was specifically intended to explain why one can't separate the two capabilities and have an export that fully works with Scrivener. Attempting to split this functionality, in an effort to reduce the number of requests, will result in an export functionality that does not fully work as it should.

[Gary_G]

Helen;

A competitor to FH has already tried to export DOCX reports without putting in place the required tags and pointers within the app itself and then "faithfully" reproducing the structure and content of the MS DOCX, as defined in the Microsoft definition for the MSWord apartment model. So; they are still having issues with their reports and they really aren't usable with all that many 3rd party word-processors.

The DOCX file (and therefor the ODT analogue) needs to be set up "just so" and while the MSWord app can correct some issues on-the-fly, most 3rd party programs can't and rely on the file being fully valid. This means that the overall project is much more than just simply writing a report generator. I'd rather see this item set up as a totally separate wish item and hope that CP tackles it as a totally separate project and in doing so gets everything working together correctly. Rolling it out in pieces will undoubtably yield the same result as for their competitor.

In summary; maybe we need to extract all the word-processor-related capabilities that might affect this report-generation project into one main wishlist item, rather than split them out.

[ColeValleyGirl]

Gary, I know you probably don't want to take on the challenge of generating the Wish List item, but I don't currently have the time or headspace.

[Gary_G]

Helen;

If you can give me a "scaffold" structure for what CP likes to see for wishlist items (Just titles is enough), then I'll have a go at it.

How do you feel about naming it. "DOCX Report Export Capability"?
In the course of the writeup, I'd explain that having a valid DOCX output is key and that the user should be able to convert to ODT for programs that require that format and cannot import it.
I think Markdown would be a whole different project.

[AdrianBruce]

Gary - if you look at viewtopic.php?p=143557#p143557, you can see the format I used, inspired partly by (receding) memories of requirements work in my professional life. Just pick out the headings from that post - and be aware that I tend to write long, rather than short.

[ColeValleyGirl]

I think this might be a valid case for updating an existing wish: Save Report As options for extra file types (DOCX, ODT, etc), improving the structure and adding extra information to support and clarify the request. I don't think the approach Gary has outlined would invalidate any of the existing votes. Does anyone disagree?

[Gary_G]

Helen;

I looked at that the thread for the noted wish. It basically says that the OP wanted to be able to add pictures and edit tables in narrative reports and, as such, would like to have DOCX output from FH. [Hope I've paraphrased it correctly].

It's unusually short for a wishlist thread and I can see nothing that I might add that might invalidate what was said.

[ColeValleyGirl]

Gary, let's proceed on that basis then. As you say, there was no discussion in the original thread.

[Gary_G]

Gary - if you look at viewtopic.php?p=143557#p143557, you can see the format I used, inspired partly by (receding) memories of requirements work in my professional life. Just pick out the headings from that post - and be aware that I tend to write long, rather than short.

Adian;
If Helen feels that this is generally a format that CP has favoured in past, then I could try using it. Each section appears to give enough text to understand the intended content.

Helen;
Might I suggest that, at some point, a bit of a suggested "skeleton" format be posted in the wishlist forum to help users to provide suitable content and detail to kick off a wishlist thread?

[ColeValleyGirl]

Helen;
Might I suggest that, at some point, a bit of a suggested "skeleton" format be posted in the wishlist forum to help users to provide suitable content and detail to kick off a wishlist thread?

Wish List guidance and rules (22417) exists already. Please suggest any improvements there.

[Gary_G]

Helen;

Should the revision be posted in this thread or in the thread for the existing wish you suggested?

Shaping the preceding discussion in this thread into the format Adrian suggested as a structure may take a day or so, because I have a few more "domestic" tasks to be done for "she who must be obeyed" and time is at a premium. Hope that's OK.

[ColeValleyGirl]

Post the revision in this thread; when it's agreed, I'll use it to update the existing Wish List item.

No problem that you're prioritizing the right things

[Gary_G]

Here is my first draft of the proposed wishlist item.
[I hope I've not made any glaring spelling or grammar errors. :>) ]

------

Title
“DOCX and/or ODT Report Generation Capability”

Summary of Proposal
To facilitate the import and post-processing of Family Historian reports, the current output-format selection should be augmented to include DOCX and/or ODT.

Background
The final format and general appearance of a report is as much an artistic endeavour as it is a scholarly one. Individual expectations of what it can contain and how it is formatted is very much a personal preference. This may be as simple as including images or as complex as major structural changes and typesetting. Some programs supply extensive configuration parameters in an effort to meet user expectations. However; no autogenerated report can meet the needs of everyone and some form of post-processing by other applications is typically required.

Issue
As noted; end-users may need to post-process reports. To date, Family Historian has a limited set of standard output formats available; “PDF”, “Web Page (HTML)”, “Word-Processor Document (RTF)” and “Text File”. Of these; only the latter two are formats that are intended to be edited. The standard RTF documents support formatting in a limited sense, but lack any ability to incorporate graphics. Text documents can be edited, but provide even less formatting capability and cannot handle graphics or tables. Currently; even the more capable RTF files can barely produce output that closely resembles the appearance and content of the on-screen report. There are other standard formats, such as DOCX and ODT that can be used to generate output that closely mimics the appearance of on-screen reports. These two formats are also standard input formats for many word-processors and desktop publishers; the tools that are typically used for post-processing. In addition; should the report content of FH evolve to include graphics, then neither RTF nor TXT will be suitable. It should also be noted that; while Markdown is also a commonly supported input for desktop publishers, it is far less standardized and its use therefore comes with higher compatibility risks. In short; users would see significant current and future benefits from the addition of an DOCX and/or ODT export capability.

Proposal
It is suggested that FH7 be augmented to provide the ability to export reports in standard DOCX and/or ODT format.

Constraints
The following key requirements stem from observing ongoing difficulties encountered by FH competitors in implementing a similar capability.

• To ensure reliability and compatibility with the largest number of word-processors and desktop publishers, the DOCX and/or ODT file must adhere to the respective published standards for the file format/content.
  [Note: Some programs, like Scrivener, may depend upon footnotes being implemented per the standard. This may require some care, since MSWord text-to-footnote linking works bidirectionally.]

• The implementation must use the standard MS default parameters, especially with respect to format tag-names tag-names.
  [Note: Some programs, like Scrivener, depend upon the presence of default tag-names to mimic the document structure and correctly extract references upon import.]

• Any optional customization and deprecated capabilities must be avoided in order to ensure the highest level of compatibility with third-party post-processors.

• As the changes could easily result in regression of existing capabilities, it is recommended that the product be thoroughly regression-tested. The resulting output files must also be verified to be compatibile with commonly-used post-processors such as MSWord, LibreOffice and Scrivener (amongst others).

Scope of Changes
While, on the surface, the requirement is simply to produce reports in standard DOCX and/or ODT format, the necessary programming changes are likely to go well beyond the code responsible for generating reports. The target formats will likely require changes to other areas of the FH code to provide the required content to the report generator code.

This request has consistently used the phrase, “DOCX and/or ODT”, but it is hoped that both can be offered. While many post-processing applications accept both formats, some like Scrivener, seem to work best with ODT. However; if only one format is possible, then external converters between DOCX and ODT do exist.

[AdrianBruce]

Do we need to add a requirement that a report saved in DOCX / ODT / whatever should look like the native FH Report? Currently, the PDF preserves the positioning of images when compared to the native FH Report, but the images in the RTF bear no relationship to their positioning in the native FH Report.

The Issue section currently says

The standard RTF documents support formatting in a limited sense, but lack any ability to incorporate graphics

My .JPG media do appear in the .RTF documents - just in the wrong place entirely. Or am I missing something?