* Legacy backup plugin and Unicode

[Mark1834]

To clarify an issue that arose in the thread regarding testing the new Windows backup plugin, spun out to avoid confusing two separate issues -

I have tested the legacy backup plugin with my userid and FH7, and I agree that it appears to handle Unicode files correctly.

Tested in FH6 with my userid, it appears to omit copying a custom query with a Unicode name.

Tested in FH6 with a Unicode userid, it omits copying the Unicode query, and also crashes with an error.

What I haven't tested is the legacy plugin with a Unicode userid and FH7, as I do not have a spare FH7 licence for my disposable "play" system, and I'm not prepared to create new accounts on my live PC just to test FH plugins. I can’t test this feature in WINE, as Unix (and therefore Mac OS and Linux) does not support Unicode userids.

I must admit, I had read across between FH6 and FH7, as I did not anticipate such differences. If it is only an FH6 issue, it's not worth any time spent fixing, although I'm curious about what may have caused the difference.

[Mark1834]

I think I have found the smoking gun...

It was never going to be an FSO error, as much more than FH plugins would break if it didn't handle Unicode files correctly. The obvious next place to look is the luacom iterator that converts the FSO output to a Lua table.

This simple script selects a folder, and displays a list of all files in that folder as an FH Message Box (which avoids any issues trying to save it to file in FH6).

Code: Select all

require('luacom')
require('iuplua')
iup.SetGlobal('CUSTOMQUITMESSAGE', 'YES')
tblT = {}

filedlg = iup.filedlg{dialogtype = 'DIR', directory = fhGetContextInfo('CI_PROJECT_DATA_FOLDER'),
		title = 'Select Folder'}
filedlg:popup()
if filedlg.Status == '-1' then return end

FSO = luacom.CreateObject("Scripting.FileSystemObject")
Folder = FSO:GetFolder(filedlg.Value)
Files = luacom.GetEnumerator(Folder.Files)
File = Files:Next()
while File do
	table.insert(tblT, File.Name)
	File = Files:Next()
end
fhMessageBox(table.concat(tblT, '\n'), 'MB_OK', 'MB_ICONINFORMATION')

The first screen grab is part of the output in FH7 looking at my custom queries. The second grab is exactly the same plugin running from FH6, selecting exactly the same OneDrive folder

See how it mishandles the first file in the list, which has a Unicode name.

FH7.png (3.38 KiB) Viewed 692 times

FH6.png (3.7 KiB) Viewed 692 times

Hopefully, there is a way of breaking this Catch-22 - the main reason for using FSO directly is to enable FH6 to handle Unicode files, but the luacom iterator in FH6 does not appear to support Unicode correctly…

[tatewise]

My initial guess is that FH V6 Lua 5.1 is miscalculating the byte length of the filename due to the Unicode character which is actually 2 bytes long but Lua 5.1 is counting characters instead of bytes.

[Mark1834]

That makes sense. The not-so-good news is that if you use the same plugin to select a folder with a Unicode name, it crashes in both FH6 and FH7.

These symptoms are exactly consistent with what I observe in FH6. With a Unicode file, it just misreads the file name, so the plugin generates erroneous and confusing messages when comparing versions to check timestamps, but carries on without copying the file. For a user with a Unicode Windows name, it crashes with an error.

FH system folders such as Queries, Plugins, Custom, etc., are all-ASCII (presumably by design). However, the AppData path for a Unicode user is a Unicode path, strongly suggesting that the plugin will always crash with a Unicode user name in all versions of FH.

The way I get around limitations in Windows support for Unicode (it is far from universal) in my latest draft is to always refer to the short path for AppData, which doesn't cause any problems with emulators as the plugin only ever manipulates folders, not individual files, and emulator users cannot have Unicode names.

You are working at the file level, so reconciling Windows and emulator support may be more of a challenge.

That's probably as much as I can contribute at the moment, as I'm deliberately keeping it high level and staying away from your detailed coding, but happy to test any candidate fixes of course if you think it is worth spending time to fix an issue with Unicode users that's never been an issue in the real world!

[tatewise]

If I can find the time I'll investigate how my plugin handles Unicode in FH V6 and FH V7 but only in Windows.

[Mark1834]

I must admit, if I was in your shoes I probably wouldn’t do anything. As I see it, there are three distinct populations who can’t use your plugin - FH6 users with Unicode diagrams, reports, queries, etc; all users with Unicode Windows names, and emulator users. The first two have always been there, but they will be tiny numbers and it’s never been an issue before.

Emulators is probably by far the most significant of the three, but still relatively small and not a group you can support very well. All three groups can use the new version of the Windows plugin.

They lose the file-level control of updates, but that’s readily available from copying directly in either Windows or MacOS/Linux (and how often is it actually needed in real backup and restore scenarios?). As has been pointed out by our Mac experts before, experienced Mac users probably wouldn’t use either plugin anyway - it’s easier to backup directly from the OS and include it in their general backup (and in emulators, the Registry is just another file that can be copied directly).