* Backup and Restore FH Settings via Windows - new plugin

[BillH]

Mark,

I tried version 0.3 and have a few questions and comments.

1. If I select both Compress backup to zip archives and Write to dated subfolder, the Script button is grayed out. Why? I had to select only Compress backup to zip archives, click on the Script button, and then select Write to dated subfolder, and then click on Backup. Is this supposed to work like this? Seems a bit confusing.

2. When I did the backup it created a dated subfolder. In that folder were 2 .reg files and a .zip file. Shouldn't there have been 2 .zip files? The only .zip file created was fhProgramData.zip. Shouldn't there have been a fhAppData.zip? The Backup said it ended successfully.

3. The fhProgramData.zip it created was a 0 KB .zip file that could not be opened. I got this error.

image1.jpg (16.5 KiB) Viewed 2422 times

4. If I understand it, it looks like I have to edit or recreate the .bat file every time if I use dated subfolders. Is that correct or am I missing something. If it is correct, I think it would be a lot friendlier if the .bat file didn't need changing or recreating and just used the current date rather than have a hardcoded date.

5. The restore Script button is grayed out unless I select Force exact copy of backup configuration. For me that is ok, as that is what I would want anyway I think. Is this the way it is supposed to work?

Thanks,
Bill

[BillH]

Mark,

Also, when the plugin ended I thought it would close the cmd window. It did not. It was open and looked like this:

image2.jpg (55.88 KiB) Viewed 2420 times

I had to close it manually.

Thanks,
Bill

[tatewise]

Bill, the backup ZIP file creation is in progress... The [ooo is the start of a progress bar.
When the plugin says Backup successful. it really means Backup initiated. and you must wait for the ZIP creation.
The ZIP file includes the \Map\Cache\ files that are often much more than all the other config files and explains why the ZIP creation takes so long.

[BillH]

Mike,

Thanks for the explanation.

Mark,

Some additional comments.

1. The initial pop-up window should really say Backup Initiated rather than Backup Successful. The latter makes it appear that it has completed when it is just starting.

2. What good is a progress bar if the cmd window is minimized and we can't see it?

3. I have a fast i7 CPU and 64 meg of memory and it took over 5 minutes to complete the backup. This is way too long. If the map cache is included intentionally, at least have an option so that we can not include it.

4. I am using dated folders. Why not automatically create the restore script when doing the backup? That would be must easier than me having to remember to recreate it every time I do a backup.

After waiting for it to complete this time I can see that there are both AppData and ProgramData .zip files and both have data in them.

Thanks,

Bill

[Mark1834]

Bill, good comments, thanks, which raise a number of points that are well worth expanding on -

If you copy the files without zipping, the Map subfolder is omitted, as the Windows robocopy (short for "robust copy") command used has an easy command line option to exclude specific files or folders. Unfortunately, neither 7-Zip nor the Windows PowerShell CompressArchive applet have options to exclude subfolders that I could find (it might be in 7-Zip somewhere, but the command line syntax is fairly opaque and not well documented). With 7-Zip, that doesn't particularly matter, as it is a very efficient and well-written utility that runs extremely quickly. The performance hit going from 7-Zip to CompressArchive is huge. What takes a couple of seconds with 7-Zip on my old very basic laptop takes all of 90 seconds with CompressArchive! Even on the i-5 desktop, the difference is still very noticeable.

So do we offer 7-Zip (fast, free, but is extra software) or CompressArchive (slow, but included in all versions of Windows from W7 SP1 onwards) as the default? What I am clear on in my mind is that we don't offer both. The core model of this plugin is that it provides basic templates that can either be used "as is" by anybody, or modified by individuals to suit their requirements.

If speed is an issue, it helps if you do not select the Pause option from the main menu, as the script runs minimised. Once it has been kicked off, you can close the plugin, and even close FH, while it is still running as it is a separate and independent Windows process.

I didn't want to include a dynamic date in the basic script, as I was wary of the scenario whereby somebody sets it up as a daily automated backup, and soon ends up with gigabytes of copied setting! Again, an individual user could create a script that contained a dynamic date and automatically deleted old backups over a certain age (for example), but that is out of scope for the plugin. If you want dated backups, the intended method is that you open the plugin and run the backup manually by clicking on the Backup button. You don't have to worry about the script file (it is actually deleted, to prevent the scenario described).

Limitations in what the zip utilities are capable of also drives the "Force exact copy..." option. Without zipping, it can be on or off, but with zipping, it can only be on, as neither 7-Zip nor CompressArchive offer file level selection of not overwriting or deleting new and changed files. Once again, somebody could probably create a more complex script that offers that level of control for zipped archives, but not from the plugin.

Hope that makes sense. The basic philosophy of this plugin is a little different to many others. Rather than getting ever more complex and bloated as more options are added, it sticks to basic scenarios that should meet most user needs and will be fully described in the help file, while providing a framework that can be adapted and extended by those interested in doing so. In the extreme, you don't need the plugin at all, as suitable files could be created from scratch, but it is always easier to modify a template than start with a blank sheet.

[BillH]

Mark,

Thanks for the expanded explanations. A couple of additional things.

What takes a couple of seconds with 7-Zip on my old very basic laptop takes all of 90 seconds with CompressArchive! Even on the i-5 desktop, the difference is still very noticeable.

If I don't use .zip files the backup does take only seconds so that would be the way I would go. Over 5 minutes for the backup is not satisfactory.

If speed is an issue, it helps if you do not select the Pause option from the main menu, as the script runs minimised. Once it has been kicked off, you can close the plugin, and even close FH, while it is still running as it is a separate and independent Windows process.

I did not have this selected.

I didn't want to include a dynamic date in the basic script, as I was wary of the scenario whereby somebody sets it up as a daily automated backup, and soon ends up with gigabytes of copied setting!

I think this is really a user issue not a plugin issue. A user needs to monitor and clean up old files on their computer. This is also necessary using Mike's plugin. I'm not sure I agree that protecting users from creating a lot of data is really a responsibility of a plugin.

If you want dated backups, the intended method is that you open the plugin and run the backup manually by clicking on the Backup button.

I guess I should have realized that. This is the way I would probably do it anyway. I don't think I would schedule the backup to run automatically. However, this limitation does defeat the idea of being able to schedule the execution of the .bat file.

You don't have to worry about the script file (it is actually deleted, to prevent the scenario described).

When is this supposed to happen? After running the plugin, fhBackup.bat still exists in my folder.

Also there were a couple of things you didn't address in my comments.

- If I select both Compress backup to zip archives and Write to dated subfolder, the Script button is grayed out. Why? I had to select only Compress backup to zip archives, click on the Script button, and then select Write to dated subfolder, and then click on Backup. Is this supposed to work like this? Seems a bit confusing.

- The restore Script button is grayed out unless I select Force exact copy of backup configuration. For me that is ok, as that is what I would want anyway I think. Is this the way it is supposed to work?

- I am using dated folders. Why not automatically create the restore script when doing the backup? That would be much easier than me having to remember to recreate it every time I do a backup.

Thanks,
Bill

[ADC65]

Mark, I would suggest you consider whether you need a Zip option at all. The Backup folders and files produced come to about 24Mb on my installation, and this compresses to about 5Mb. Considering the number of backups I intend to keep, this is a trivial amount of disk space. If I did want to Zip the files, I can do it in File Explorer by selecting the relevant files and right-clicking, and choosing "Compress to Zip File", which took approximately 10 seconds (excluding Map Cache). I agree with your philosophy of keeping it simple, so I would suggest dropping the Zip option altogether.

[Mark1834]

You might be right. Somebody else suggested the same in an off-line discussion on scope. I agree with Bill that the performance hit from zipping is unacceptable, and it is limitations in what the zip utilities are capable of that drives much of the complexity of the UI (no folder exclusion, limited overwrite protection on restoring, etc). It was driven originally by the number of files rather than disk space. Even without the map cache, a settings backup is about 1500 files. Further copying of lots of small files (e.g. upload to cloud storage) is a lot slower than a handful of zip files.

As a general point, optimising dated backups is probably more useful than providing zipping.

[Mark1834]

Picking up Bill's other points -

You don't have to worry about the script file (it is actually deleted, to prevent the scenario described).
When is this supposed to happen? After running the plugin, fhBackup.bat still exists in my folder.

Oops - I commented out that line during testing and forgot to reinstate it! It is not actually deleted.

However, this limitation does defeat the idea of being able to schedule the execution of the .bat file.

I'm sympathetic to the idea of being able to create automatic dated backups if it can be done without undue complexity. That might be easier if zipping is not offered, so I'll have a play.

I am using dated folders. Why not automatically create the restore script when doing the backup?

I picked up Mike's suggestion of storing individual restore scripts in the dated folders rather than the parent folder, so they are there, but just in a different place,
Much of the other complexity is driven by zipping. Keep that turned off, and more options are available. I might post a parallel version with zipping permanently disabled so users can easily compare and contrast.

[BillH]

I picked up Mike's suggestion of storing individual restore scripts in the dated folders rather than the parent folder, so they are there, but just in a different place,

Sorry I wasn't clear. What I meant was, can the plugin automatically create the restore .bat file without us having to click on the Script button? Then if we ever need it, it is there along with the backups? If we don't ever need it, it takes up almost no space at all. I know that I for one would at some point forget to click on the Script button and wouldn't have the .bat file when I need it. This would make the UI even simpler.

Thanks,
Bill

[Woodg]

Even without the map cache, a settings backup is about 1500 files. Further copying of lots of small files (e.g. upload to cloud storage) is a lot slower than a handful of zip files.

As a general point, optimising dated backups is probably more useful than providing zipping.

Would archiving the files into zip but without compression be faster? That way you get less files (rather than 1500). Of course, if it's no faster, then just ignore me!

[Mark1834]

That's a reasonable thing to try, but unfortunately it doesn't make any difference. All that happened when I turned off compression was that the resulting file was nearly five times the size with no significant reduction in generation time.

I have a "version 3 lite" with all the compression removed, and it does feel a lot simpler to use. I do want to explore whether there is a simple option to incorporate a dynamic date in the script file, as a desktop shortcut that immediately generates a dated backup would be a useful tool.

So far, I have got as far as establishing that the following command generates a correctly formatted date suffix (e.g. "_2022-04-15").

Code: Select all

PowerShell Get-Date -Format '_yyyy-MM-dd'

All I have to do now is work out how to append that automatically to the destination folder in the general command "robocopy source destination" so it becomes "robocopy source destination_datesuffix".

I'm not that familiar with more detailed scripting, so I'll need to do some homework. If there are any scripting experts out there who know the answer already, I'm happy to take a steer...

[Ron Melby]

I dont know what you got so far, but as long as everything is strings....

$pth = 'whatever you have for a path now'
$dat = whatever date you get
(depending on what that output is, looks like its a string but if not...) then it would be almost exactly like lua except different (lol, I had to say it) --- getdate.tostring (or whatever the command is to get your date....
$flr = ($pth + $dat)
and you can get fancier like
$flr = ($pth + "whateverstringlikethingyouwant" + $dat + " ... ")
its just gluing u
p with () so it knows when to start and quit, + instead of the lua .. and EVERYTHING must be cast into strings....

but as powershell works for member functions its like

tst = tostring(fhGetRecordID(indiPointer))

tst = fhGetRecordID(indiPointer).tostring
first lua second powershell styles.

[tatewise]

You can invoke a block of Powershell commands and pass variables between them.

The outer block structure is Powershell Invoke-Command -ScriptBlock { "Commands" } that must all be on one line.

To pass arguments: Powershell Invoke-Command -ScriptBlock { param($p1,$p2) "Commands" } -ArgumentList "A","B"

Powershell commands are enclosed in string quotes and separated by semicolons: "Command1; Command2"

So to pass arguments to Powershell and form robocopy file names use such as:

Code: Select all

Powershell Invoke-Command -ScriptBlock { param($p1) "$date=Get-Date -Format '_yyyy-MM-dd'; robocopy $p1 ('XYZ' + $date)" } -ArgumentList "C:\Path"

i.e.
$date=Get-Date -Format '_yyyy-MM-dd' sets variable $date to the desired date string
robocopy $p1 ('XYZ' + $date) runs robocopy with source = $p1 = "C:\Path" and destination = 'XYZ' + $date from above giving robocopy 'C:\Path' 'XYZ_2022-04-15'

[Mark1834]

Thanks Mike. I was getting a little tied up with double versus single quotes (PS parameters in single, file paths in double to allow for spaces). The more verbose version I came up with is this. It's a little more like old fashioned DOS batch file structure. I'll have a play with which is easier to generate within the plugin and for users to potentially modify. All the plugin has to do is generate this file, substituting the selected path for my test value of D:\Junk.

Code: Select all

@echo off

set f="C:\ProgramData\Calico Pie\Family Historian\Plugin Data\~date.tmp"
PowerShell Get-Date -Format '_yyyy-MM-dd' > %f%
set /p suffix=<%f%
del %f%

set mypath=D:\Junk
set newpath=%mypath%\fhBackup%suffix%

robocopy "%PROGRAMDATA%\Calico Pie\Family Historian" "%newpath%\fhProgramData" /MIR /XD Map
robocopy "%APPDATA%\Calico Pie\Family Historian" "%newpath%\fhAppData" /MIR
Reg export "HKEY_CURRENT_USER\SOFTWARE\Calico Pie\Family Historian" "%newpath%\fhCURRENT_USER.reg" /y
Reg export "HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Calico Pie\Family Historian\2.0\Preferences" "%newpath%\fhLOCAL_MACHINE.reg" /y

[tatewise]

You can avoid the tmp file with a little command line magic:

Code: Select all

@echo off

for /f %%D in ( 'PowerShell Get-Date -Format "_yyyy-MM-dd"' ) do set suffix=%%D

set mypath=D:\Junk
set newpath=%mypath%\fhBackup%suffix%

robocopy "%PROGRAMDATA%\Calico Pie\Family Historian" "%newpath%\fhProgramData" /MIR /XD Map
robocopy "%APPDATA%\Calico Pie\Family Historian" "%newpath%\fhAppData" /MIR
Reg export "HKEY_CURRENT_USER\SOFTWARE\Calico Pie\Family Historian" "%newpath%\fhCURRENT_USER.reg" /y
Reg export "HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Calico Pie\Family Historian\2.0\Preferences" "%newpath%\fhLOCAL_MACHINE.reg" /y

[Mark1834]

Perfect! That is exactly the hybrid I was trying to construct, but was struggling with the syntax of the first line. I think that is the best solution, as it is fairly easy both to code and follow. If the dated option is not selected, I simply miss out the first part of the script.

Probably be tomorrow before I post an alternative full plugin with no zipping and fully functional dated backups, but glad to see that it is possible without undue complexity.

[Mark1834]

The attached plugin update implements changes discussed above. Zip options have been removed, and the backup script now contains a dynamic date rather than a static one if this option is selected. I have also removed some unnecessary confirmation messages.

I will leave the earlier version with zipping up for now, so users can try either. You can have both loaded as they have different names, but the version number will disappear from the name in the final store version. Overall, I think that this version is easier to use, but I'm guided by users - we all have different requirements and priorities!

• Once the backup location is selected, all buttons remain active at all times.
• Clicking on either of the Script buttons generates the appropriate script for the options you have defined, and stores it in the backup folder.
• Clicking on Backup generates both scripts automatically and runs the backup immediately.
• The backup script can be run at any time, even when FH is not open. That means that both single and dated backups can be generated directly from the script or automated if required.
• Note that if dated backups are selected, the restore script is a standard static restore, and not adjusted for the dynamically created backup locations. Clearly, FH cannot know in advance what they are, and creating a script that automatically generates dynamic restores as well is beyond the scope of this plugin (but perfectly possible if anyone want to do it manually!). In practice, I don't think this is a constraint, as you simply have to copy the selected dated backup back to the parent folder to run the standard script. Users with dated backups probably keep them "just in case", rather than frequently having to revert to an older configuration.

[Attachment deleted as the plugin is now in the store]

Over to users for testing. I think this simpler update without zipping is the way to go, but what do you think?

Edit: Attachment revised to correct a minor typo that only become apparent when I did a full backup/uninstall/reinstall/restore! Apologies to the early downloaders.

[BillH]

Mark,

I think the UI on this new version is much cleaner. No .zip processing works for me.

Bill

[BillH]

Mark,

Thank you for having the plugin build the restore script at the same time as the backup script when using the Backup button. I only have one comment.

When using dated subfolders and the Backup button, the old version put the date the backup was done into the path names in the restore script and put the script in the dated subfolder. This was ideal as the restore script for each dated subfolder was already there to use and no .bat file editing was required. Could this be incorporated into the new version?

So to summarize, when using dated subfolders and the Backup button, could the plugin again put the hardcoded date that the backup was done in the path names in the restore script and put the restore script in the dated subfolder.

Thanks,
Bill

[Mark1834]

Bill,

Consider the actual sequence of events. The plugin creates the scripts, and the scripts create the backup. The date that the script was created is not relevant.

If you only ever create backups in FH, the backup and script are created at the same time. However, the same backup script can be used over and over again to create either new dated backups or update a single undated backup. Running a backup doesn’t create a new script, so there is nothing to store.

Take it back a stage. What is it that you can’t do at the moment?

[tatewise]

What Bill is asking for is the fhBackup.bat file creates the fhRestore.bat file within the dated subfolder that restores the files from that dated subfolder. I agree with him. It avoids editing mistakes in the fhRestore.bat file. Otherwise, it is not as easy to perform the restore as it is to perform the backup.
i.e.

Code: Select all

robocopy "E:\Mike\...\Backup\fhBackup_2022-04-16\fhProgramData" "%PROGRAMDATA%\Calico Pie\Family Historian" /E /XO
robocopy "E:\Mike\...\Backup\fhBackup_2022-04-16\fhAppData" "%APPDATA%\Calico Pie\Family Historian" /E /XO
Reg import "E:\Mike\...\Backup\fhBackup_2022-04-16\fhCURRENT_USER.reg"
Reg import "E:\Mike\...\Backup\fhBackup_2022-04-16\fhLOCAL_MACHINE.reg"

[BillH]

Mark,

Sorry if I was not clear. I am talking about the restore script.

After doing a bunch of backups, I will have multiple dated folders with the backups in them. To do a restore I need to have the correct path name in the restore script which includes the correct dated folder name. For example:

Reg import "D:\My Documents\FH\Backup and Restore Family Historian Settings via Windows\fhBackup_2022-04-14\fhLOCAL_MACHINE.reg"

Version 0.3.2 builds just one restore script and puts it in the parent folder. That restore script doesn't have the dated folder name in the paths. For example:

Reg import "D:\My Documents\FH\Backup and Restore Family Historian Settings via Windows\fhLOCAL_MACHINE.reg"

Version 0.3 did create restore scripts with the dated folder name as part of the path name. The first example above was create with version 0.3. It also put the restore script in the dated subfolder along with the restore commands so that each dated subfolder had everything you needed to do the restore.

Version 0.3.2 requires you to modify the batch file to include the dated subfolder you want before you do the restore.

I just think that the way version 0.3 handled the restore .bat file was more user friendly.

Thanks,
Bill

[tatewise]

May I also suggest that the fhRestore.bat script tests if FH is running and simply notifies the user if it is.
That is important because otherwise the restored registry settings will get overwritten when FH closes with its current settings and would undo the restore.
A Google search for cmd script test if process is running reveals how to do it in one or two lines.

[Mark1834]

Gents,

I think there are mutually exclusive requirements here. If we want a dynamic date, such that the backup script can be run automatically at any time, we can’t have individually tailored restore scripts for each one (unless we create a backup script that also dynamically creates additional restore scripts each time it runs, but that’s way out of scope for this plugin).

There are many ways to restore a dated backup, all of which will be described in the help file. It doesn’t need the script to be edited. The simplest way is to run the plugin, select the appropriate folder, and create a fresh script. I agree that it’s slightly more complex than backing up, but I don’t think it is any more difficult than the current plugin.

I need to understand why you want easy restore of dated backups. Is it something you do regularly, or “just in case”? My working assumption is that most users run far more backups than restores (they should do!).