From ebfbbf9abaa63f7070d304e8d579bd082750ce94 Mon Sep 17 00:00:00 2001 From: Raz Ben Aharon <181751257+razbenaharon@users.noreply.github.com> Date: Sat, 7 Nov 2026 17:46:56 +0200 Subject: [PATCH] Fix five serious findings from the repository review; bring docs and CI up to date Fixes: - Plain copy never copies a private password/recovery-key folder or key file, also when it is itself the selected folder or file (robocopy applies /XD only below the folder it copies). Older private copies in the mirror are reported. - A new source never gets the plain copy or versions folder of a removed source of the same drive, also while the drive is unplugged; a removed job is not adopted when a current source uses its repository. - Before the drive check asks for administrator rights, the window checks who could change what it runs or rename a folder above it (links included) and asks first. smartctl runs only when its SHA-256 matches the official 7.5 build and is held open while it runs; the installer is held open from its checksum check until it has run. - The drive check judges the newest short and long self-test that passed or failed each on its own, so a passing short test no longer hides a failed one. - Keep versions deletes, then frees space in a separate step, and a failed repository check after the clean-up now fails the backup. Maintenance: - README, BACKUP-SYSTEM.md, CHANGELOG and THIRD-PARTY match the code again. - CI uses actions/checkout@v7, builds Backup.Taskbar.cs and runs Test-BackupResourceControls.ps1. - New-DevDriveBackupIcon.ps1 draws the current icon from design/icon. - Two tests no longer leave empty folders in %TEMP%; new tests cover each fix. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/validate.yml | 8 ++- BACKUP-SYSTEM.md | 24 +++---- Backup-Worker.ps1 | 37 ++++++++-- Backup.DriveCheck.psm1 | 119 ++++++++++++++++++++++++++------- Backup.Selection.psm1 | 12 +++- CHANGELOG.md | 18 +++-- DevDriveBackup.Data.psm1 | 50 ++++++++++---- DevDriveBackup.ps1 | 19 +++++- Install-DriveTools.ps1 | 19 ++++-- Invoke-DriveCheck.ps1 | 19 +++++- Invoke-FastMirror.ps1 | 22 ++++-- New-DevDriveBackupIcon.ps1 | 94 +++++++++++++------------- README.md | 24 ++++--- THIRD-PARTY.md | 2 +- Test-BackupCopyRules.ps1 | 28 ++++++++ Test-BackupDriveCheck.ps1 | 51 ++++++++++++++ Test-BackupSecuritySafety.ps1 | 43 ++++++------ Test-BackupStatus.ps1 | 28 ++++---- 18 files changed, 455 insertions(+), 162 deletions(-) diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index c4eebae..795e270 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -6,7 +6,7 @@ jobs: syntax: runs-on: windows-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v7 - name: Check for private backup details shell: pwsh run: ./Test-PrivacyGuard.ps1 @@ -22,11 +22,12 @@ jobs: if ([IO.File]::ReadAllText($_.FullName) -match '[\x00-\x08\x0B\x0C\x0E-\x1F]') { Write-Error "Control character in $($_.Name)"; $failed = $true } } if ($failed) { exit 1 } - - name: Compile native helper and report generator + - name: Compile native helpers and report generator shell: pwsh run: | Add-Type -Path ./Backup.Process.cs Add-Type -Path ./Backup.MirrorVerifier.cs + Add-Type -Path ./Backup.Taskbar.cs python -m py_compile ./Build-BackupReport.py - name: Check example configuration shell: pwsh @@ -57,3 +58,6 @@ jobs: - name: Verify plain copy rules shell: pwsh run: ./Test-BackupCopyRules.ps1 + - name: Verify the hard memory limit + shell: pwsh + run: ./Test-BackupResourceControls.ps1 diff --git a/BACKUP-SYSTEM.md b/BACKUP-SYSTEM.md index 75b6158..34e3677 100644 --- a/BACKUP-SYSTEM.md +++ b/BACKUP-SYSTEM.md @@ -1,6 +1,6 @@ # DevDriveBackup: versioned engine -Windows backup workbench for development drives and large numbers of small files. It combines metadata inventory, USB link diagnostics, a bounded disk benchmark, and an encrypted, versioned [Restic](https://restic.net/) repository. The English Windows Forms manager provides source selection, progress, ETA, speed presets, and editable bandwidth/memory limits. +Windows backup workbench for development drives and large numbers of small files. It combines metadata inventory, USB link diagnostics, a bounded disk benchmark, and an encrypted, versioned [Restic](https://restic.net/) repository. The advanced window (`Backup-Manager.ps1`) provides source selection, progress, ETA, speed presets, and editable bandwidth/memory limits. The plain copy (`Invoke-FastMirror.ps1`) is a separate engine; both read the same per-folder selection. This document covers the versioned engine. @@ -21,7 +21,7 @@ pwsh -NoProfile -File ./Initialize-BackupConfig.ps1 -Source X:\ -DestinationDriv Edit `backup.local.json`: set the source, both volume identities, relative source selections, and two absolute key-file paths. Keep the password and recovery file outside the backed-up source. `RepositoryFolder` is a new folder on the selected destination, not the existing mirror. Destination drive-letter changes are supported through volume identity lookup; source identity changes fail closed. -The initializer fills volume identities and private key paths automatically, refuses to overwrite an existing configuration, and starts with non-system top-level folders, except common temporary/package-cache roots. Review these defaults. Alternatively copy `config.example.json` and fill in its placeholders manually. Each entry in `Choices` has a `Mode` (`Mirror`, `Versioned`, `Both` or `Skip`) shared with `Invoke-FastMirror.ps1`; `DefaultMode` covers paths no entry lists. Entries can be folders or single files (for example `README.md` at the root) and can nest. The worker derives its sources from `Versioned`/`Both` entries; `Include` is only kept up to date for other readers. See "Choose per folder" in README.md. +The initializer fills volume identities and private key paths automatically, refuses to overwrite an existing configuration, and starts with non-system top-level folders, except common temporary/package-cache roots. Review these defaults. Alternatively copy `config.example.json` and fill in its placeholders manually. Each entry in `Choices` has a `Mode` (`Mirror`, `Versioned`, `Both` or `Skip`) shared with `Invoke-FastMirror.ps1`; `DefaultMode` covers paths no entry lists. Entries can be folders or single files (for example `README.md` at the root) and can nest. The worker derives its sources from `Versioned`/`Both` entries; `Include` is only kept up to date for other readers. See "How it works" in README.md. Start `DevDriveBackup-Advanced.cmd` (or use the main DevDriveBackup window) or: @@ -29,7 +29,7 @@ Start `DevDriveBackup-Advanced.cmd` (or use the main DevDriveBackup window) or: pwsh -NoProfile -File ./Backup-Manager.ps1 ``` -Profiles are starting points, not a hardware-independent optimum. Review exclusions and deselect anything you do not want backed up. Backup runs create snapshots; source deletions do not automatically delete older snapshots. No scheduled task or automatic retention/pruning is installed. Manual history deletion and optional pruning are available through **History and restore**. +Profiles are starting points, not a hardware-independent optimum. Review exclusions and deselect anything you do not want backed up. Backup runs create snapshots; source deletions do not automatically delete older snapshots. No scheduled task is installed. Old versions are deleted automatically only when **Keep versions** is set on the main window's overview (`Backup-Worker.ps1 -KeepDays -KeepWeeks -KeepMonths`; off by default). Manual history deletion and optional pruning are available through **History and restore**. ## History browser @@ -65,14 +65,14 @@ Interruption leaves the protected journal available for **Resume interrupted cha | Balanced | 45 MB/s | 1024 MiB | 2 | 2 | 16 MiB | | Fast | Unlimited | 2048 MiB | 4 | 4 | 32 MiB | -MB/s means decimal bytes; MiB means binary bytes. Bandwidth and memory are editable. The worker converts the rate to Restic's KiB/s units. `0` means no bandwidth cap. Local backend concurrency defaults to 1 to keep a rotational destination's writes mostly sequential, except history deletion with prune, which requires at least 2 connections. Set `BackendConnections` (1–8) in the configuration for an SSD; `Set-BackupDestination.ps1 -Apply` sets 4 when it detects one. Restic packs small objects together and avoids storing identical chunks again; this addresses much of the per-file destination overhead. +These presets belong to the advanced window. The main window instead computes Light, Balanced, Strong, Maximum or Custom from the detected processor and memory and writes it into the job when a backup starts: threads, memory and file reads from the level, no bandwidth cap, 16 MiB packs for Light and 32 MiB otherwise, 2 backend connections for Light and Balanced and 4 otherwise, and the level's priority. MB/s means decimal bytes; MiB means binary bytes. Bandwidth and memory are editable. The worker converts the rate to Restic's KiB/s units. `0` means no bandwidth cap. Local backend concurrency defaults to 1 to keep a rotational destination's writes mostly sequential, except history deletion with prune and the automatic clean-up of old versions, which use at least 2 connections. Set `BackendConnections` (1–8) in the configuration for an SSD; `Set-BackupDestination.ps1 -Apply` sets 4 when it detects one. Restic packs small objects together and avoids storing identical chunks again; this addresses much of the per-file destination overhead. - `GOMAXPROCS` limits Go execution parallelism. Quiet/balanced processes use BelowNormal CPU priority. This is not a hard CPU-time quota. - `GOMEMLIMIT` is a soft Go target at 70% of the selected budget. A Windows Job Object enforces the selected hard per-process committed-memory limit. Exceeding it can cause allocation failure and an incomplete run. Increase the budget rather than assuming every repository fits in 512 MiB. - Job-object accounting does **not** include the Windows system file cache, the GUI, or unrelated processes. Peak resident memory is reported separately from the committed-memory cap. - Restic bandwidth limits constrain repository transfers; they do not throttle every source metadata/read operation. Compression and deduplication can make processed source MB/s exceed repository MB/s. - There is no promise of zero impact on interactive work. Measure with the actual development workload, not just an idle desktop. More RAM does not inherently make a USB HDD faster. -- The current version chooses presets manually; it does not implement an adaptive load controller, pause/resume, or an automatic hard free-space floor during the run. It refuses to start below 5 GiB free. Disk-full/I/O failures are reported. +- The current version chooses presets manually; it does not implement an adaptive load controller, pause/resume, or an automatic hard free-space floor during the run. A backup or preview refuses to start below 5 GiB free; deleting old versions has no such floor. Disk-full/I/O failures are reported. See [Restic tuning guidance](https://restic.readthedocs.io/en/stable/047_tuning_parameters.html). @@ -109,9 +109,9 @@ pwsh -NoProfile -File ./Invoke-DriveCheck.ps1 -Drive Y -OutputDirectory ./audit/ pwsh -NoProfile -File ./Invoke-DriveCheck.ps1 -Drive Y -OutputDirectory ./audit/check -Mode LongSelfTest ``` -The window runs these through a Windows administrator prompt and keeps the results per profile in `%LOCALAPPDATA%\DevDriveMirror\drive-check\`: `report.json` (last quick check), `surface.json` (full scan position and findings), `run.json` (last run), `progress.json` (live). An elevated process cannot be ended by the window, so *Stop* creates `stop.request` and the check stops itself after the current step; a full scan saves its position every 512 MiB and continues from there. +Before it asks Windows for administrator rights, the window checks who could change what the check runs: `Invoke-DriveCheck.ps1`, the modules and script it loads, `smartctl` and its folder, and every folder above the tool folder (whoever can rename one of those can swap the tool folder itself). When an account other than you, SYSTEM or Administrators can, it names those accounts and asks before it starts; keep the tool under your user folder to avoid that. The check runs the bundled `smartctl.exe` only when its SHA-256 matches the 64-bit build in the official 7.5 installer, and keeps the file open, readable only, until it ends. The window runs these through a Windows administrator prompt and keeps the results per profile in `%LOCALAPPDATA%\DevDriveMirror\drive-check\`: `report.json` (last quick check), `surface.json` (full scan position and findings), `run.json` (last run), `progress.json` (live). An elevated process cannot be ended by the window, so *Stop* creates `stop.request` and the check stops itself after the current step; a full scan saves its position every 512 MiB and continues from there. -Reads use `\\.\PhysicalDriveN` with unbuffered aligned 4 MiB reads. A failing chunk is read again in 64 KiB pieces to locate the damaged place. A chunk over one second counts as very slow (the drive retried), over ten times the median and 250 ms as slow; many slow places (over 0.2% of what was read, at least 20) are a warning. Errors that mean the drive disappeared (not ready, no such device, disconnected) end the run instead of being recorded as bad areas. Nothing is written. The health report tries smartctl's own device-type detection, then standard SAT pass-through; vendor-specific USB bridge commands are not tried blindly. Windows log entries (warnings and errors of the last 30 days) name a disk by number, and numbers move between drives as they are plugged in. Each entry is matched to the drive that held that number at the time, using Windows' own record of drive arrivals (`Microsoft-Windows-Partition/Diagnostic`, event 1006: disk number, serial number, model, size), so another drive's errors are not held against this one. The verdict rules live in `Backup.DriveCheck.psm1` and are tested by `Test-BackupDriveCheck.ps1` without hardware. +Reads use `\\.\PhysicalDriveN` with unbuffered aligned 4 MiB reads. A failing chunk is read again in 64 KiB pieces to locate the damaged place. A chunk over one second counts as very slow (the drive retried); one over 250 ms and over ten times the median of its 512 MiB stretch counts as slow; many slow places (more than 20, and more than 0.2% of what was read) are a warning. Errors that mean the drive disappeared (not ready, no such device, disconnected) end the run instead of being recorded as bad areas. The check itself writes nothing to the drive; Windows' online file system scan may note what it finds in the file system's own records. The health report tries smartctl's own device-type detection, then standard SAT pass-through; vendor-specific USB bridge commands are not tried blindly. Windows log entries (warnings and errors of the last 30 days) name a disk by number, and numbers move between drives as they are plugged in. Each entry is matched to the drive that held that number at the time, using Windows' own record of drive arrivals (`Microsoft-Windows-Partition/Diagnostic`, event 1006: disk number, serial number, model, size), so another drive's errors are not held against this one. The verdict rules live in `Backup.DriveCheck.psm1` and are tested by `Test-BackupDriveCheck.ps1` without hardware. ## What to preserve @@ -127,7 +127,7 @@ Live-file backups are not application-consistent snapshots. This wrapper does no Every successful Backup is followed by structural `restic check`. The **Full verification** button runs `check --read-data`, which reads and verifies all repository data and takes additional time. Restic's default pre-upload verification remains enabled. -Incremental backups normally use Restic's metadata change detection. On Windows, unchanged path, size and modification time can cause content to be reused without reading it again. That can miss a source change preserving all those fields. **Re-read all content** in the UI, or `-ForceRead` with `Backup-Worker.ps1`, disables that shortcut. Use it periodically or when source-integrity concerns warrant a full read. Repository verification and a fresh source-content scan answer different questions; neither should be presented as the other. +Incremental backups normally use Restic's metadata change detection. On Windows, unchanged path, size and modification time can cause content to be reused without reading it again. That can miss a source change preserving all those fields. **Re-read all file contents** in the advanced window, or `-ForceRead` with `Backup-Worker.ps1`, disables that shortcut. Use it periodically or when source-integrity concerns warrant a full read. Repository verification and a fresh source-content scan answer different questions; neither should be presented as the other. ```powershell pwsh -NoProfile -File ./Backup-Worker.ps1 -ConfigPath ./backup.local.json -Action Snapshots @@ -135,7 +135,7 @@ pwsh -NoProfile -File ./Backup-Worker.ps1 -ConfigPath ./backup.local.json -Actio pwsh -NoProfile -File ./Backup-Worker.ps1 -ConfigPath ./backup.local.json -Action Restore -Snapshot latest -RestoreTarget C:\Backup-Restore-Test ``` -Restore requires a **new, nonexistent folder** outside the source and repository. `--verify` validates restored file contents. Do a real restore test and inspect the application/data. Retain a second independently stored backup; a single USB drive cannot protect against its own failure or theft. The history window offers explicit selected-version deletion and optional prune with confirmation; automatic retention remains unimplemented. +Restore requires a **new, nonexistent folder** outside the source and repository. `--verify` validates restored file contents. Do a real restore test and inspect the application/data. Retain a second independently stored backup; a single USB drive cannot protect against its own failure or theft. The history window offers explicit selected-version deletion and optional prune with confirmation; automatic clean-up is the optional **Keep versions** setting on the main window's overview. If the original computer/account is lost, install Restic on another Windows computer and use the portable recovery password directly; do not depend on the old DPAPI file or source drive being present: @@ -144,7 +144,7 @@ If the original computer/account is lost, install Restic on another Windows comp ./bin/restic_0.19.1_windows_amd64.exe -r Y:\DevDrive-Backup restore latest --target C:\Recovered-DevDrive --verify ``` -Enter the saved repository password at Restic's prompt. Drive letters may differ. Recreate a local configuration only if you want to continue using the manager on that machine. +Enter the saved repository password at Restic's prompt. Drive letters may differ. Use your repository's own folder name: `DevDrive-Backup` is the `RepositoryFolder` that `Initialize-BackupConfig.ps1` writes, and the main window names each repository `-Versions`. Recreate a local configuration only if you want to continue using the manager on that machine. ## Validation and public repository hygiene @@ -152,12 +152,12 @@ Enter the saved repository password at Restic's prompt. Drive letters may differ pwsh -NoProfile -File ./Test-BackupSystem.ps1 -DestinationDrive Y ``` -This cross-volume integration test creates disposable source/repository/key directories. It verifies encrypted snapshots, cache exclusions, Hebrew/bracket filenames, SHA-256 of a restored binary, a full data check, old-version restoration, literal selected-file restore, history listing, rejection of unconfirmed/newest/empty deletions, explicit old-history deletion and prune, and restoration of the retained version. Artifacts are printed and left for inspection. It does not touch the configured real repository. `Test-BackupHistory.ps1` tests deletion-plan protection without a repository. The legacy suite is `Test-DevDriveMirror.ps1`. +This cross-volume integration test creates disposable source/repository/key directories. It verifies encrypted snapshots, cache exclusions, Hebrew/bracket filenames, SHA-256 of a restored binary, a full data check, old-version restoration, literal selected-file restore, history listing, rejection of unconfirmed/newest/empty deletions, explicit old-history deletion and prune, and restoration of the retained version. Artifacts are printed and left for inspection. It does not touch the configured real repository. `Test-BackupHistory.ps1` tests deletion-plan protection without a repository, and `Test-BackupRetention.ps1` tests the automatic clean-up against a disposable repository. `Test-BackupStatus.ps1` exercises Windows reader-lock contention. Progress publication uses atomic replacement with bounded retries and is best effort, so a locked progress file cannot abort the backup engine. Each worker also publishes an independent terminal `result.json`; the manager and external monitors should prefer it over `status.json`. Readers should use `Read-BackupStatus` from `Backup.Status.psm1`, which permits delete sharing. `Test-BackupResourceControls.ps1` checks the native hard memory cap by requesting a 512 MiB allocation inside a process limited to 256 MiB. The integration suite also covers forced rereads of same-size/same-time changes and rejection of wrong source identities and existing restore destinations. Partial restore tests can use `Backup-Worker.ps1 -Action Restore -RestoreInclude /X/Projects/example/README.md`; this is a Restic snapshot-path pattern, not a Windows source path. -Before publishing, review `git status` and the full staged diff. `.gitignore` excludes local configs, audit/reports, engine binaries, logs, protected passwords, recovery keys and temporary test sources. The example config contains placeholders only. Audit CSV/HTML can reveal private names and paths even when no file contents were read. Never use `git add -f` on them. Downloaded Restic is a separate BSD-licensed dependency; its official source/license/release are linked in `THIRD-PARTY.md`. +Before publishing, review `git status` and the full staged diff. `.gitignore` excludes local configs, audit/reports, engine binaries, logs, protected passwords, recovery keys and temporary test sources. The example config contains placeholders only. Audit CSV/HTML can reveal private names and paths even when no file contents were read. Never use `git add -f` on them. Downloaded restic (BSD-2-Clause) and smartmontools (GPL-2.0-or-later) are separate dependencies; their official sources, licenses and releases are linked in `THIRD-PARTY.md`. This wrapper is MIT-licensed (see `LICENSE`). Real-drive backup, full verification, restore evidence and unresolved limitations should remain visible in release notes. diff --git a/Backup-Worker.ps1 b/Backup-Worker.ps1 index cd22f3e..82bf6e1 100644 --- a/Backup-Worker.ps1 +++ b/Backup-Worker.ps1 @@ -35,7 +35,8 @@ function Assert-PlainAncestors([string]$Path) { function Run-Engine([string[]]$EngineArguments,[string]$Stage,[string]$InputSecret) { $script:status.State=$Stage;Save-Status $psi=[Diagnostics.ProcessStartInfo]::new($script:engine);$psi.UseShellExecute=$false;$psi.CreateNoWindow=$true;$psi.RedirectStandardOutput=$true;$psi.RedirectStandardError=$true - $connections=if($Stage -in 'DeletingHistory','DeletingOldVersions' -and $EngineArguments -contains '--prune'){[math]::Max(2,$script:backendConnections)}else{$script:backendConnections} + # Pruning needs at least two connections, whether it runs with forget or on its own. + $connections=if(($Stage -in 'DeletingHistory','DeletingOldVersions' -and $EngineArguments -contains '--prune') -or $Stage -eq 'FreeingSpace'){[math]::Max(2,$script:backendConnections)}else{$script:backendConnections} foreach($a in @('-r',$script:repository,'--json','--cache-dir',$script:cache,'--pack-size',[string]$script:config.PackMiB,'--compression','auto','--limit-upload',[string][int]($script:config.RateMBps*1000000/1024),'--limit-download',[string][int]($script:config.RateMBps*1000000/1024),'-o',('local.connections='+$connections))+$EngineArguments){[void]$psi.ArgumentList.Add($a)} $psi.Environment['RESTIC_PASSWORD']=$script:password foreach($name in @('RESTIC_PASSWORD_FILE','RESTIC_PASSWORD_COMMAND','RESTIC_REPOSITORY_FILE','RESTIC_KEY_HINT')){[void]$psi.Environment.Remove($name)} @@ -252,8 +253,13 @@ try { 'Backup' { Run-Engine ($base+@('--')+$sources) 'BackingUp';Run-Engine @('check') 'CheckingStructure' if($KeepDays+$KeepWeeks+$KeepMonths -gt 0){ - # The backup itself has succeeded and been checked. A failed clean-up is reported, not a failed backup. + # The backup itself has succeeded and been checked. A clean-up that fails is reported, not a failed + # backup. Old versions are deleted first and their space freed in a second step, so the result says + # how far it got. Once versions were deleted the repository is checked again, outside that report: + # a repository that fails this check fails the run, since it may be damaged. $new=$script:status.Snapshot + $retention=[ordered]@{Days=$KeepDays;Weeks=$KeepWeeks;Months=$KeepMonths;Removed=0;Kept=$null;SpaceFreed=$false} + $remove=@();$deleting=$false try{ Run-Engine (Get-RetentionArguments $KeepDays $KeepWeeks $KeepMonths) 'PlanningCleanUp' $plan=@(Get-Content -LiteralPath (Join-Path $RunDirectory 'PlanningCleanUp.jsonl') -Raw|ConvertFrom-Json) @@ -261,10 +267,31 @@ try { $available=@(Get-Content -LiteralPath (Join-Path $RunDirectory 'ReadingHistory.jsonl') -Raw|ConvertFrom-Json) $remove=@(Get-RetentionRemovals $plan $new $available) [IO.File]::WriteAllText((Join-Path $RunDirectory 'clean-up-plan.json'),(ConvertTo-Json @($remove) -Depth 8)) - if($remove.Count){Run-Engine (@('forget')+@($remove|ForEach-Object {$_.id})+'--prune') 'DeletingOldVersions';Run-Engine @('check') 'CheckingStructure'} - $script:status.Retention=[ordered]@{Days=$KeepDays;Weeks=$KeepWeeks;Months=$KeepMonths;Removed=$remove.Count;Kept=($available.Count-$remove.Count)} + $retention.Kept=$available.Count-$remove.Count + if($remove.Count){ + $deleting=$true + Run-Engine (@('forget')+@($remove|ForEach-Object {$_.id})) 'DeletingOldVersions' + $retention.Removed=$remove.Count + Run-Engine @('prune') 'FreeingSpace' + $retention.SpaceFreed=$true + } }catch{ - $script:status.Retention=[ordered]@{Days=$KeepDays;Weeks=$KeepWeeks;Months=$KeepMonths;Removed=0;Error=$_.Exception.Message} + $retention.Error=$_.Exception.Message + # Deleting may have stopped part way: count the versions that are really gone. + if($remove.Count -and -not $retention.Removed){ + try{ + Run-Engine @('snapshots','--tag','devdrive') 'ReadingHistoryAgain' + $still=@(Get-Content -LiteralPath (Join-Path $RunDirectory 'ReadingHistoryAgain.jsonl') -Raw|ConvertFrom-Json|ForEach-Object {$_.id}) + $retention.Removed=@($remove|Where-Object {$_.id -notin $still}).Count + $retention.Kept=@($still).Count + }catch{} + } + } + $script:status.Retention=$retention + # Checked whenever deleting started, also when how far it got is unknown. + if($deleting){ + try{Run-Engine @('check') 'CheckingAfterCleanUp'} + catch{throw "The new version was saved, but the repository check after deleting old versions failed, so the repository may be damaged. Run a full check of the versions before relying on them. $($_.Exception.Message)"} } } } diff --git a/Backup.DriveCheck.psm1 b/Backup.DriveCheck.psm1 index 876554e..76abb91 100644 --- a/Backup.DriveCheck.psm1 +++ b/Backup.DriveCheck.psm1 @@ -7,8 +7,9 @@ # Connection the way to the drive: link speed, transfer errors, resets and disconnections # FileSystem the file system's own structures (Windows online scan) # -# Pure functions over data that Test-BackupDrive.ps1 collects as administrator, so the rules -# are tested without hardware. Statuses: OK, Warning, Problem, Unavailable, NotRun. +# Pure functions over data that Invoke-DriveCheck.ps1 collects as administrator, so the rules +# are tested without hardware. Statuses: OK, Warning, Problem, Unavailable, NotRun. The one +# exception, Read-DriveCheckPathAccess, reads who may change the files the check runs. Set-StrictMode -Version Latest $ErrorActionPreference='Stop' @@ -36,25 +37,40 @@ function Test-DriveSmartAnswer($Smart) { ($null -ne (Get-Field $Smart 'smart_status','passed')) -or [bool](Get-Field $Smart 'ata_smart_attributes','table') -or [bool](Get-Field $Smart 'nvme_smart_health_information_log') } function Get-DriveSelfTest($Smart) { - # The self-test running now, if any, and the newest finished one from the drive's own log. - $now=$null;$last=$null + # The self-test running now, if any, and from the drive's own log (newest first) the newest entry + # (Last) and the newest finished short and long test (LastShort, LastLong). Each kind is judged on + # its own: a short test that passes says nothing about a long one that failed before it. + $now=$null $ataNow=Get-Field $Smart 'ata_smart_data','self_test','status','value' if($null -ne $ataNow -and ([int]$ataNow -band 0xF0) -eq 0xF0){$now=[pscustomobject]@{Remaining=[int](Get-Field $Smart 'ata_smart_data','self_test','status','remaining_percent')}} $nvmeNow=Get-Field $Smart 'nvme_self_test_log','current_self_test_operation','value' if($nvmeNow){$now=[pscustomobject]@{Remaining=100-[int](Get-Field $Smart 'nvme_self_test_log','current_self_test_completion_percent')}} - $ata=@(Get-Field $Smart 'ata_smart_self_test_log','standard','table') - $nvme=@(Get-Field $Smart 'nvme_self_test_log','table') - if($ata.Count -and $ata[0]){ - $e=$ata[0];$code=[int](Get-Field $e 'status','value') -shr 4 - $outcome=switch($code){0{'Passed'}{$_ -in 1,2}{'Interrupted'}15{'Running'}default{'Failed'}} - $last=[pscustomobject]@{Kind=if([string](Get-Field $e 'type','string') -match 'Extended|Long'){'Long'}else{'Short'};Outcome=$outcome;Text=[string](Get-Field $e 'status','string');Hours=Get-Field $e 'lifetime_hours'} - }elseif($nvme.Count -and $nvme[0]){ - $e=$nvme[0];$code=[int](Get-Field $e 'self_test_result','value') - $outcome=switch($code){0{'Passed'}{$_ -in 5,6,7}{'Failed'}default{'Interrupted'}} - $last=[pscustomobject]@{Kind=if([int](Get-Field $e 'self_test_code','value') -eq 2){'Long'}else{'Short'};Outcome=$outcome;Text=[string](Get-Field $e 'self_test_result','string');Hours=Get-Field $e 'power_on_hours'} - } + $ata=@(Get-Field $Smart 'ata_smart_self_test_log','standard','table'|Where-Object {$_}) + $nvme=@(Get-Field $Smart 'nvme_self_test_log','table'|Where-Object {$_}) + $entries=@(if($ata.Count){ + foreach($e in $ata){ + $code=[int](Get-Field $e 'status','value') -shr 4 + $outcome=switch($code){0{'Passed'}{$_ -in 1,2}{'Interrupted'}15{'Running'}default{'Failed'}} + [pscustomobject]@{Kind=if([string](Get-Field $e 'type','string') -match 'Extended|Long'){'Long'}else{'Short'};Outcome=$outcome;Text=[string](Get-Field $e 'status','string');Hours=Get-Field $e 'lifetime_hours'} + } + }else{ + foreach($e in $nvme){ + $code=[int](Get-Field $e 'self_test_result','value') + $outcome=switch($code){0{'Passed'}{$_ -in 5,6,7}{'Failed'}default{'Interrupted'}} + [pscustomobject]@{Kind=if([int](Get-Field $e 'self_test_code','value') -eq 2){'Long'}else{'Short'};Outcome=$outcome;Text=[string](Get-Field $e 'self_test_result','string');Hours=Get-Field $e 'power_on_hours'} + } + }) + $last=if($entries.Count){$entries[0]}else{$null} + # A test that was interrupted proves nothing either way, so the verdict of each kind comes from its + # newest test that passed or failed; a newer interrupted one is only mentioned (ShortInterrupted, LongInterrupted). + $lastShort=@($entries|Where-Object {$_.Kind -eq 'Short' -and $_.Outcome -in 'Passed','Failed'})|Select-Object -First 1 + $lastLong=@($entries|Where-Object {$_.Kind -eq 'Long' -and $_.Outcome -in 'Passed','Failed'})|Select-Object -First 1 + $newestShort=@($entries|Where-Object {$_.Kind -eq 'Short' -and $_.Outcome -ne 'Running'})|Select-Object -First 1 + $newestLong=@($entries|Where-Object {$_.Kind -eq 'Long' -and $_.Outcome -ne 'Running'})|Select-Object -First 1 + $shortInterrupted=if($newestShort -and $newestShort.Outcome -eq 'Interrupted'){$newestShort}else{$null} + $longInterrupted=if($newestLong -and $newestLong.Outcome -eq 'Interrupted'){$newestLong}else{$null} $supported=[bool](Get-Field $Smart 'ata_smart_data','capabilities','self_tests_supported') -or [bool](Get-Field $Smart 'nvme_optional_admin_commands','self_test') - [pscustomobject]@{Supported=$supported;Running=$now;Last=$last;ShortMinutes=Get-Field $Smart 'ata_smart_data','self_test','polling_minutes','short';LongMinutes=Get-Field $Smart 'ata_smart_data','self_test','polling_minutes','extended'} + [pscustomobject]@{Supported=$supported;Running=$now;Last=$last;LastShort=$lastShort;LastLong=$lastLong;ShortInterrupted=$shortInterrupted;LongInterrupted=$longInterrupted;ShortMinutes=Get-Field $Smart 'ata_smart_data','self_test','polling_minutes','short';LongMinutes=Get-Field $Smart 'ata_smart_data','self_test','polling_minutes','extended'} } function Get-DriveHealthFindings($Smart,[string]$Unavailable='') { @@ -99,15 +115,19 @@ function Get-DriveHealthFindings($Smart,[string]$Unavailable='') { } $t=Get-DriveSelfTest $Smart if($t.Running){$info.Add("A self-test is running now, about $($t.Running.Remaining)% left.")} - if($t.Last){ - $kind=if($t.Last.Kind -eq 'Long'){'Long self-test'}else{'Short self-test'} - $when=if($null -ne $t.Last.Hours -and $null -ne $hours){$ago=[double]$hours-[double]$t.Last.Hours;if($ago -lt 1){', just now'}else{' {0:N0} running hours ago' -f $ago}}else{''} - switch($t.Last.Outcome){ + # The newest finished test of each kind counts, so a quick check's short test cannot hide a long + # test that failed: that stays a problem until a newer long test passes. + $tests=@(@($t.LongInterrupted,$t.LastLong,$t.ShortInterrupted,$t.LastShort)|Where-Object {$_}) + foreach($test in $tests){ + $kind=if($test.Kind -eq 'Long'){'Long self-test'}else{'Short self-test'} + $when=if($null -ne $test.Hours -and $null -ne $hours){$ago=[double]$hours-[double]$test.Hours;if($ago -lt 1){', just now'}else{' {0:N0} running hours ago' -f $ago}}else{''} + switch($test.Outcome){ 'Passed'{$info.Add("${kind}: passed$when.")} - 'Failed'{$bad.Add("${kind}: failed$when ($($t.Last.Text)).")} - 'Interrupted'{$info.Add("${kind}: did not finish$when ($($t.Last.Text)). Run it again when the drive can stay on.")} + 'Failed'{$bad.Add("${kind}: failed$when ($($test.Text)).$(if($test.Kind -eq 'Long'){' This counts until a newer long self-test passes.'})")} + 'Interrupted'{$info.Add("${kind}: did not finish$when ($($test.Text)). Run it again when the drive can stay on.")} } - }elseif(-not $t.Supported){$info.Add('This drive does not offer self-tests.')} + } + if(-not $tests.Count -and -not $t.Supported){$info.Add('This drive does not offer self-tests.')} $status=if($bad.Count){'Problem'}elseif($warn.Count){'Warning'}else{'OK'} New-DriveGroup $status @(@($bad)+@($warn)+@($info)) } @@ -251,10 +271,61 @@ function Get-DriveCheckView($Report,$Scan) { [pscustomobject]@{Groups=$groups;Overall=Get-WorstStatus @($groups.Values.Status);When=if($Report){$Report.Finished}else{$null}} } +# --- Running as administrator ----------------------------------------------------------------- +# The check runs as administrator. Whoever can change a file it runs or loads, put a file next to +# smartctl.exe, or rename a folder above them could make it run anything as administrator. + +# SHA-256 of bin\smartctl.exe (64-bit) inside the official smartmontools 7.5 installer, whose own +# hash Install-DriveTools.ps1 checks before it runs it. +function Get-DriveCheckSmartToolHash {'B5DB94E5082C042BE44994B7A4FA8F7B5C8E713B2AB1C9A560D8F7A7995EA27D'} + +function Get-DriveCheckPathRisks($Items,[string]$UserSid) { + # Pure rule over what Read-DriveCheckPathAccess reads. Items: Path, Role, Owner (a SID) and Rules + # (Sid, Allow, Rights as a number, InheritOnly). Role Run: a file the check runs or loads, or a + # folder holding them. Above: a folder further up the path, which must not be renamed. Root: the + # drive's root folder, which cannot be renamed itself. Returns each path that an account other than + # this user, SYSTEM, Administrators, TrustedInstaller or the owner placeholders could change, with + # those accounts. Deny rules are ignored, which can only report more, never less. + $trusted=@('S-1-5-18','S-1-5-32-544','S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464','S-1-3-0','S-1-3-4',$UserSid) + $masks=@{ + # Add a file or folder, delete children or itself, change permissions or owner, generic all/write. + Run=(0x2 -bor 0x4 -bor 0x40 -bor 0x10000 -bor 0x40000 -bor 0x80000 -bor 0x10000000 -bor 0x40000000) + # Rename it (delete) or the folder below it (delete children), or take it over. + Above=(0x40 -bor 0x10000 -bor 0x40000 -bor 0x80000 -bor 0x10000000) + Root=(0x40 -bor 0x40000 -bor 0x80000 -bor 0x10000000)} + foreach($i in @($Items)){ + if(-not $masks.ContainsKey([string]$i.Role)){throw "Unknown role $($i.Role) for $($i.Path)"} + $mask=[long]$masks[[string]$i.Role];$who=@() + if($i.Owner -and $i.Owner -notin $trusted){$who+=[string]$i.Owner} + foreach($r in @($i.Rules)){if($r.Allow -and -not $r.InheritOnly -and $r.Sid -notin $trusted -and (([long]$r.Rights -band $mask) -ne 0)){$who+=[string]$r.Sid}} + # A junction or symbolic link leads somewhere whose folders above are not checked here, so a link + # anywhere on the way counts as a risk by itself. + $link=[bool]($i.PSObject.Properties['Link'] -and $i.Link) + if($who.Count -or $link){[pscustomobject]@{Path=[string]$i.Path;Role=[string]$i.Role;Sids=@($who|Sort-Object -Unique);Link=$link}} + } +} + +function Read-DriveCheckPathAccess([string]$ToolRoot) { + # The one function here that reads the computer: owner and access rules of what the check runs or + # loads (Invoke-DriveCheck.ps1, its modules, Measure-BackupDisk.ps1, smartctl and its folder), and + # of every folder above the tool's folder. + $root=[IO.Path]::GetFullPath($ToolRoot).TrimEnd('\') + $run=@($root)+@('Invoke-DriveCheck.ps1','Backup.Status.psm1','Backup.DriveCheck.psm1','Measure-BackupDisk.ps1','bin','bin\smartmontools','bin\smartmontools\bin','bin\smartmontools\bin\smartctl.exe','bin\smartmontools\bin\drivedb.h'|ForEach-Object {Join-Path $root $_}) + $read={param($Path,$Role) + $acl=Get-Acl -LiteralPath $Path + $isLink=[bool]((Get-Item -LiteralPath $Path -Force).Attributes -band [IO.FileAttributes]::ReparsePoint) + [pscustomobject]@{Path=$Path;Role=$Role;Link=$isLink;Owner=$acl.GetOwner([Security.Principal.SecurityIdentifier]).Value + Rules=@($acl.GetAccessRules($true,$true,[Security.Principal.SecurityIdentifier])|ForEach-Object {[pscustomobject]@{Sid=$_.IdentityReference.Value;Allow=($_.AccessControlType -eq 'Allow');Rights=[long]$_.FileSystemRights;InheritOnly=[bool]($_.PropagationFlags -band [Security.AccessControl.PropagationFlags]::InheritOnly)}})} + } + foreach($p in $run){if(Test-Path -LiteralPath $p){& $read $p 'Run'}} + $d=[IO.DirectoryInfo]::new($root).Parent + while($d){& $read $d.FullName $(if($d.Parent){'Above'}else{'Root'});$d=$d.Parent} +} + function Format-DriveAge($When) { if(-not $When){return 'at an unknown time'} $span=(Get-Date)-[datetime]$When if($span.TotalMinutes -lt 2){'just now'}elseif($span.TotalHours -lt 1){'{0:N0} minutes ago' -f $span.TotalMinutes}elseif($span.TotalHours -lt 36){'{0:N0} hours ago' -f $span.TotalHours}else{'{0:N0} days ago' -f $span.TotalDays} } -Export-ModuleMember -Function Get-Field,New-DriveGroup,Get-WorstStatus,Test-DriveSmartAnswer,Get-DriveSelfTest,Get-DriveHealthFindings,Get-DriveSurfaceFindings,Get-DriveConnectionKey,Get-DriveEventFindings,Get-DriveConnectionFindings,Get-DriveFileSystemFindings,Merge-DriveEvents,Get-DriveCheckView,Format-DriveOffset,Format-DriveAge +Export-ModuleMember -Function Get-Field,New-DriveGroup,Get-WorstStatus,Test-DriveSmartAnswer,Get-DriveSelfTest,Get-DriveHealthFindings,Get-DriveSurfaceFindings,Get-DriveConnectionKey,Get-DriveEventFindings,Get-DriveConnectionFindings,Get-DriveFileSystemFindings,Merge-DriveEvents,Get-DriveCheckView,Format-DriveOffset,Format-DriveAge,Get-DriveCheckSmartToolHash,Get-DriveCheckPathRisks,Read-DriveCheckPathAccess diff --git a/Backup.Selection.psm1 b/Backup.Selection.psm1 index 553aae1..907b0c8 100644 --- a/Backup.Selection.psm1 +++ b/Backup.Selection.psm1 @@ -65,6 +65,16 @@ function Test-IsUnder([string]$Path,[string]$Parent) { $Path.StartsWith($Parent+'\',[StringComparison]::OrdinalIgnoreCase) } +function Test-BackupPathInside([string]$Path,[string[]]$Roots) { + # True when the full path Path is one of Roots or lies below one of them. Whole folder names count, + # in any case, trailing backslashes ignored: C:\Data2 is not inside C:\Data. + $p=$Path.TrimEnd('\') + foreach($r in @($Roots|Where-Object {$_}|ForEach-Object {([string]$_).TrimEnd('\')})){ + if([string]::Equals($p,$r,[StringComparison]::OrdinalIgnoreCase) -or $p.StartsWith($r+'\',[StringComparison]::OrdinalIgnoreCase)){return $true} + } + $false +} + function Get-BackupMode($Selection,[string]$Path) { $p=if($Path){ConvertTo-SelectionPath $Path}else{''} $best=$null @@ -112,4 +122,4 @@ function Get-RoboExtraPaths([string[]]$Lines,[string]$Root) { } } -Export-ModuleMember -Function Get-BackupModeNames,Test-BackupModeIncludes,ConvertTo-SelectionPath,Get-BackupSelection,Get-BackupMode,Get-BackupRegions,Get-RoboExtraPaths +Export-ModuleMember -Function Get-BackupModeNames,Test-BackupModeIncludes,ConvertTo-SelectionPath,Get-BackupSelection,Get-BackupMode,Get-BackupRegions,Get-RoboExtraPaths,Test-BackupPathInside diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a4546a..2719b57 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,13 +2,18 @@ ## Unreleased +- Plain copy never copies DevDriveBackup's password and recovery key folders or key files, also when one of them is itself a selected folder or file. robocopy skips folders only below the one it copies, so such a selection used to be copied as it was, readable recovery keys included. The run now names it as not copied, and names an older copy of a private folder still in the plain copy so it can be deleted. +- A new source never gets the plain copy or versions folder of a removed source of the same drive. Removed sources keep their folders on the drive, but only connected drives were checked for names in use, so a folder with the same name added while the drive was unplugged wrote into the old copy, and with *Keep deleted* moved the old source's other files aside. A removed folder added again gets its own folders back, unless another source now uses its repository; then it starts with new ones. +- Before a drive check asks for administrator rights, the window checks whether other accounts can change what it runs (its scripts, `smartctl` and its folder) or rename a folder above them, names those accounts and asks first; a junction or link on the way counts too. The check runs the bundled `smartctl.exe` only when its SHA-256 matches the official 7.5 build and keeps it open, readable only, while it runs; **Install health tool** keeps the installer open in the same way from its checksum check until it has run. +- The drive check judges the newest short and the newest long self-test that passed or failed each on its own; a newer interrupted one is only mentioned. Before, only the newest entry counted, so the short test of a quick check that passed hid a long self-test that had failed. +- After deleting old versions, *Keep versions* frees their space in a separate step and checks the repository again outside the clean-up's own error handling: if that check fails, the backup is reported as failed, because the repository may be damaged. Before, it was only a note on a successful backup. The note now also says how many versions were deleted when freeing their space did not finish. - Renamed from Backup Center to **DevDriveBackup** for the public release. Start it with `DevDriveBackup.cmd`; the window, shortcut, icon and modules carry the new name. New recovery keys go to the "DevDriveBackup recovery keys" folder; an existing "Backup Center recovery keys" folder keeps being used and is still never copied. Re-create the desktop shortcut and re-pin the taskbar button once (`New-DevDriveBackupShortcut.ps1`). - Keep deleted now finds every file gone from the source. robocopy's multi-threaded listing can put a record's path on the next line or start a line with the previous record's path; those files were not recognised, so they stayed in the plain copy and were never moved aside (282 of about 20,000 in one real run). `Get-RoboExtraPaths` reads every field of the listing, and a path that still exists in the source is never moved, checked again just before each move. Tested in `Test-BackupCopyRules.ps1`. -- The window has its own taskbar button with the Backup Center icon instead of PowerShell's, and a pinned button starts Backup Center again (`Backup.Taskbar.cs`). -- **Keep versions** on the overview: after each versioned backup, keep one version a day for N days, a week for N weeks and a month for N months, and delete older versions (`Backup-Worker.ps1 -KeepDays -KeepWeeks -KeepMonths`). The windows count back from the newest version, so a long pause never empties the history. restic plans it as a dry run; `Get-RetentionRemovals` refuses a plan that drops the version just made or every version of a folder set, keeps the newest version of each set protected, and the versions are then deleted by exact ID with `--prune` and the repository checked again. Only this tool's versions (tag `devdrive`) are considered. A failed clean-up is reported on the finished backup, never turned into a failed backup. Default: every version kept. Tested by `Test-BackupRetention.ps1` against a disposable repository with versions over 200 days. -- **Look** page: themes Light, Dark, Sand, Ocean and High contrast, and *Custom* from four chosen colours (window background, panels, text, main buttons) with secondary text, lines, selection and button text derived to stay readable. Every built-in theme passes WCAG 2.2 contrast for all text the window draws; for a custom theme the page names any pair below the minimum. Dark themes also use Windows' dark title bar, scroll bars and .NET's dark drawing of number boxes and drop-downs. The choice is saved in `%LOCALAPPDATA%\DevDriveMirror\center\appearance.json`. Colour logic in `BackupCenter.Theme.psm1`, tested by `Test-BackupTheme.ps1`. +- The window has its own taskbar button with the DevDriveBackup icon instead of PowerShell's, and a pinned button starts DevDriveBackup again (`Backup.Taskbar.cs`). +- **Keep versions** on the overview: after each versioned backup, keep one version a day for N days, a week for N weeks and a month for N months, and delete older versions (`Backup-Worker.ps1 -KeepDays -KeepWeeks -KeepMonths`). The windows count back from the newest version, so a long pause never empties the history. restic plans it as a dry run; `Get-RetentionRemovals` refuses a plan that drops the version just made or every version of a folder set, keeps the newest version of each set protected, and the versions are then deleted by exact ID, their space freed with `prune` and the repository checked again. Only this tool's versions (tag `devdrive`) are considered. A failed clean-up is reported on the finished backup, never turned into a failed backup. Default: every version kept. Tested by `Test-BackupRetention.ps1` against a disposable repository with versions over 200 days. +- **Look** page: themes Light, Dark, Sand, Ocean and High contrast, and *Custom* from four chosen colours (window background, panels, text, main buttons) with secondary text, lines, selection and button text derived to stay readable. Every built-in theme passes WCAG 2.2 contrast for all text the window draws; for a custom theme the page names any pair below the minimum. Dark themes also use Windows' dark title bar, scroll bars and .NET's dark drawing of number boxes and drop-downs. The choice is saved in `%LOCALAPPDATA%\DevDriveMirror\center\appearance.json`. Colour logic in `DevDriveBackup.Theme.psm1`, tested by `Test-BackupTheme.ps1`. - The window scales: a larger or maximised window enlarges every position, size and font by the same factor and centres the result; the smallest size is the original 1200 x 800. -- New program icon: the drive label with its status stripe, redrawn as vector with a separate small-size drawing for 16 to 24 px and a light rim that keeps it visible on dark taskbars. Sources, the three routes considered and their contact sheets are in `design/icon`. +- New program icon: the drive label with its status stripe, redrawn as vector with a separate small-size drawing for 16 to 24 px and a light rim that keeps it visible on dark taskbars. Sources, the three routes considered and their contact sheets are in `design/icon`; `New-DevDriveBackupIcon.ps1` draws the icon from those sources (it still drew the 0.1.0 icon). - The light theme's secondary text is a little darker (#56636A), so it reaches 4.5:1 on a selected row too (was 4.31:1). - The speed page no longer hides "No speed test yet..." behind the empty result labels. Disabled buttons stay readable on dark themes. The page tabs are text, not buttons. - Opening the window lists the drives once instead of three times. @@ -18,7 +23,7 @@ - `Install-DriveTools.ps1` (and **Install health tool**) downloads smartmontools 7.5 with a pinned SHA-256 and installs only `smartctl` and its drive database into `bin\smartmontools`. - `Test-PrivacyGuard.ps1` blocks private backup details (volume identifiers, profile paths, personal email, drive models and labels, folder names from your `*.local.json`) from commits, commit messages and pushes; hooks in `.githooks`, also run in CI. - Plain copy no longer stops after copying when robocopy writes two "extra" records on one line during multi-threaded listing. -- Plain copy and versions never copy Backup Center's own folders of passwords, recovery keys and state, even when the source holds them (for example a whole user folder or drive). Before, a plain copy could put the readable recovery keys on the backup drive. +- Plain copy and versions never copy DevDriveBackup's own folders of passwords, recovery keys and state, even when the source holds them (for example a whole user folder or drive). Before, a plain copy could put the readable recovery keys on the backup drive. - *Check backup content* now also finds files that are missing from the plain copy, not only files that differ. It no longer creates an empty copy folder, and it no longer counts as a completed backup. - Plain copy refuses to run when the drive letter now belongs to a different drive than the profile's backup drive. - A folder inside a *NeverBackUp* folder is never backed up, even if it is marked for backup itself. @@ -30,9 +35,10 @@ - The content check skips the folders and names the copy skips, so a folder excluded later no longer shows up as a difference. - Saving a source's settings waits for, and never overwrites, a password change in progress; if a backup of that source is running, it says so instead. - **Install health tool** and **Long self-test** now see an installed smartctl (a stray character in its path made the window think it was missing). -- A load level changed during a run now applies to the next step, threads, priority and versions settings alike. A step that cannot start or whose result cannot be recorded is reported as failed instead of leaving the run stuck. Step logs no longer overwrite each other. *Back up selected source* prepares only that source. +- A load level changed during a run now applies to the next step, threads, priority and versions settings alike; the running step's priority changes at once. A step that cannot start or whose result cannot be recorded is reported as failed instead of leaving the run stuck. Step logs no longer overwrite each other. *Back up selected source* prepares only that source. - Closing the window asks about a running backup and a running drive check first, and stops nothing unless it really closes. A drive check cannot start while an earlier one (also from a closed window) still runs, and its stop request is kept. A profile that is being checked or speed-tested cannot be removed. - The speed test button stays off while a test runs, and the result is kept after **Refresh**. +- The documentation matches the code again: BACKUP-SYSTEM.md describes the automatic clean-up of old versions, the main window's load levels, the real button names and the drive check thresholds, and the README lists every program, test and document file. The checks on every push also build `Backup.Taskbar.cs`, run `Test-BackupResourceControls.ps1` and use `actions/checkout@v7`. Two tests no longer leave an empty folder in `%TEMP%` after each run. - Look and feel: disabled main buttons stay readable; small status text uses darker colours that meet 4.5:1 contrast; long names end in "…"; links are underlined; the Overview shows the last drive check and why buttons are off when the drive is not connected; drive check findings get more room and scroll only when needed; the Help page has headings. ## 0.1.0 — first public release diff --git a/DevDriveBackup.Data.psm1 b/DevDriveBackup.Data.psm1 index c09b89b..178af91 100644 --- a/DevDriveBackup.Data.psm1 +++ b/DevDriveBackup.Data.psm1 @@ -156,21 +156,36 @@ function Add-CenterSource { # Names that are not Latin letters all become "Source"; keep them apart. if($safe -eq 'Source'){$safe='Source-'+(New-ShortId).Substring(0,4)} $drive=Get-CenterDrive $profile.DestinationVolumeId - # A name is taken when another source of this drive has it, or its plain copy or versions folder - # is already on the drive (an earlier source's copy is never shared). The label is compared by its - # safe name, so a second folder with the same label gets a new name instead of looping forever. - $taken={param($n)@($profile.Sources|Where-Object {$_.MirrorFolder -ieq $n+'-Mirror' -or (Get-SafeName $_.Label) -ieq $n}).Count -or ($drive.Connected -and ((Test-Path -LiteralPath ($drive.Letter+':\'+$n+'-Versions')) -or (Test-Path -LiteralPath ($drive.Letter+':\'+$n+'-Mirror'))))} + # The folder names this drive's job files record, also those of sources removed earlier: their plain + # copy and versions stay on the drive, so a new source must never get one, even while it is unplugged. + $jobFiles=@(Get-ChildItem -LiteralPath (Get-CenterPaths).Jobs -Filter '*.local.json' -File -ErrorAction SilentlyContinue|ForEach-Object { + try{$j=Get-Content -LiteralPath $_.FullName -Raw|ConvertFrom-Json}catch{return} + if($j.PSObject.Properties['DestinationVolumeId'] -and $j.DestinationVolumeId -eq $profile.DestinationVolumeId){[pscustomobject]@{Path=$_.FullName;Job=$j}} + }) + $jobNames=@($jobFiles|ForEach-Object { + if($_.Job.PSObject.Properties['MirrorFolder'] -and $_.Job.MirrorFolder){[string]$_.Job.MirrorFolder} + if($_.Job.PSObject.Properties['RepositoryFolder'] -and $_.Job.RepositoryFolder){[string]$_.Job.RepositoryFolder;if([string]$_.Job.RepositoryFolder -match '^(.+)-Versions$'){$Matches[1]+'-Mirror'}} + }) + # A name is taken when another source of this drive has it, a job file of this drive records it, or + # its plain copy or versions folder is already on the drive (an earlier source's copy is never shared). + # The label is compared by its safe name, so a second folder with the same label gets a new name + # instead of looping forever. + $taken={param($n)@($profile.Sources|Where-Object {$_.MirrorFolder -ieq $n+'-Mirror' -or (Get-SafeName $_.Label) -ieq $n}).Count -or ($jobNames -icontains ($n+'-Mirror')) -or ($jobNames -icontains ($n+'-Versions')) -or ($drive.Connected -and ((Test-Path -LiteralPath ($drive.Letter+':\'+$n+'-Versions')) -or (Test-Path -LiteralPath ($drive.Letter+':\'+$n+'-Mirror'))))} $used={param($m)@($profile.Sources|Where-Object {$_.MirrorFolder -ieq $m}).Count -gt 0} if($MirrorFolder -and (& $used $MirrorFolder)){throw "The plain copy folder $MirrorFolder is already used by another source of this drive"} $id=New-ShortId if(-not $ExistingConfigPath){ # A source removed earlier keeps its job file (with its keys); adding the same folder again to - # the same drive picks it up instead of creating a second repository next to the first. - $jobs=(Get-CenterPaths).Jobs - $old=@(Get-ChildItem -LiteralPath $jobs -Filter '*.local.json' -File -ErrorAction SilentlyContinue|Where-Object { - try{$j=Get-Content -LiteralPath $_.FullName -Raw|ConvertFrom-Json;(([IO.Path]::GetFullPath([string]$j.Source).TrimEnd('\')+'\') -ieq $root) -and $j.DestinationVolumeId -eq $profile.DestinationVolumeId}catch{$false} - }|Where-Object {$f=$_.FullName;-not @($profile.Sources|Where-Object {$_.ConfigPath -ieq $f}).Count}|Sort-Object LastWriteTime -Descending) - if($old.Count){$ExistingConfigPath=$old[0].FullName} + # the same drive picks it up instead of creating a second repository next to the first. A job whose + # versions folder a current source of this drive uses now is not picked up: two sources never share + # one repository, so the folder then starts with names of its own. + $current=@($profile.Sources|Where-Object {$_.PSObject.Properties['ConfigPath'] -and $_.ConfigPath}|ForEach-Object {[IO.Path]::GetFullPath([string]$_.ConfigPath)}) + # Read from each current source's own job file, wherever it is kept, not only those in the jobs folder. + $inUse=@($current|ForEach-Object {try{$j=Get-Content -LiteralPath $_ -Raw|ConvertFrom-Json;if($j.PSObject.Properties['RepositoryFolder'] -and $j.RepositoryFolder){[string]$j.RepositoryFolder}}catch{}}) + $old=@($jobFiles|Where-Object { + try{(([IO.Path]::GetFullPath([string]$_.Job.Source).TrimEnd('\')+'\') -ieq $root)}catch{$false} + }|Where-Object {$current -inotcontains $_.Path -and -not ($_.Job.PSObject.Properties['RepositoryFolder'] -and $inUse -icontains [string]$_.Job.RepositoryFolder)}|Sort-Object {(Get-Item -LiteralPath $_.Path).LastWriteTime} -Descending) + if($old.Count){$ExistingConfigPath=$old[0].Path} } if(-not $ExistingConfigPath){while(& $taken $safe){$safe=(Get-SafeName $Label)+'-'+(New-ShortId).Substring(0,4)}} if($ExistingConfigPath){ @@ -435,6 +450,11 @@ function Complete-CenterStep($Running,[string]$ProfileId) { if($r.PSObject.Properties['Verify'] -and $r.Verify){$e.CheckedFiles=$r.Verify.FilesMatched;$verifyOk=$r.Verify.Differences -eq 0;if(-not $verifyOk){$e.Error="$($r.Verify.Differences) files differ from the source; see verify-differences.txt in $($r.RunFolder)"}} if($r.CopyErrors -gt 0){$e.Error="$($r.CopyErrors) files could not be copied; see copy-errors.txt in $($r.RunFolder)"} $e.Success=($p.ExitCode -eq 0 -and $r.CopyErrors -eq 0 -and $verifyOk) + # DevDriveBackup's own password and recovery key folders are never copied, even when selected. + $notes=@() + if($r.PSObject.Properties['SelectionsSkippedAsPrivate'] -and @($r.SelectionsSkippedAsPrivate).Count){$notes+="Not copied, because DevDriveBackup never copies its own password and recovery key folders: $(@($r.SelectionsSkippedAsPrivate) -join ', ')."} + if($r.PSObject.Properties['PrivateCopiesOnDrive'] -and @($r.PrivateCopiesOnDrive).Count){$notes+="An older copy of DevDriveBackup's private folders is still in the plain copy and holds readable recovery keys: $(@($r.PrivateCopiesOnDrive) -join ', '). Delete it from the backup drive."} + if($notes.Count){$e.Note=$notes -join ' '} }else{ $resultPath=Join-Path $s.RunDirectory 'result.json' if(-not(Test-Path -LiteralPath $resultPath)){throw ($stderr.Trim() -replace '\s+',' ')} @@ -442,10 +462,14 @@ function Complete-CenterStep($Running,[string]$ProfileId) { $e.Details=$s.RunDirectory if($r.State -eq 'Completed'){$e.Success=$true;if($r.Progress -and $r.Progress.PSObject.Properties['files_new']){$e.CopiedFiles=[long]$r.Progress.files_new+[long]$r.Progress.files_changed;$e.CopiedBytes=[long]$r.Progress.data_added}} else{$e.Error=[string]$r.Error} - # The automatic clean-up of old versions: a note, never a failed backup. + # The automatic clean-up of old versions: a note. Only a failed check of the repository after it + # fails the backup, and the worker reports that as the run's error. if($r.PSObject.Properties['Retention'] -and $r.Retention){ - $e.Note=if($r.Retention.PSObject.Properties['Error'] -and $r.Retention.Error){'Backed up, but old versions were not cleaned up: '+$r.Retention.Error} - elseif($r.Retention.Removed -gt 0){"Deleted $($r.Retention.Removed) old versions; $($r.Retention.Kept) kept."}else{"No old versions to delete; $($r.Retention.Kept) kept."} + $ret=$r.Retention + $err=if($ret.PSObject.Properties['Error'] -and $ret.Error){[string]$ret.Error}else{''} + $removed=if($ret.PSObject.Properties['Removed'] -and $ret.Removed){[int]$ret.Removed}else{0} + $kept=if($ret.PSObject.Properties['Kept'] -and $null -ne $ret.Kept){"; $($ret.Kept) kept"}else{''} + $e.Note=if($err -and $removed){"Deleted $removed old versions$kept, but the clean-up did not finish: $err"}elseif($err){"Backed up, but old versions were not cleaned up: $err"}elseif($removed){"Deleted $removed old versions$kept."}else{"No old versions to delete$kept."} } } }catch{$e.Error=$_.Exception.Message} diff --git a/DevDriveBackup.ps1 b/DevDriveBackup.ps1 index f10aa15..c514caf 100644 --- a/DevDriveBackup.ps1 +++ b/DevDriveBackup.ps1 @@ -95,7 +95,7 @@ Profiles: each backup drive has its own profile, recognised by the drive itself Sources: any folder on this computer, such as a whole drive, a project or Documents. Inside each source you choose per folder how it is backed up. -Where things are on the backup drive: plain copies are in one folder per source (shown on the "What to back up" tab, for example DevDrive-Mirror). Versions are in one encrypted repository per source (for example DevDrive-Backup). +Where things are on the backup drive: plain copies are in one folder per source (shown on the "What to back up" tab, for example DevDrive-Mirror). Versions are in one encrypted repository per source (for example DevDrive-Versions). Keys: versions are encrypted. The first versioned backup of a source creates a recovery key file in the "DevDriveBackup recovery keys" folder in your user folder. Copy it to a password manager or print it. Without it, versions cannot be opened on another computer. @@ -426,7 +426,7 @@ $lookNote=New-Note $tLook 24 496 640 142 $Help.Look;$lookNote.ScrollBars='None' $script:data=Read-CenterProfiles $script:queue=[Collections.Generic.Queue[object]]::new();$script:current=$null;$script:runResults=@();$script:runProfile=$null;$script:totalSteps=0;$script:waiting=$null $script:lastFree=$null;$script:lastTick=$null;$script:loadingModes=$false;$script:meterRatio=0;$script:shown=$false;$script:zoomedFor=$null -$script:check=$null;$script:install=$null +$script:check=$null;$script:install=$null;$script:adminRiskAccepted=$false $smartTool=Join-Path $PSScriptRoot 'bin\smartmontools\bin\smartctl.exe' $checkColor=@{};$checkInk=@{} function Update-CheckColors { @@ -576,6 +576,19 @@ function Start-DriveCheck([string]$Mode){ if([Windows.Forms.MessageBox]::Show(("Read every part of this drive?`n`nAt a typical 150 MB/s this takes about {0:N0} hours. It only reads; you can keep working, stop it any time and continue later. The computer stays awake while it runs." -f [math]::Max(1,$hours)),'DevDriveBackup','YesNo','Question') -ne 'Yes'){return} } if($Mode -eq 'LongSelfTest' -and [Windows.Forms.MessageBox]::Show("Ask the drive to read itself completely?`n`nThe drive does this on its own and it takes hours. Keep it connected and the computer awake; a USB box that goes to sleep stops the test. Run a quick check afterwards to see the result.",'DevDriveBackup','YesNo','Question') -ne 'Yes'){return} + # The check runs as administrator. If other accounts can change what it runs, or rename a folder above + # it, they could run anything as administrator through it: say so before Windows is asked, once per window. + if(-not $script:adminRiskAccepted){ + $risks=@(Get-DriveCheckPathRisks (Read-DriveCheckPathAccess $PSScriptRoot) ([Security.Principal.WindowsIdentity]::GetCurrent().User.Value)) + if($risks.Count){ + $names=@($risks|ForEach-Object {$_.Sids}|Sort-Object -Unique|ForEach-Object {$sid=$_;try{([Security.Principal.SecurityIdentifier]::new($sid)).Translate([Security.Principal.NTAccount]).Value}catch{$sid}}) + $where=@($risks|Select-Object -First 3|ForEach-Object {if($_.Link){$_.Path+' (a link to another place)'}else{$_.Path}})+@(if($risks.Count -gt 3){"and $($risks.Count-3) more"}) + if(-not $names.Count){$names=@('none found, but a link on the way leads to folders that were not checked')} + $ask="Other accounts on this computer can change the files this check runs as administrator, or rename a folder above them:`n`n$($where -join "`n")`n`nAccounts: $($names -join ', ')`n`nThat includes accounts other people or sandboxed tools use. If someone else could use one of them, choose No and move DevDriveBackup into your own user folder first (see README). If only you use this computer, the check is safe.`n`nRun the check as administrator anyway?" + if([Windows.Forms.MessageBox]::Show($ask,'DevDriveBackup','YesNo','Warning','Button2') -ne 'Yes'){$checkStatus.Text='Not started: other accounts can change the drive check files. Move DevDriveBackup into your user folder, then try again.';return} + $script:adminRiskAccepted=$true + } + } $dir=Get-CheckFolder $p;New-Item -ItemType Directory -Path $dir -Force|Out-Null # A check keeps check.lock open while it runs, also one started by an earlier window that was closed. try{([IO.File]::Open((Join-Path $dir 'check.lock'),'OpenOrCreate','ReadWrite','None')).Dispose()}catch{throw 'A drive check of this drive is still running or finishing. Try again in a minute.'} @@ -691,7 +704,7 @@ function Finish-Run { $ok=@($script:runResults|Where-Object {$_.Success}).Count;$bad=@($script:runResults|Where-Object {-not $_.Success}) $msg="$ok of $($script:runResults.Count) steps finished successfully." foreach($b in $bad){$msg+="`n`n$($b.Label) ($($b.Kind)): $($b.Error)"} - foreach($n in @($script:runResults|Where-Object {$_.Success -and $_.PSObject.Properties['Note'] -and $_.Note})){$msg+="`n`n$($n.Label), versions: $($n.Note)"} + foreach($n in @($script:runResults|Where-Object {$_.Success -and $_.PSObject.Properties['Note'] -and $_.Note})){$msg+="`n`n$($n.Label), $(if($n.Kind -like 'Mirror*'){'plain copy'}else{'versions'}): $($n.Note)"} # A first versioned backup creates a recovery key; say where it is so it gets copied somewhere safe. foreach($r in @($script:runResults|Where-Object {$_.Kind -eq 'Versioned' -and $_.Success})){ $src=@($script:runProfile.Sources|Where-Object {$_.Id -eq $r.SourceId})[0] diff --git a/Install-DriveTools.ps1 b/Install-DriveTools.ps1 index 48fae4d..4d65957 100644 --- a/Install-DriveTools.ps1 +++ b/Install-DriveTools.ps1 @@ -20,10 +20,21 @@ if(-not(Test-Path -LiteralPath $setup)){ Move-Item -LiteralPath $part -Destination $setup }finally{if(Test-Path -LiteralPath $part){Remove-Item -LiteralPath $part}} } -if((Get-FileHash -LiteralPath $setup -Algorithm SHA256).Hash -ne $expected){throw 'The health tool installer in bin does not match the pinned checksum. Delete it and run this script again.'} -# NSIS rules: /D= comes last and is never quoted, even when the path has spaces. -$p=Start-Process -FilePath $setup -ArgumentList "/S /SO x64,smartctl,drivedb /D=$target" -Verb RunAs -Wait -PassThru +# The installer runs as administrator. It stays open (others may only read it) from the checksum +# check until it has finished, so the file that runs is the file that was checked. +$hold=[IO.File]::Open($setup,'Open','Read','Read') +try{ + if((Get-FileHash -InputStream $hold -Algorithm SHA256).Hash -ne $expected){throw 'The health tool installer in bin does not match the pinned checksum. Delete it and run this script again.'} + # NSIS rules: /D= comes last and is never quoted, even when the path has spaces. + $p=Start-Process -FilePath $setup -ArgumentList "/S /SO x64,smartctl,drivedb /D=$target" -Verb RunAs -Wait -PassThru +}finally{$hold.Dispose()} if($p.ExitCode){throw "The health tool installer stopped with code $($p.ExitCode)."} -$v=& $tool -V +# The drive check runs only the official build, so the installed file is checked the same way. +Import-Module (Join-Path $PSScriptRoot 'Backup.DriveCheck.psm1') -Force +$toolHold=[IO.File]::Open($tool,'Open','Read','Read') +try{ + if((Get-FileHash -InputStream $toolHold -Algorithm SHA256).Hash -ne (Get-DriveCheckSmartToolHash)){throw 'Health tool self-check failed: the installed smartctl.exe is not the official 7.5 build.'} + $v=& $tool -V +}finally{$toolHold.Dispose()} if($LASTEXITCODE -ne 0 -or -not ($v -match "smartctl $([regex]::Escape($version))")){throw 'Health tool self-check failed'} Write-Output ($v|Select-Object -First 1) diff --git a/Invoke-DriveCheck.ps1 b/Invoke-DriveCheck.ps1 index 7f7b880..a5e7240 100644 --- a/Invoke-DriveCheck.ps1 +++ b/Invoke-DriveCheck.ps1 @@ -6,7 +6,8 @@ # Surface reads every part of the disk; stops on request and later continues from there # LongSelfTest asks the drive to read itself completely. The drive works alone for hours; # the next quick check shows the result. -# Only reads. Nothing on the drive is written, moved or repaired. +# Only reads. Nothing on the drive is written, moved or repaired. The bundled smartctl runs only +# when its SHA-256 matches the official 7.5 build. # Into -OutputDirectory it writes progress.json while it runs and run.json at the end, plus # report.json (quick check) and surface.json (full scan). A file named stop.request ends the run. param( @@ -27,6 +28,18 @@ $started=Get-Date # here without touching the first one's files. try{$lock=[IO.File]::Open((Join-Path $OutputDirectory 'check.lock'),'OpenOrCreate','ReadWrite','None')} catch{[Console]::Error.WriteLine('Another drive check of this drive is running.');exit 1} +# smartctl runs as administrator here, so the bundled copy must be the official 7.5 build. It stays +# open (others may only read it) until this check ends, so it cannot be replaced after the check. +# A tool that is missing now is never run later in this check, even if a file appears meanwhile. +$script:smartProblem='The health tool is not installed. Use "Install health tool" on this page, then check again.' +if(Test-Path -LiteralPath $SmartTool -PathType Leaf){ + $script:smartProblem='' + $script:smartHold=[IO.File]::Open($SmartTool,'Open','Read','Read') + $bundled=[IO.Path]::GetFullPath($SmartTool) -ieq [IO.Path]::GetFullPath((Join-Path $PSScriptRoot 'bin\smartmontools\bin\smartctl.exe')) + if($bundled -and (Get-FileHash -InputStream $script:smartHold -Algorithm SHA256).Hash -ne (Get-DriveCheckSmartToolHash)){ + $script:smartProblem='The health tool in bin\smartmontools is not the official smartctl 7.5, so it was not run. Use "Install health tool" on this page again, then check again.' + } +} Add-Type -TypeDefinition @' using System; using System.Collections.Generic; using System.ComponentModel; using System.Diagnostics; using System.Globalization; using System.Runtime.InteropServices; using System.Text; using Microsoft.Win32.SafeHandles; @@ -93,12 +106,14 @@ function Complete-Check([bool]$Success,[string]$Message){ # A request left over from an earlier run does not count, only one made after this run started. function Test-StopRequested{(Test-Path -LiteralPath $stopFile) -and (Get-Item -LiteralPath $stopFile).LastWriteTime -ge $started} function Invoke-Smart([string[]]$Arguments){ + if($script:smartProblem){return [pscustomobject]@{Code=-1;Json=$null}} $out=& $SmartTool @Arguments '--json' 2>$null;$code=$LASTEXITCODE;$json=$null try{$json=($out -join "`n")|ConvertFrom-Json}catch{} [pscustomobject]@{Code=$code;Json=$json} } function Get-SmartReport{ - if(-not(Test-Path -LiteralPath $SmartTool)){return [pscustomobject]@{Json=$null;Type=$null;Why='The health tool is not installed. Use "Install health tool" on this page, then check again.'}} + # Decided once at the start: missing, not the official build, or held open and checked. + if($script:smartProblem){return [pscustomobject]@{Json=$null;Type=$null;Why=$script:smartProblem}} # First the type smartctl detects itself; many USB boxes answer only when the standard SAT pass-through is named. foreach($t in @('auto','sat')){ $r=Invoke-Smart @('-a','-d',$t,$script:smartDevice) diff --git a/Invoke-FastMirror.ps1 b/Invoke-FastMirror.ps1 index 8d6026f..f889b34 100644 --- a/Invoke-FastMirror.ps1 +++ b/Invoke-FastMirror.ps1 @@ -10,7 +10,8 @@ # in the mirror and are listed in the summary. -Verify (default on the first run) re-reads # both copies and compares SHA-256 for every selected file; a selected source file missing from # the mirror is a difference. A selected folder that is missing or became a link fails the run. -# DevDriveBackup's own password, key and state folders are never copied. +# DevDriveBackup's own password, key and state folders are never copied, even when one of them is +# selected itself; an older copy of one found on the backup drive is reported. param( [string]$ConfigPath=(Join-Path $PSScriptRoot 'backup.local.json'), [string]$Source, @@ -61,9 +62,11 @@ foreach($entry in @(if($config.PSObject.Properties['Exclude']){$config.Exclude}) # copied, whatever the selection: the copy would not keep their private permissions. The job's own key # files count too, in case the configuration keeps them somewhere else. $privatePaths=@((Join-Path $env:LOCALAPPDATA 'DevDriveMirror'),(Join-Path $env:USERPROFILE 'DevDriveBackup recovery keys'),(Join-Path $env:USERPROFILE 'Backup Center recovery keys'))+@('PasswordDpapiPath','RecoveryKeyPath'|Where-Object {$config.PSObject.Properties[$_] -and $config.$_}|ForEach-Object {[string]$config.$_}) -foreach($p in @($privatePaths|ForEach-Object {[IO.Path]::GetFullPath($_).TrimEnd('\')})){ +$privateRoots=@($privatePaths|ForEach-Object {[IO.Path]::GetFullPath($_).TrimEnd('\')}) +$privateInSource=@() +foreach($p in $privateRoots){ if($srcRoot.StartsWith($p+'\',[StringComparison]::OrdinalIgnoreCase)){throw "The source is inside $p, which holds DevDriveBackup's passwords and recovery keys and is never copied"} - if($p.StartsWith($srcRoot,[StringComparison]::OrdinalIgnoreCase)){$skipPaths+=$p.Substring($srcRoot.Length)} + if($p.StartsWith($srcRoot,[StringComparison]::OrdinalIgnoreCase)){$skipPaths+=$p.Substring($srcRoot.Length);$privateInSource+=$p.Substring($srcRoot.Length)} } $src=[IO.Path]::GetFullPath($Source).TrimEnd('\')+'\' @@ -126,10 +129,14 @@ function Get-RoboTotals([string]$Log){ function Get-LogName([string]$Relative){'copy-'+$(if($Relative){$Relative -replace '[\\/:*?"<>|]','_'}else{'main'})+'.log'} $started=Get-Date -$summary=[ordered]@{Source=$src;Destination=$dst;Started=$started.ToString('o');DefaultMode=$selection.DefaultMode;Selected=@($regions|ForEach-Object {if($_.Path){$_.Path}else{'(source root)'}});Runs=@();SelectionsMissing=@();SelectionsSkippedAsLinks=@()} +$summary=[ordered]@{Source=$src;Destination=$dst;Started=$started.ToString('o');DefaultMode=$selection.DefaultMode;Selected=@($regions|ForEach-Object {if($_.Path){$_.Path}else{'(source root)'}});Runs=@();SelectionsMissing=@();SelectionsSkippedAsLinks=@();SelectionsSkippedAsPrivate=@()} $copied=@() foreach($region in $regions){ $from=Join-Rel $src $region.Path;$to=Join-Rel $dst $region.Path + # /XD only skips folders below the one robocopy copies, never that folder itself, and a file selected + # on its own gets no /XF at all. So a selection that is a private folder or key file, or lies inside + # one, is skipped here, whatever its mode says. + if($region.Path -and (Test-BackupPathInside ([IO.Path]::GetFullPath($from)) $privateRoots)){$summary.SelectionsSkippedAsPrivate+=$region.Path;continue} if(-not(Test-Path -LiteralPath $from)){$summary.SelectionsMissing+=$region.Path;continue} $item=Get-Item -LiteralPath $from -Force if($region.Path -and ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)){$summary.SelectionsSkippedAsLinks+=$region.Path;continue} @@ -143,6 +150,13 @@ foreach($region in $regions){ if(-not $VerifyOnly){$summary.Runs+=[ordered]@{Part=$part;ExitCode=(Invoke-Robo $a $part);Log=$log}} $copied+=[pscustomobject]@{Region=$region;IsFolder=$item.PSIsContainer} } +# An earlier version could copy a private folder. Such a copy holds readable recovery keys; it is +# reported so it can be deleted by hand, never deleted here. +$summary.PrivateCopiesOnDrive=@($privateInSource|Where-Object {$_ -and (Test-Path -LiteralPath (Join-Rel $dst $_))}) +if($copied.Count -eq 0 -and $summary.SelectionsSkippedAsPrivate.Count){ + $old=if($summary.PrivateCopiesOnDrive.Count){' An older copy of one is still on the backup drive and holds readable recovery keys; delete it: '+(@($summary.PrivateCopiesOnDrive|ForEach-Object {Join-Rel $dst $_}) -join ', ')+'.'}else{''} + throw ("Everything selected is one of DevDriveBackup's private folders, which hold passwords and recovery keys and are never copied: "+($summary.SelectionsSkippedAsPrivate -join ', ')+'.'+$old) +} if($copied.Count -eq 0){throw ('None of the selected paths exist under '+$src+': '+($summary.Selected -join ', '))} # robocopy gives a folder copied from a drive root the root's hidden/system attributes. if(-not $VerifyOnly){& attrib.exe -s -h $dst|Out-Null} diff --git a/New-DevDriveBackupIcon.ps1 b/New-DevDriveBackupIcon.ps1 index a8664b4..c42cbe2 100644 --- a/New-DevDriveBackupIcon.ps1 +++ b/New-DevDriveBackupIcon.ps1 @@ -1,60 +1,64 @@ #Requires -Version 7.2 -# Draws the DevDriveBackup icon: a slate drive label with a green status stripe and two -# label lines, the same marks the window uses for a backup drive. Writes DevDriveBackup.ico -# with bitmap frames for small sizes and a PNG frame for 256 px. +# Draws the DevDriveBackup icon from its vector sources in design\icon: a slate drive label with a +# green status stripe, label lines and an activity light, and a light rim that keeps it visible on +# a dark taskbar. 16 to 24 px use the separate small drawing (dev-drive-backup-small.svg), larger +# sizes the full one (dev-drive-backup.svg). Writes DevDriveBackup.ico with a PNG frame per size. param([string]$OutputPath=(Join-Path $PSScriptRoot 'DevDriveBackup.ico'),[string]$PreviewPng) $ErrorActionPreference='Stop' Add-Type -AssemblyName System.Drawing -$ink=[Drawing.Color]::FromArgb(31,42,46);$green=[Drawing.Color]::FromArgb(56,166,104);$line=[Drawing.Color]::FromArgb(214,224,226) +# The two drawings as in the SVG files: x, y, width, height and corner radius on their own grid. +# A circle is a square whose corner radius is half its side. +$large=@{Grid=256;Body=@(20,40,216,176,30);Rim=8;Stripe=@(46,66,34,124,17);Light=@(181,159,30,30,15) + Lines=@(@(104,78,104,26,13,'#ECF0EF'),@(104,124,66,26,13,'#7E9298'))} +$small=@{Grid=16;Body=@(0.5,2.5,15,11,2.5);Rim=1;Stripe=@(3,5,2,6,1);Light=@(11,9,2,2,1) + Lines=@(,@(7,5,6,2,1,'#ECF0EF'))} -function New-RoundedPath([single]$X,[single]$Y,[single]$W,[single]$H,[single]$R){ - $p=[Drawing.Drawing2D.GraphicsPath]::new();$d=$R*2 - $p.AddArc($X,$Y,$d,$d,180,90);$p.AddArc($X+$W-$d,$Y,$d,$d,270,90);$p.AddArc($X+$W-$d,$Y+$H-$d,$d,$d,0,90);$p.AddArc($X,$Y+$H-$d,$d,$d,90,90);$p.CloseFigure();$p +function New-RoundedPath([double[]]$R){ + $x=[single]$R[0];$y=[single]$R[1];$w=[single]$R[2];$h=[single]$R[3];$d=[single]([math]::Min(2*$R[4],[math]::Min($w,$h))) + $p=[Drawing.Drawing2D.GraphicsPath]::new() + $p.AddArc($x,$y,$d,$d,180,90);$p.AddArc($x+$w-$d,$y,$d,$d,270,90);$p.AddArc($x+$w-$d,$y+$h-$d,$d,$d,0,90);$p.AddArc($x,$y+$h-$d,$d,$d,90,90);$p.CloseFigure();$p } +function Use-Brush([string]$Hex,[scriptblock]$Draw){$b=[Drawing.SolidBrush]::new([Drawing.ColorTranslator]::FromHtml($Hex));try{& $Draw $b}finally{$b.Dispose()}} function New-Frame([int]$Size){ + $d=if($Size -le 24){$small}else{$large} $bmp=[Drawing.Bitmap]::new($Size,$Size,[Drawing.Imaging.PixelFormat]::Format32bppArgb) - $g=[Drawing.Graphics]::FromImage($bmp);$g.SmoothingMode='AntiAlias';$g.PixelOffsetMode='HighQuality';$g.Clear([Drawing.Color]::Transparent) - $s=$Size/32.0 - # Body: a drive seen from the front, wider than tall. - $body=New-RoundedPath (1*$s) (5*$s) (30*$s) (22*$s) ([math]::Max(1.5,3.2*$s)) - $g.FillPath([Drawing.SolidBrush]::new($ink),$body) - # A light edge keeps the shape visible on a dark taskbar. - $g.DrawPath([Drawing.Pen]::new([Drawing.Color]::FromArgb(140,150,160),[single][math]::Max(1,0.9*$s)),$body) - # Status stripe, as in the drive list. - $g.FillRectangle([Drawing.SolidBrush]::new($green),[single](5*$s),[single](9*$s),[single][math]::Max(2,3*$s),[single](14*$s)) - # Label lines; at 16 px a single line keeps the shape readable. - $lh=[single][math]::Max(1.5,2.4*$s) - $g.FillRectangle([Drawing.SolidBrush]::new($line),[single](11*$s),[single](10.5*$s),[single](15*$s),$lh) - if($Size -ge 24){$g.FillRectangle([Drawing.SolidBrush]::new([Drawing.Color]::FromArgb(150,$line)),[single](11*$s),[single](16*$s),[single](10*$s),$lh)} - # Activity light, bottom right. - if($Size -ge 32){$g.FillEllipse([Drawing.SolidBrush]::new($green),[single](24.2*$s),[single](20.2*$s),[single](3*$s),[single](3*$s))} - $g.Dispose();$bmp -} -function Get-DibBytes([Drawing.Bitmap]$Bmp){ - # 32-bit BGRA bottom-up DIB with an all-zero AND mask, as icon frames expect. - $n=$Bmp.Width;$ms=[IO.MemoryStream]::new();$w=[IO.BinaryWriter]::new($ms) - $w.Write([int]40);$w.Write([int]$n);$w.Write([int]($n*2));$w.Write([int16]1);$w.Write([int16]32);$w.Write([int]0);$w.Write([int]0);$w.Write([int]0);$w.Write([int]0);$w.Write([int]0);$w.Write([int]0) - for($y=$n-1;$y -ge 0;$y--){for($x=0;$x -lt $n;$x++){$c=$Bmp.GetPixel($x,$y);$w.Write([byte]$c.B);$w.Write([byte]$c.G);$w.Write([byte]$c.R);$w.Write([byte]$c.A)}} - $stride=[int]([math]::Ceiling($n/32.0)*4);$w.Write([byte[]]::new($stride*$n)) - $w.Flush() - ,$ms.ToArray() # the comma keeps PowerShell from unrolling the bytes + $g=[Drawing.Graphics]::FromImage($bmp) + $paths=[Collections.Generic.List[IDisposable]]::new() + try{ + $g.SmoothingMode='AntiAlias';$g.PixelOffsetMode='HighQuality';$g.Clear([Drawing.Color]::Transparent) + $g.ScaleTransform([single]($Size/$d.Grid),[single]($Size/$d.Grid)) + $body=New-RoundedPath $d.Body;$paths.Add($body) + $stripe=New-RoundedPath $d.Stripe;$paths.Add($stripe) + $light=New-RoundedPath $d.Light;$paths.Add($light) + Use-Brush '#1F2A2E' {param($b) $g.FillPath($b,$body)} + $pen=[Drawing.Pen]::new([Drawing.ColorTranslator]::FromHtml('#6F868D'),[single]$d.Rim) + try{$g.DrawPath($pen,$body)}finally{$pen.Dispose()} + Use-Brush '#3FB37A' {param($b) $g.FillPath($b,$stripe);$g.FillPath($b,$light)} + foreach($l in $d.Lines){$line=New-RoundedPath $l[0..4];$paths.Add($line);Use-Brush $l[5] {param($b) $g.FillPath($b,$line)}} + }catch{$bmp.Dispose();throw}finally{$g.Dispose();foreach($p in $paths){$p.Dispose()}} + $bmp } -$sizes=@(16,20,24,32,40,48,64,128,256) +$sizes=@(16,20,24,32,40,48,64,96,128,256) $frames=foreach($size in $sizes){ - $bmp=New-Frame $size - if($size -eq 256){$ms=[IO.MemoryStream]::new();$bmp.Save($ms,[Drawing.Imaging.ImageFormat]::Png);$bytes=$ms.ToArray()}else{$bytes=[byte[]](Get-DibBytes $bmp)} - if($PreviewPng -and $size -eq 256){$bmp.Save($PreviewPng,[Drawing.Imaging.ImageFormat]::Png)} - $bmp.Dispose();[pscustomobject]@{Size=$size;Bytes=$bytes} + $bmp=New-Frame $size;$ms=[IO.MemoryStream]::new() + try{ + $bmp.Save($ms,[Drawing.Imaging.ImageFormat]::Png) + if($PreviewPng -and $size -eq 256){$bmp.Save($PreviewPng,[Drawing.Imaging.ImageFormat]::Png)} + [pscustomobject]@{Size=$size;Bytes=$ms.ToArray()} + }finally{$bmp.Dispose();$ms.Dispose()} } +# ICO container: header, one 16-byte entry per frame, then the PNG frames (Windows Vista and later). $out=[IO.MemoryStream]::new();$w=[IO.BinaryWriter]::new($out) -$w.Write([int16]0);$w.Write([int16]1);$w.Write([int16]$frames.Count) -$offset=6+16*$frames.Count -foreach($f in $frames){ - $d=if($f.Size -ge 256){0}else{$f.Size} - $w.Write([byte]$d);$w.Write([byte]$d);$w.Write([byte]0);$w.Write([byte]0);$w.Write([int16]1);$w.Write([int16]32);$w.Write([int]$f.Bytes.Length);$w.Write([int]$offset);$offset+=$f.Bytes.Length -} -foreach($f in $frames){$w.Write([byte[]]$f.Bytes)} -$w.Flush();[IO.File]::WriteAllBytes($OutputPath,$out.ToArray()) +try{ + $w.Write([int16]0);$w.Write([int16]1);$w.Write([int16]$frames.Count) + $offset=6+16*$frames.Count + foreach($f in $frames){ + $edge=if($f.Size -ge 256){0}else{$f.Size} # 0 means 256 in an icon directory entry + $w.Write([byte]$edge);$w.Write([byte]$edge);$w.Write([byte]0);$w.Write([byte]0);$w.Write([int16]1);$w.Write([int16]32);$w.Write([int]$f.Bytes.Length);$w.Write([int]$offset);$offset+=$f.Bytes.Length + } + foreach($f in $frames){$w.Write([byte[]]$f.Bytes)} + $w.Flush();[IO.File]::WriteAllBytes($OutputPath,$out.ToArray()) +}finally{$w.Dispose()} Write-Output $OutputPath diff --git a/README.md b/README.md index 00fc251..feb7d21 100644 --- a/README.md +++ b/README.md @@ -19,10 +19,10 @@ Most backup tools either copy everything slowly or hide your files in a format o | **Per-folder choice** | Each folder is *Plain copy*, *Versions*, *Both* or *Skip*. A folder inside a skipped folder can still be backed up. | | **Proof, not hope** | *Check backup content* re-reads both sides of every file and compares SHA-256 hashes (plain copy) or verifies all repository data (versions). | | **Nothing deleted by surprise** | Files you delete from the source stay in the copy, or, with *Keep deleted for N backups*, move to a dated recovery folder beside it; only the newest N recovery folders are kept. | -| **Keep versions** | Optional clean-up after each versioned backup: one version a day for *N* days, a week for *N* weeks and a month for *N* months (restic `--keep-within-daily/weekly/monthly`, counted back from the newest version so a long pause never empties the history). Dry run first; the newest version, the one just made and versions from other tools are never deleted; deleted versions free their space and the repository is checked again. | +| **Keep versions** | Optional clean-up after each versioned backup: one version a day for *N* days, a week for *N* weeks and a month for *N* months (restic `--keep-within-daily/weekly/monthly`, counted back from the newest version so a long pause never empties the history). Dry run first; the newest version, the one just made and versions from other tools are never deleted; deleted versions free their space and the repository is checked again, and a failed check there marks the backup as failed. | | **Themes and size** | The *Look* page offers Light, Dark, Sand, Ocean and High contrast, each checked for WCAG 2.2 contrast, plus *Custom*: pick four colours and the rest is derived so text stays readable; the page reports any pair that is hard to read. Dark themes darken the title bar, scroll bars and drop-downs too. A larger or maximised window scales everything, text included, in proportion. | | **Waits for AI sessions** | Before a backup it looks for Claude Code or Codex sessions that worked in the last 10 minutes (their conversation files change while they work). If there are any, it names them and offers to wait and start by itself once all have been quiet for 10 minutes. | -| **Load control for your hardware** | Detects your processor and memory and offers Light, Balanced, Strong and Maximum levels, or your own threads, memory limit, parallel files and priority. You can lower the load while a backup runs. | +| **Load control for your hardware** | Detects your processor and memory and offers Light, Balanced, Strong and Maximum levels, or your own threads, memory limit, parallel files and priority. You can lower the load while a backup runs: the priority changes at once, the other settings from the next step. | | **Several backup drives, each with its own content** | One profile per drive, recognised by the drive itself rather than its letter. | | **Knows how fresh each backup is** | The age of the last complete backup, coloured green, blue or orange against the number of days you choose. | | **Skips what rebuilds** | `node_modules`, `.venv`, `__pycache__`, test and build caches, recycle bin and system folders are skipped by default; your own skips go in a personal settings file. | @@ -40,6 +40,8 @@ Most backup tools either copy everything slowly or hide your files in a format o Backups need no administrator rights, PATH changes or services. Only *Drive check* asks Windows for administrator rights, each time it runs, because reading a drive directly needs them. +Keep DevDriveBackup in a folder only you can change, for example `C:\Users\YOUR_USER\DevDriveBackup`. Anyone who can change its files, or rename a folder above it, can make it run their own code, and the drive check runs as administrator. A secondary drive such as a Dev Drive usually lets every signed-in account change files, sandbox accounts that tools create included, so before a drive check starts the window lists such accounts and asks first. The drive check also runs `smartctl` only when its SHA-256 matches the official 7.5 build. + ## Quick start 1. Connect the backup drive (NTFS recommended) and start `DevDriveBackup.cmd`. @@ -51,7 +53,7 @@ The first plain copy is verified automatically, file by file. After that, back u ## How it works -**Plain copy** (`Invoke-FastMirror.ps1`) runs robocopy per selected folder with `/E /COPY:DAT /DCOPY:DAT /XJ`, retries once, and skips caches and links. A file that stays locked is counted and reported without stopping the run. Files that exist only in the copy are reported, and moved to `-recovery\` only when you ask for it (`-MoveExtras`, or **Keep deleted for** on the overview); `-KeepRecovery N` then deletes all but the newest N dated recovery folders. Nothing is moved aside after a run in which robocopy failed. Verification hashes both sides in parallel and counts selected files missing from the copy. DevDriveBackup's own password, recovery-key and state folders are never copied by either engine. +**Plain copy** (`Invoke-FastMirror.ps1`) runs robocopy per selected folder with `/E /COPY:DAT /DCOPY:DAT /XJ`, retries once, and skips caches and links. A file that stays locked is counted and reported without stopping the run. Files that exist only in the copy are reported, and moved to `-recovery\` only when you ask for it (`-MoveExtras`, or **Keep deleted for** on the overview); `-KeepRecovery N` then deletes all but the newest N dated recovery folders. Nothing is moved aside after a run in which robocopy failed. Verification hashes both sides in parallel and counts selected files missing from the copy. DevDriveBackup's own password, recovery-key and state folders are never copied by either engine, even when one of them is selected itself; an older copy of one found in the plain copy is named in the run's result so you can delete it. **Versions** (`Backup-Worker.ps1`) run restic with a random 256-bit password stored for your Windows account (DPAPI) plus a portable recovery key file, a hard memory cap (Windows Job Object), CPU and I/O limits, and a structural `restic check` after every backup. Restore always goes to a new folder and verifies restored content. @@ -71,14 +73,20 @@ The first plain copy is verified automatically, file by file. After that, back u | File | Purpose | |---|---| | `DevDriveBackup.cmd`, `DevDriveBackup.ps1` | The main window | +| `DevDriveBackup.Theme.psm1`, `Backup.Taskbar.cs` | Colour themes and contrast checks; the window's own taskbar button | | `DevDriveBackup.Data.psm1` | Profiles, sources, history, freshness, load levels | +| `Backup.Selection.psm1` | Folder modes, read by both engines | | `Invoke-FastMirror.ps1`, `Backup.MirrorVerifier.cs` | Plain copy and verification | -| `Backup-Worker.ps1` | Versioned backup, check, restore, history, keys | -| `Backup-History.ps1`, `Backup-Security.ps1` | Versions and restore; password and recovery key windows | -| `DevDriveBackup-Advanced.cmd`, `Backup-Manager.ps1` | Advanced window for one versioned source | -| `Set-BackupDestination.ps1`, `Measure-BackupDisk.ps1` | Drive inspection, USB link speed, benchmark | +| `Backup-Worker.ps1`, `Backup.Process.cs`, `Backup.Status.psm1` | Versioned backup, check, restore, history, keys; hard memory cap; progress files | +| `Backup-History.ps1`, `Backup-Security.ps1`, `Backup.History.psm1`, `Backup.Security.psm1` | Versions and restore; password and recovery key windows; their safety rules | +| `DevDriveBackup-Advanced.cmd`, `Backup-Manager.ps1`, `Initialize-BackupConfig.ps1` | Advanced window for one versioned source, and its first configuration | +| `Install-BackupEngine.ps1` | Downloads restic into `bin\` and checks its pinned SHA-256 | +| `Set-BackupDestination.ps1`, `Measure-BackupDisk.ps1`, `Get-BackupHardware.ps1` | Drive inspection, USB link speed, benchmark, hardware summary | | `Invoke-DriveCheck.ps1`, `Backup.DriveCheck.psm1`, `Install-DriveTools.ps1` | Drive check (runs as administrator, read only), its verdict rules, health tool installer | | `Inspect-DevDrive.ps1`, `Build-BackupReport.py` | Optional size inventory and offline HTML report | +| `New-DevDriveBackupShortcut.ps1`, `New-DevDriveBackupIcon.ps1`, `DevDriveBackup.ico`, `design/icon` | Desktop shortcut; the program icon and its vector sources | +| `Test-*.ps1`, `.githooks`, `.github/workflows` | Tests, the privacy hooks and the checks that run on every push | +| `config.example.json`, `BACKUP-SYSTEM.md`, `CHANGELOG.md`, `THIRD-PARTY.md`, `LICENSE` | Example configuration, versioned engine details, changes, third-party software, license | | `profiles.local.json`, `jobs\*.local.json`, `backup.local.json` | Your personal settings (git-ignored) | Personal skips go in a job file: `Exclude` takes folder names (wildcards allowed) or paths below the source, and `NeverBackUp` lists folders that are never backed up by either engine, such as a folder holding credentials. See [config.example.json](config.example.json) and [BACKUP-SYSTEM.md](BACKUP-SYSTEM.md). @@ -107,7 +115,7 @@ Plain copy: open the folder on the backup drive and copy files back. Versions: * Yes. Each drive has its own profile with its own folders and its own freshness. **How do I know the backup drive itself is healthy?** -Open **Drive check** and run a **Quick check** (a few minutes). For a full answer, run a **Full surface scan**: it reads every part of the drive, takes hours, and can stop and continue later. Many USB boxes do not pass the drive's own health report on; then that part shows *Not available* and the full scan answers the same question: are there places that cannot be read. Nothing is written to the drive. +Open **Drive check** and run a **Quick check** (a few minutes). For a full answer, run a **Full surface scan**: it reads every part of the drive, takes hours, and can stop and continue later. Many USB boxes do not pass the drive's own health report on; then that part shows *Not available* and the full scan answers the same question: are there places that cannot be read. DevDriveBackup writes nothing to the drive; only Windows' own file system scan may note problems it finds. **Is there a scheduler or reminder?** No, by design. You decide when to back up; the window shows how old each backup is. diff --git a/THIRD-PARTY.md b/THIRD-PARTY.md index 00fa6dc..6c69472 100644 --- a/THIRD-PARTY.md +++ b/THIRD-PARTY.md @@ -1,7 +1,7 @@ # Third-party software - **Restic 0.19.1**, BSD 2-Clause. [Source and license](https://github.com/restic/restic/tree/v0.19.1), [official release](https://github.com/restic/restic/releases/tag/v0.19.1), [documentation](https://restic.readthedocs.io/en/stable/). Downloaded locally by `Install-BackupEngine.ps1`; not vendored into this repository. Archive SHA-256: `da948ad707ed690426473aaba2046cd61f8f90f6f0e7dab6be0d5796531de67d`. This hash was checked against the official release asset digest; it is not an independent publisher-signature verification. -- **smartmontools 7.5**, GPL-2.0-or-later. [Source and license](https://github.com/smartmontools/smartmontools/tree/RELEASE_7_5), [official release](https://github.com/smartmontools/smartmontools/releases/tag/RELEASE_7_5), [smartctl manual](https://www.smartmontools.org/browser/trunk/smartmontools/smartctl.8.in). Downloaded locally by `Install-DriveTools.ps1` and run as a separate program; not vendored into this repository. Installer SHA-256: `896337fcc253220614cf8cdbd5cf2321c5aa326a37a04160a672a281e6104c70`. Computed from the release asset, whose MD5 matched the published `.md5` file (`bb1e199ad6a3db3e1c27ae54b835cbd5`); this is not an independent publisher-signature verification. +- **smartmontools 7.5**, GPL-2.0-or-later. [Source and license](https://github.com/smartmontools/smartmontools/tree/RELEASE_7_5), [official release](https://github.com/smartmontools/smartmontools/releases/tag/RELEASE_7_5), [smartctl manual](https://www.smartmontools.org/browser/trunk/smartmontools/smartctl.8.in). Downloaded locally by `Install-DriveTools.ps1` and run as a separate program; not vendored into this repository. Installer SHA-256: `896337fcc253220614cf8cdbd5cf2321c5aa326a37a04160a672a281e6104c70`. Computed from the release asset, whose MD5 matched the published `.md5` file (`bb1e199ad6a3db3e1c27ae54b835cbd5`); this is not an independent publisher-signature verification. The 64-bit `bin\smartctl.exe` inside that installer has SHA-256 `b5db94e5082c042be44994b7a4fa8f7b5c8e713b2ab1c9a560d8f7a7995ea27d`; the drive check runs the bundled copy only when it matches. - **Chart.js 4.5.1**, MIT. Used only in the optional private HTML audit report. [Source and license](https://github.com/chartjs/Chart.js/tree/v4.5.1). The report generator requires a locally supplied `chart.umd.min.js` bundle (npm package `chart.js@4.5.1`, file `dist/chart.umd.min.js`) and checks its pinned SHA-384 before embedding it: `jb8JQMbMoBUzgWatfe6COACi2ljcDdZQ2OxczGA3bGNeWe+6DChMTBJemed7ZnvJ` (base64). Generated reports are offline and require no CDN connection. Preserve the library's license header if distributing an HTML report. - Windows, PowerShell, Python and .NET are platform/runtime dependencies, not included binaries. diff --git a/Test-BackupCopyRules.ps1 b/Test-BackupCopyRules.ps1 index f8a8009..43ec6be 100644 --- a/Test-BackupCopyRules.ps1 +++ b/Test-BackupCopyRules.ps1 @@ -13,6 +13,16 @@ function Throws([scriptblock]$Block,[string]$Pattern){try{& $Block|Out-Null;$fal $root=Join-Path ([IO.Path]::GetTempPath()) ('dev-drive-backup-copy-rules-'+[guid]::NewGuid().ToString('N')) function Put([string]$Path,[string]$Text){New-Item -ItemType Directory -Path (Split-Path $Path) -Force|Out-Null;[IO.File]::WriteAllText($Path,$Text)} try{ + # --- Private folders: a selection that is one of them, or lies inside one, is never copied ---- + $keys='C:\Users\YOUR_USER\DevDriveBackup recovery keys' + Assert (Test-BackupPathInside $keys @($keys)) 'a selection that is the private folder itself counts as private' + Assert (Test-BackupPathInside ($keys+'\old') @($keys)) 'a folder inside a private folder counts as private' + Assert (Test-BackupPathInside 'c:\users\your_user\appdata\local\devdrivemirror\' @('C:\Users\YOUR_USER\AppData\Local\DevDriveMirror')) 'case and a trailing backslash do not matter' + Assert (Test-BackupPathInside 'D:\Safe\recovery-key.txt' @('C:\Elsewhere','D:\Safe\recovery-key.txt')) 'a key file selected on its own counts as private' + Assert (-not (Test-BackupPathInside ($keys+' 2') @($keys))) 'a folder whose name only starts the same is not private' + Assert (-not (Test-BackupPathInside 'C:\Users\YOUR_USER' @($keys))) 'a parent folder is copied (robocopy skips the private folder below it)' + Assert (-not (Test-BackupPathInside 'C:\Users\YOUR_USER\Projects' @('',$null))) 'empty private paths are ignored' + # --- Content check ---------------------------------------------------------------------- $src=Join-Path $root 'source';$dst=Join-Path $root 'mirror' foreach($f in 'work\same.txt','work\changed.txt','work\resized.txt','work\not-copied.txt','work\node_modules\x.js','work\build\cache\c.bin','work\notes.secret','private\keys\k.txt','skipped\s.txt'){Put (Join-Path $src $f) $f} @@ -83,6 +93,24 @@ try{ $adopted=Add-CenterSource -Data $data -ProfileId $p.Id -Path (Join-Path $root 'one\app') Assert ($adopted.ConfigPath -eq $a.ConfigPath -and $adopted.MirrorFolder -ne $c.MirrorFolder) 'an adopted job whose folder name is now used by another source gets a new one' + # A removed source keeps its copy and versions on the drive, so another folder with the same name, + # added while the drive is unplugged, must not get its folders. + $q=New-CenterProfile $data 'Test 2' 'volume-not-connected-2' + foreach($f in 'three\work','four\work'){New-Item -ItemType Directory -Path (Join-Path $root $f) -Force|Out-Null} + $x=Add-CenterSource -Data $data -ProfileId $q.Id -Path (Join-Path $root 'three\work') + Remove-CenterSource $data $q.Id $x.Id + $y=Add-CenterSource -Data $data -ProfileId $q.Id -Path (Join-Path $root 'four\work') + $xJob=Read-CenterJob $x.ConfigPath;$yJob=Read-CenterJob $y.ConfigPath + Assert ($y.MirrorFolder -ne $x.MirrorFolder -and $yJob.RepositoryFolder -ne $xJob.RepositoryFolder) 'a new folder never gets the plain copy or versions folder of a removed source' + $back=Add-CenterSource -Data $data -ProfileId $q.Id -Path (Join-Path $root 'three\work') + Assert ($back.ConfigPath -eq $x.ConfigPath -and $back.MirrorFolder -eq $x.MirrorFolder) 'the removed folder added again still gets its own job and folders back' + # Before this rule, a new source could already share a removed source's repository. Adding the removed + # folder again must then start fresh instead of a second source writing to the same repository. + Remove-CenterSource $data $q.Id $back.Id + $yJob.RepositoryFolder=$xJob.RepositoryFolder;[IO.File]::WriteAllText($y.ConfigPath,($yJob|ConvertTo-Json -Depth 8)) + $fresh2=Add-CenterSource -Data $data -ProfileId $q.Id -Path (Join-Path $root 'three\work') + Assert ($fresh2.ConfigPath -ne $x.ConfigPath -and (Read-CenterJob $fresh2.ConfigPath).RepositoryFolder -ne $xJob.RepositoryFolder) 'a job whose repository another source uses now is not adopted' + # Saving a job never reverts a recovery-key location written meanwhile (by a password change). $stale=Read-CenterJob $c.ConfigPath $fresh=Read-CenterJob $c.ConfigPath;$fresh.RecoveryKeyPath=$fresh.RecoveryKeyPath+'.new';[IO.File]::WriteAllText($c.ConfigPath,($fresh|ConvertTo-Json -Depth 8)) diff --git a/Test-BackupDriveCheck.ps1 b/Test-BackupDriveCheck.ps1 index faf6037..a62b4c3 100644 --- a/Test-BackupDriveCheck.ps1 +++ b/Test-BackupDriveCheck.ps1 @@ -95,4 +95,55 @@ $fix=Get-DriveFileSystemFindings 'E' 'NTFS' 'SpotFixNeeded' '' 'Healthy' Assert ($fix.Status -eq 'Warning' -and $fix.Lines[0] -match 'chkdsk E: /spotfix') 'a small repair is a warning and names the command' Assert ((Get-DriveFileSystemFindings 'E' 'NTFS' 'FullRepairNeeded' '' 'Healthy').Status -eq 'Problem') 'a full repair is a problem' Assert ((Get-DriveFileSystemFindings 'E' 'exFAT' '' '' 'Healthy').Status -eq 'Unavailable') 'a file system Windows cannot scan online is unavailable' +# --- Self-tests: the newest of each kind counts -------------------------------------------------- +function SelfTest([string]$Type,[int]$Status,[int]$Hours){[pscustomobject]@{type=[pscustomobject]@{string=$Type};status=[pscustomobject]@{value=$Status;string=$(if($Status){'Completed: read failure'}else{'Completed without error'})};lifetime_hours=$Hours}} +$masked=Ata @{5=0};$masked.ata_smart_self_test_log.standard.table=@((SelfTest 'Short offline' 0 19999),(SelfTest 'Extended offline' 0x70 19900)) +$m=Get-DriveHealthFindings $masked +Assert ($m.Status -eq 'Problem' -and ($m.Lines -join ' ') -match 'Long self-test: failed' -and ($m.Lines -join ' ') -match 'Short self-test: passed') 'a short self-test that passed does not hide a long one that failed before it' +$healed=Ata @{5=0};$healed.ata_smart_self_test_log.standard.table=@((SelfTest 'Extended offline' 0 19999),(SelfTest 'Extended offline' 0x70 19900)) +Assert ((Get-DriveHealthFindings $healed).Status -eq 'OK') 'a newer long self-test that passed replaces an older one that failed' +$shorts=Ata @{5=0};$shorts.ata_smart_self_test_log.standard.table=@((SelfTest 'Short offline' 0 19999),(SelfTest 'Short offline' 0x70 19900)) +Assert ((Get-DriveHealthFindings $shorts).Status -eq 'OK') 'only the newest short self-test counts' +$cut=Ata @{5=0};$cut.ata_smart_self_test_log.standard.table=@((SelfTest 'Extended offline' 0x10 19999),(SelfTest 'Extended offline' 0x70 19900)) +$c=Get-DriveHealthFindings $cut +Assert ($c.Status -eq 'Problem' -and ($c.Lines -join ' ') -match 'did not finish') 'an interrupted long self-test does not hide an earlier one that failed' +$noLog=Ata @{5=0};$noLog.ata_smart_self_test_log.standard.table=@() +$n=Get-DriveSelfTest $noLog +Assert ($null -eq $n.Last -and $null -eq $n.LastShort -and $null -eq $n.LastLong -and (Get-DriveHealthFindings $noLog).Status -eq 'OK') 'a drive with an empty self-test log is fine' +$nvme.nvme_self_test_log.table=@([pscustomobject]@{self_test_code=[pscustomobject]@{value=1};self_test_result=[pscustomobject]@{value=0;string='Completed without error'};power_on_hours=99},[pscustomobject]@{self_test_code=[pscustomobject]@{value=2};self_test_result=[pscustomobject]@{value=7;string='Completed: failed segments'};power_on_hours=90}) +Assert ((Get-DriveHealthFindings $nvme).Status -eq 'Problem') 'on NVMe too, a short self-test that passed does not hide a failed long one' + +# --- Running as administrator: who could change what the check runs --------------------------- +$me='S-1-5-21-1-2-3-1001' +function Rule([string]$Sid,[long]$Rights,[switch]$Deny,[switch]$InheritOnly){[pscustomobject]@{Sid=$Sid;Allow=-not $Deny;Rights=$Rights;InheritOnly=[bool]$InheritOnly}} +function Item([string]$Path,[string]$Role,[string]$Owner,$Rules){[pscustomobject]@{Path=$Path;Role=$Role;Owner=$Owner;Rules=@($Rules)}} +$modify=0x1301BF;$readOnly=0x1200A9;$full=0x1F01FF +$safe=@((Rule 'S-1-5-18' $full),(Rule 'S-1-5-32-544' $full),(Rule $me $full),(Rule 'S-1-5-32-545' $readOnly)) +Assert (@(Get-DriveCheckPathRisks @(Item 'C:\Tool' 'Run' $me $safe) $me).Count -eq 0) 'only this user, SYSTEM and Administrators may change it and everyone may read: safe' +$open=@(Get-DriveCheckPathRisks @(Item 'E:\Tool\Invoke-DriveCheck.ps1' 'Run' $me ($safe+@(Rule 'S-1-5-11' $modify))) $me) +Assert ($open.Count -eq 1 -and $open[0].Sids -contains 'S-1-5-11') 'a file every signed-in account may change is reported with that account' +Assert (@(Get-DriveCheckPathRisks @(Item 'E:\Projects' 'Above' $me ($safe+@(Rule 'S-1-5-11' $modify))) $me).Count -eq 1) 'a folder above that other accounts may rename is reported' +Assert (@(Get-DriveCheckPathRisks @(Item 'E:\' 'Root' 'S-1-5-32-544' ($safe+@(Rule 'S-1-5-11' $modify))) $me).Count -eq 0) 'change rights on the drive root itself cannot swap a folder below it' +Assert (@(Get-DriveCheckPathRisks @(Item 'E:\' 'Root' 'S-1-5-32-544' ($safe+@(Rule 'S-1-5-11' $full))) $me).Count -eq 1) 'full control of the drive root can delete the folder below it' +Assert (@(Get-DriveCheckPathRisks @(Item 'C:\Tool\bin' 'Run' $me ($safe+@(Rule 'S-1-5-11' $modify -InheritOnly))) $me).Count -eq 0) 'a rule only for new files below does not count for the folder itself' +Assert (@(Get-DriveCheckPathRisks @(Item 'C:\Tool' 'Run' $me ($safe+@(Rule 'S-1-5-11' $modify -Deny))) $me).Count -eq 0) 'a deny rule is never a risk' +Assert (@(Get-DriveCheckPathRisks @(Item 'C:\Tool' 'Run' 'S-1-5-21-1-2-3-1003' $safe) $me).Count -eq 1) 'a file owned by another account is reported, since an owner can change its permissions' +$linked=Item 'E:\Tool' 'Run' $me $safe;$linked|Add-Member Link $true +Assert (@(Get-DriveCheckPathRisks @($linked) $me).Count -eq 1) 'a junction or link on the way counts as a risk, since what it leads to is not checked' +Assert ((Get-DriveCheckSmartToolHash) -match '^[0-9A-F]{64}$') 'the pinned smartctl hash is a SHA-256' +# The same rule on a real folder. Its permissions are set first: the temporary folder's own vary between computers. +$tool=Join-Path ([IO.Path]::GetTempPath()) ('drive-check-acl-'+[guid]::NewGuid().ToString('N')) +New-Item -ItemType Directory -Path (Join-Path $tool 'bin\smartmontools\bin') -Force|Out-Null +Set-Content -LiteralPath (Join-Path $tool 'Invoke-DriveCheck.ps1') -Value '# test' +try{ + $user=[Security.Principal.WindowsIdentity]::GetCurrent().User.Value + # The folder gets its own rules; everything inside inherits them. + & icacls.exe $tool /inheritance:r /grant:r "*${user}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' /Q|Out-Null + & icacls.exe (Join-Path $tool '*') /reset /T /Q|Out-Null + $mine=@(Get-DriveCheckPathRisks (Read-DriveCheckPathAccess $tool) $user|Where-Object {$_.Role -eq 'Run'}) + Assert ($mine.Count -eq 0) 'a tool folder only this user and administrators may change is safe to run as administrator' + & icacls.exe $tool /grant '*S-1-5-11:(OI)(CI)M' /Q|Out-Null + $opened=@(Get-DriveCheckPathRisks (Read-DriveCheckPathAccess $tool) $user|Where-Object {$_.Role -eq 'Run'}) + Assert ((@($opened|Where-Object {$_.Path -eq $tool}).Sids -contains 'S-1-5-11') -and @($opened|Where-Object {$_.Path -like '*\Invoke-DriveCheck.ps1'}).Count) 'once every signed-in account may change it, the folder and its files are reported' +}finally{if(Test-Path -LiteralPath $tool){& icacls.exe $tool /reset /T /Q|Out-Null;Remove-Item -LiteralPath $tool -Recurse -Force}} Write-Output 'All drive-check rule tests passed.' diff --git a/Test-BackupSecuritySafety.ps1 b/Test-BackupSecuritySafety.ps1 index 432e900..1790d3e 100644 --- a/Test-BackupSecuritySafety.ps1 +++ b/Test-BackupSecuritySafety.ps1 @@ -4,22 +4,27 @@ Import-Module (Join-Path $PSScriptRoot 'Backup.Security.psm1') -Force function Assert($ok,$message){if(-not $ok){throw $message};Write-Output ('PASS '+$message)} function Refuses($action,$message){$refused=$false;try{& $action}catch{$refused=$true};Assert $refused $message} $root=Join-Path $env:TEMP ('DevDriveSecuritySafety-'+[guid]::NewGuid().ToString('N'));$source=Join-Path $root 'source';$outside=Join-Path $root 'outside';New-Item -ItemType Directory -Path $root,$source,$outside|Out-Null -Refuses {Assert-NewBackupPassword 'short'} 'short password is refused' -Refuses {Assert-NewBackupPassword "long-enough-but`nmultiline"} 'multiline password is refused' -Refuses {Assert-NewBackupPassword ' leading-space-password'} 'leading whitespace is refused' -$key=Join-Path $outside 'key.txt';New-PrivateTextFile $key 'nonsecret test value' -$acl=Get-Acl -LiteralPath $key -Assert ($acl.AreAccessRulesProtected -and $acl.Access.Count -eq 2) 'file starts with a protected two-principal ACL' -Refuses {New-PrivateTextFile $key 'must not replace'} 'private file creation never overwrites existing data' -Assert ([IO.File]::ReadAllText($key) -ceq 'nonsecret test value') 'failed overwrite leaves existing data intact' -Set-PrivateTextFile $key 'replacement nonsecret value' -Assert ([IO.File]::ReadAllText($key) -ceq 'replacement nonsecret value' -and (Get-Acl -LiteralPath $key).AreAccessRulesProtected) 'atomic private-file replacement preserves restricted permissions' -Refuses {Assert-PrivateExportPath (Join-Path $source 'key.txt') @($source) @()} 'source directory cannot hold the portable key' -Refuses {Assert-PrivateExportPath $key @($source) @($key)} 'credential and configuration paths cannot be overwritten by export' -Refuses {Assert-PrivateExportPath 'relative-key.txt' @($source) @()} 'relative export paths are refused' -Refuses {Assert-PrivateExportPath ($key+':hidden.txt') @($source) @()} 'alternate data-stream export is refused' -$link=Join-Path $root 'linked';New-Item -ItemType Junction -Path $link -Target $outside|Out-Null -Refuses {Assert-PrivateExportPath (Join-Path $link 'key.txt') @($source) @()} 'linked export ancestors are refused' -# Compare full forms: a temp path with a short (8.3) name, as on CI runners, comes back expanded. -Assert ((Assert-PrivateExportPath (Join-Path $outside 'new-key.txt') @($source) @()) -eq [IO.Path]::GetFullPath((Join-Path $outside 'new-key.txt'))) 'new outside recovery path is accepted' -Write-Output ('Artifacts: '+$root) +try{ + Refuses {Assert-NewBackupPassword 'short'} 'short password is refused' + Refuses {Assert-NewBackupPassword "long-enough-but`nmultiline"} 'multiline password is refused' + Refuses {Assert-NewBackupPassword ' leading-space-password'} 'leading whitespace is refused' + $key=Join-Path $outside 'key.txt';New-PrivateTextFile $key 'nonsecret test value' + $acl=Get-Acl -LiteralPath $key + Assert ($acl.AreAccessRulesProtected -and $acl.Access.Count -eq 2) 'file starts with a protected two-principal ACL' + Refuses {New-PrivateTextFile $key 'must not replace'} 'private file creation never overwrites existing data' + Assert ([IO.File]::ReadAllText($key) -ceq 'nonsecret test value') 'failed overwrite leaves existing data intact' + Set-PrivateTextFile $key 'replacement nonsecret value' + Assert ([IO.File]::ReadAllText($key) -ceq 'replacement nonsecret value' -and (Get-Acl -LiteralPath $key).AreAccessRulesProtected) 'atomic private-file replacement preserves restricted permissions' + Refuses {Assert-PrivateExportPath (Join-Path $source 'key.txt') @($source) @()} 'source directory cannot hold the portable key' + Refuses {Assert-PrivateExportPath $key @($source) @($key)} 'credential and configuration paths cannot be overwritten by export' + Refuses {Assert-PrivateExportPath 'relative-key.txt' @($source) @()} 'relative export paths are refused' + Refuses {Assert-PrivateExportPath ($key+':hidden.txt') @($source) @()} 'alternate data-stream export is refused' + $link=Join-Path $root 'linked';New-Item -ItemType Junction -Path $link -Target $outside|Out-Null + Refuses {Assert-PrivateExportPath (Join-Path $link 'key.txt') @($source) @()} 'linked export ancestors are refused' + # Compare full forms: a temp path with a short (8.3) name, as on CI runners, comes back expanded. + Assert ((Assert-PrivateExportPath (Join-Path $outside 'new-key.txt') @($source) @()) -eq [IO.Path]::GetFullPath((Join-Path $outside 'new-key.txt'))) 'new outside recovery path is accepted' +}finally{ + # Remove the link first so the clean-up can never follow it out of the test folder. + if($link -and (Test-Path -LiteralPath $link)){[IO.Directory]::Delete($link)} + if(Test-Path -LiteralPath $root){Remove-Item -LiteralPath $root -Recurse -Force} +} diff --git a/Test-BackupStatus.ps1 b/Test-BackupStatus.ps1 index c39e1c8..cc236af 100644 --- a/Test-BackupStatus.ps1 +++ b/Test-BackupStatus.ps1 @@ -2,17 +2,19 @@ $ErrorActionPreference='Stop' Import-Module (Join-Path $PSScriptRoot 'Backup.Status.psm1') -Force $dir=Join-Path $env:TEMP ('DevDrive-StatusTest-'+[guid]::NewGuid().ToString('N'));[void][IO.Directory]::CreateDirectory($dir) -$path=Join-Path $dir 'status.json' -[void](Write-BackupStatus @{State='Old'} $path) -$reader=[IO.File]::Open($path,[IO.FileMode]::Open,[IO.FileAccess]::Read,[IO.FileShare]::Read) -try { - $published=Write-BackupStatus @{State='New'} $path -BestEffort - if($published -or (Read-BackupStatus $path).State -ne 'Old'){throw 'Locked status was not handled safely'} - [void](Write-BackupStatus @{State='Completed'} (Join-Path $dir 'result.json')) - if((Read-BackupStatus (Join-Path $dir 'result.json')).State -ne 'Completed'){throw 'Terminal result was not published independently'} -}finally{$reader.Dispose()} -if(-not(Write-BackupStatus @{State='New'} $path) -or (Read-BackupStatus $path).State -ne 'New'){throw 'Status did not recover after reader released its lock'} -$reader=[IO.File]::Open($path,[IO.FileMode]::Open,[IO.FileAccess]::Read,([IO.FileShare]::ReadWrite -bor [IO.FileShare]::Delete)) -try{if(-not(Write-BackupStatus @{State='Newest'} $path)){throw 'A delete-sharing reader blocked publication'}}finally{$reader.Dispose()} -if((Read-BackupStatus $path).State -ne 'Newest'){throw 'Updated status was not readable'} +try{ + $path=Join-Path $dir 'status.json' + [void](Write-BackupStatus @{State='Old'} $path) + $reader=[IO.File]::Open($path,[IO.FileMode]::Open,[IO.FileAccess]::Read,[IO.FileShare]::Read) + try { + $published=Write-BackupStatus @{State='New'} $path -BestEffort + if($published -or (Read-BackupStatus $path).State -ne 'Old'){throw 'Locked status was not handled safely'} + [void](Write-BackupStatus @{State='Completed'} (Join-Path $dir 'result.json')) + if((Read-BackupStatus (Join-Path $dir 'result.json')).State -ne 'Completed'){throw 'Terminal result was not published independently'} + }finally{$reader.Dispose()} + if(-not(Write-BackupStatus @{State='New'} $path) -or (Read-BackupStatus $path).State -ne 'New'){throw 'Status did not recover after reader released its lock'} + $reader=[IO.File]::Open($path,[IO.FileMode]::Open,[IO.FileAccess]::Read,([IO.FileShare]::ReadWrite -bor [IO.FileShare]::Delete)) + try{if(-not(Write-BackupStatus @{State='Newest'} $path)){throw 'A delete-sharing reader blocked publication'}}finally{$reader.Dispose()} + if((Read-BackupStatus $path).State -ne 'Newest'){throw 'Updated status was not readable'} +}finally{if(Test-Path -LiteralPath $dir){Remove-Item -LiteralPath $dir -Recurse -Force}} Write-Output 'PASS: locked status is nonfatal, final results are independent, and delete-sharing readers allow atomic replacement'