Skip to content

Add opt-in archive size and entry count limits to PaArchive - #822

Open
Crash Collison (tehcrashxor) wants to merge 1 commit into
masterfrom
joshuac-microsoft-paarchive-opt-in-size-limits
Open

Crash Collison (tehcrashxor) wants to merge 1 commit into
masterfrom
joshuac-microsoft-paarchive-opt-in-size-limits

Conversation

@tehcrashxor

Copy link
Copy Markdown
Member

Related issue: N/A

Problem

Some callers need to bound the resources used when they open .msapp archives, for example to keep memory and processing within service limits. Today PaArchive has no option for this, so those callers wrap IMsappArchiveFactory with their own checks of the zip central directory. That duplicates archive-format knowledge outside this library.

Solution

Add opt-in limits to PaArchive. When a caller passes PaArchiveLimits, opening an existing archive in Read or Update mode checks the sizes and entry count recorded in the zip central directory, before any entry is read. It throws PaArchiveLimitExceededException when a limit is exceeded. Create mode is never checked. Callers that don't pass limits see no change in behavior, including MsappArchiveFactory.Default and AddMsappArchiveFactory().

The raw stats and the evaluator are public and work without limits. A caller can open an archive normally, then call GetStats() and Evaluate() to observe how archives compare to the limits before enforcing them.

Changes

  • New types in Microsoft.PowerPlatform.PowerApps.Persistence.Compression:
    • PaArchiveLimits: a record with MaxExpandedSize, MaxCompressedSize, MaxCompressionRatio, RatioCheckMinExpandedSize, MaxEntrySize, MaxEntryCount and MaxTextEntrySize.
      • Default holds recommended values: 4 GiB expanded, 1 GiB compressed, ratio 250 above 100 MiB expanded, 512 MiB per entry, 10,000 entries, and 256 MiB per text entry.
      • Default is only used when a caller passes it.
      • A text entry is one whose path ends with .json, .yaml, .yml, .xml or .txt.
    • PaArchiveLimits.Evaluate(PaArchiveStats) returns the first limit exceeded. The order is EntryCount, CompressedSize, ExpandedSize, EntrySize, TextEntrySize, CompressionRatio. It returns None when no limit is exceeded. A value equal to a limit is allowed.
    • PaArchiveStats is a record with CompressedSize, ExpandedSize, EntryCount, LargestEntrySize and LargestTextEntrySize, plus FromEntries(...). Totals saturate instead of overflowing.
    • PaArchiveLimit enum.
    • PaArchiveLimitExceededException, a subclass of PersistenceLibraryException, exposes Limit, Stats and Limits.
  • New error code: PersistenceErrorCode.PaArchiveLimitExceeded = 5005.
  • Constructor overloads: new overloads that take PaArchiveLimits? limits on PaArchive and MsappArchive. The existing constructors are unchanged and chain to them. Both types get a Limits property.
  • MsappArchiveFactory: a new init-only Limits property, used as new MsappArchiveFactory(logger) { Limits = limits }. The factory keeps its single constructor, so dependency injection still builds a factory without limits, even when PaArchiveLimits is registered in the container.
  • PaArchive.GetStats() computes stats from all raw zip entries. That includes the directory, duplicate and invalid-path entries that Entries skips. It doesn't load Entries.
  • Entry-count pre-check: when limits are set and the stream is seekable, the entry count declared in the end of central directory record (including Zip64) is checked before ZipArchive reads the central directory. If the record is missing or can't be read, ZipArchive handles the archive as it does today.
  • Update mode is checked through a separate read-only ZipArchive pass. An archive that exceeds the limits is never opened in Update mode and is left unchanged. When a constructor with limits throws, the stream is disposed unless leaveOpen is true.
  • DI: new AddMsappArchiveFactory(PaArchiveLimits) overload. It replaces any existing IMsappArchiveFactory registration, so the limits apply even if AddMsappPackingService() was called first.
  • Logging: a Warning is logged (event 108) when an archive is rejected and a logger was provided.
  • Refactor: the net48 invalid-path-character handling moved into a shared helper, used by both Entries and GetStats().
  • Docs: updated .github/copilot-instructions.md, including fixing the archive error-code range to 5xxx.

Validation

  • dotnet build src/PASopa.sln: passed, 0 warnings, 0 errors.
  • dotnet test --no-build --project src/Persistence.Tests/Persistence.Tests.csproj (net48, net8.0, net10.0): 1137 passed, 3 failed.
    • The 3 failures are net48-only MsappPackingServiceTests.PackFromMsappReferenceFile_* cases. They throw DirectoryNotFoundException because my local worktree path pushes their test output paths past MAX_PATH (271 chars).
    • They pass when rerun with a short artifacts path: dotnet test --no-build --project src/Persistence.Tests/Persistence.Tests.csproj -f net48 -p:ArtifactsPath=%TEMP%\pat\ --filter "FullyQualifiedName~PackFromMsappReferenceFile" reported 6 passed, 0 failed. They aren't related to this change.
  • dotnet pack --no-build -c Debug src/Persistence/Microsoft.PowerPlatform.PowerApps.Persistence.csproj: passed.
  • New tests in PaArchiveLimitsTests, plus additions to MsappArchiveTests, cover:
    • the evaluation order and boundaries
    • stats that include ignored entries without loading Entries
    • rejection for each limit in Read and Update modes
    • archive bytes left unchanged after a rejected open, including a file opened via MsappArchiveFactory.Update(path)
    • leaveOpen handling
    • Create mode and empty Update streams not being checked
    • pre-checks of the declared entry count, with an archive comment and with Zip64 records
    • the non-seekable stream path
    • malformed archives, which throw the same exception type with and without limits
    • the factory and DI registration, including a container that registers PaArchiveLimits but still resolves a factory without limits
  • The tests use small in-memory archives with small limits, plus hand-edited end of central directory and Zip64 records for declared entry counts. No large test files were added.

Callers that need to bound the resources used when opening archives can
now pass PaArchiveLimits to PaArchive, MsappArchive, MsappArchiveFactory
or AddMsappArchiveFactory(limits). When set, opening an existing archive
in Read or Update mode checks the sizes and entry count recorded in the
zip central directory before any entry is read, and throws
PaArchiveLimitExceededException (PersistenceErrorCode 5005) when a limit
is exceeded. Create mode is not checked.

- PaArchive.GetStats() and PaArchiveLimits.Evaluate() can be used
  without limits.
- When the stream is seekable, the entry count declared in the end of
  central directory record (including Zip64) is checked before
  ZipArchive reads the central directory.
- Update mode is checked with a separate read-only pass, so an archive
  that exceeds the limits is left unchanged.

Behavior is unchanged for callers that don't pass limits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant