Repository navigation
Excel tables and whole-sheet writes in the bindings, parallel shared strings, runtime-stable decimals - #156
Merged
Merged
Conversation
xl_write_typed writes a single-sheet workbook, so a binding that wanted several sheets from columnar data had to stream them through xl_writer_handle one call per cell. xl_write_table writes an xl_table as one sheet of an open handle, between sheets started with xl_start_sheet, and fails while one is open. The sheet body WriteWorkbook wrote moves into WriteSheetBody, shared by both paths, and the handle now keeps the parallel options it used to discard, so a table sheet honours degree_of_parallelism and batch_rows. The export is additive; the ABI stays at 6.
….write_sheet write_workbook writes one sheet, and the only multi-sheet path was start_sheet/write_row, one ctypes call per cell. SheetWriter.write_sheet writes a whole table as one sheet through xl_write_table, taking the same data and types as write_workbook: a TypedTable with its types, or a pandas, polars or pyarrow frame. Options come from open_writer, so every sheet is written in parallel when degree_of_parallelism allows it. The data/types checks write_workbook did inline move into _typed_table, shared by both.
WriterHandle could only stream a sheet one cell at a time. write_columns and write_sheet take what the free functions of the same name take, plus the sheet's name, and write it through xl_write_table in one call, using the handle's parallelism options. raw_table and OwnedColumn::as_column become crate-visible so the handle reuses them.
xl::WriterHandle could only stream a sheet one cell at a time. write_columns and write_sheet take what the free functions of the same name take, plus the sheet's name, and write it through xl_write_table in one call, using the handle's parallelism options. detail::write_rows now hands its ColumnRefs to a callable instead of a destination, so the free write_sheet overloads and the handle share the transpose.
Each column was pointed at with ctypes.cast(copy, ...), which stores the copy in its own _objects and so puts it in a reference cycle. Dropping keepalive freed nothing, and the copies waited for the cyclic collector, which on Python 3.14 runs about every 11 write_sheet calls: twenty 30-column, 1M-row sheets in one writer peaked at 2.8 GB. The pointers now come from ctypes.addressof, which keeps no reference, and the same run peaks at 620 MB.
… span WorkerStringCache.GetOrAdd(ReadOnlySpan<char>) turned every local miss into a string before taking the table's lock, so a high-cardinality column allocated one string per cell even for texts the table already held: about 1M per sheet once the 65,536-entry local cache filled. The miss now looks the span up under the lock, and a string is only made when the table or the local cache adds one. The end-to-end gain is within noise (about 6% on four 1M-row string columns at four threads); the lock itself is the bottleneck.
…serializing Every text a worker had not cached took the workbook's single shared string lock, so a high-cardinality column ran its lookups one thread at a time and a parallel write was slower than a serial one: four 1M-row string columns, two of them unique IDs, took 0.80 s a sheet at eight threads against 0.50 s serially. Workers now add new texts to 64 shards split by hash, each with its own lock, and take the index from an interlocked counter. The serial writer keeps one dictionary looked up with a single hash, which workers read without a lock: the serial and parallel paths never run at once. Each worker counts its references locally and adds them in bulk, and the value list is rebuilt from the dictionaries when the part is written. Same data, median sheet: 0.33 s at eight threads (2.4x), serial unchanged at 0.50 s, low-cardinality columns unchanged.
.NET could list tables, read one as a sheet and write a sheet as a table; the C ABI could do none of it. Reading: XL_TABLE_SHEET(table), -2 - table, passed as the sheet of any read call reads only that table's range, header row first. Every read resolves its sheet in ReadApi.ResolveSheet, so rows, callbacks, typed, Arrow, schema inference and aggregation take tables with no new entry point; a table read is never partitioned. xl_table_count, xl_table_info_at, xl_table_name_at and xl_table_index describe them. Writing: the unreleased xl_write_table becomes xl_write_sheet, since "table" already means xl_table in this ABI, and takes an optional xl_sheet_options carrying a table name, style and XL_TABLE_* flags. A table needs every column named and an XLSX or XLSB handle. The ABI stays at 6. The bindings still call xl_write_table until their own commits.
workbook.tables lists a workbook's tables with the lookups sheets has: index, slice or name, ignoring case. A Table is a Sheet over XL_TABLE_SHEET, so rows, parse, Arrow and pandas reads all work on it; its name and index are the table's, and sheet, the range and the header and totals row counts come from xl_table_info_at. SheetWriter.write_sheet takes table=TableOptions(name, style=..., row_stripes=..., ...) and calls xl_write_sheet with it.
Workbook::tables returns each table's TableInfo; table and table_by_name return a Sheet over XL_TABLE_SHEET, so every Sheet read works on a table. WriterHandle::write_columns and write_sheet, not yet released, take an Option<&TableOptions> and call xl_write_sheet.
Workbook::tables returns each table's xl::TableInfo; table and table_by_name return an xl::Sheet over XL_TABLE_SHEET, so every read works on a table. WriterHandle::write_columns and write_sheet take an optional xl::TableOptions and call xl_write_sheet.
…B writers XlsxWorkbookWriter and XlsbWorkbookWriter carried the same 260 lines of end, abandon, dispose and stream-release logic. ZipWorkbookPackage now owns it; each writer implements IZipWorkbookParts to write its own parts, and the sheet writers implement IZipSheet so the package can end or abandon the open sheet. The failure ordering is unchanged. The sheet writers share the column-set union through SheetColumnValidation.Columns, and XlsxSheetWriter opens its entry in one place for the sync and async starts. An empty XLSB workbook now reports "An XLSB workbook must contain at least one sheet.", matching the XLSX message.
Thirteen internal streams repeated the same capability flags and NotSupportedException overrides. ReadOnlyStream and WriteOnlyStream now carry them: ten read streams (three of them seekable, which override CanSeek, Length, Position and Seek) and three write streams derive from them. WriteOnlyStream keeps Flush abstract so a new write stream cannot inherit a silent no-op. The Read and ReadAsync overloads stay on the concrete sealed classes, so the hot read paths are unchanged.
ffi::native_slice, moved out of workbook.rs and taking any integer length, replaces eight hand-written null-or-non-positive checks before slice::from_raw_parts. Shared::guard holds the catch_unwind mapping the four aggregation callbacks repeated. TypedBatches and TableBatches share a BatchReader that opens, advances and closes the native typed reader. RowRef::from_decoded now takes its length from the slice, as from_views already did: a null cells pointer with a positive count reported that count as len() and made get(0) panic in safe code.
Cell.TryParse<decimal> cast a binary cell's double with (decimal)x. That conversion lives in the runtime: .NET 10 rounds it to 15 significant digits, and .NET 11 makes it correctly rounded, so the same XLSB or XLS cell would read as 1.23 on one runtime and 1.2299999999999999822... on the next, with no change to the caller. A cell that keeps its text (XLSX) now parses that text. A binary cell looks for the fewest decimal places, up to 15 significant digits, whose mantissa divides back to the same double, and builds the decimal from it; only 16- and 17-digit values go through their shortest round-trip text. The result is what parsing the double's shortest text gives, so XLSB, XLS and XLSX agree: 0.1 + 0.2 now reads as 0.30000000000000004 from XLSB and XLS too, where the cast gave 0.3. Text cells in exponent form, such as 1E-07, now parse as decimal; the fallback used NumberStyles.Number, which has no exponent. Measured on 1M XLSB cells: 13.6 ns per two-decimal value against 12.4 for the cast, and about 83 ns per 16- or 17-digit value.
ZipIndex, XlsCompoundFile and EncryptedPackageOpener each wrapped a ReadOnlyMemory<byte> in a MemoryStream, copying it with ToArray() when it was not backed by an array. ReadOnlyMemoryStream reads any memory in place and replaces the three helpers.
The net10.0 test build runs on the .NET 11 release candidate through --roll-forward LatestMajor, so runtime behavior changes surface before users upgrade. The job fails if no .NET 11 runtime is installed, so it cannot silently test on .NET 10, and it is non-gating while .NET 11 is a release candidate.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #156 +/- ##
==========================================
+ Coverage 91.22% 91.47% +0.25%
==========================================
Files 278 284 +6
Lines 22633 22904 +271
Branches 3652 3697 +45
==========================================
+ Hits 20647 20952 +305
+ Misses 1423 1374 -49
- Partials 563 578 +15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ader.Benchmarks project
Utf8StringCache was direct-mapped on HashCode, which is seeded per process, so which values shared a bucket changed from run to run. In about 3% of processes (6 of 200) two of a 10-value column's strings landed in one bucket and evicted each other on every row, and a third of that column was materialized again: +58 MB per run on the 4.3M-row interned CSV parse, the 66 MB BenchmarkDotNet row. The cache is now 4-way set-associative (256 sets of 4, the same 1024 entries) and keeps each entry's hash, so a mismatch is rejected by an int compare. 0 of 200 processes evict after the change; a lookup costs about 0.9 ns more and the parallel parse shows no end-to-end change.
FromAttributes bound a class model's property through a wrapper lambda around an open setter delegate, two indirect calls per cell, and created every row with Activator.CreateInstance<T>. Both are now one compiled expression when the runtime JITs dynamic code; without it (NativeAOT, interpreters) the previous path stays. Typed XLSX parse of 50,000 rows into a class: 11.47 to 10.53 ms median over 10 processes, now level with the struct model.
Dynamic PGO devirtualizes and inlines the column parsers into RowProjector.Project (2.3 KB, a 344-byte frame), which spends the inline budget: RowCellEnumerator.MoveNext stayed a call per cell, even marked AggressiveInlining, and the loop's state lived on the stack. ParseRow now indexes Row.Descriptors directly and builds a Cell only after the empty check, through the new internal Row.CellAt. Typed XLSX parse of 50,000 rows against a hand-written loop doing the same conversions, median of 10 processes: class model 1.25x to 1.18x, struct model 1.25x to 1.17x. The parallel sheet and mapped CSV paths share this loop.
…anges The read and typed-parse tables, the ref struct table and the ADO.NET bridge come from a --job Short --launchCount 5 run: Tiered PGO settles each process into a different code shape, and single-launch runs moved these rows by up to 10%. With that variance gone, typed parsing is ~1.1x faster than OfficeIMO again, and GetBytes costs ~7% over a raw read, not 20%. The parallel CSV, parallel sheet and large-sheet rows are a Medium rerun on the current code; the interned struct parse no longer shows the 66 MB allocation the old direct-mapped intern cache produced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The single-launch run had the small fixture's open-only row (27.43 ms) above a full encrypted read of the same file (26.97 ms). With five launches the two match (26.10 ms against 26.13 ms), the fixed key derivation cost agrees across both fixtures, and decrypting while reading costs ~3% over a plain read, not ~5%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The .NET 11 release-candidate job keeps failing for reasons outside the library and adds noise to every run. Remove it until .NET 11 is GA; it was non-gating, so no required check changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Seventeen commits since v7.2.0: features for the bindings, performance work on the writer, one reader fix, and internal cleanups.
Writer performance: shared strings are sharded across 64 locks, so parallel workers adding different texts no longer serialize on one lock.
Reader fix:
TryParse<decimal>on a stored number used the(decimal)doublecast. The runtime implements that cast: .NET 10 rounds it to 15 digits and .NET 11 rounds it exactly, so the same XLSB cell read differently depending on the runtime.Text cells in exponent form (
1E-07) now parse as decimal too.Cleanups, no behavior change:
ZipWorkbookPackageowns the end, abandon and dispose logic the XLSX and XLSB workbook writers duplicated.ReadOnlyStreamandWriteOnlyStreamhold theStreamoverrides 13 internal streams repeated.ReadOnlyMemoryStreamreads caller memory in place where three helpers copied it withToArray().native_slice,Shared::guardand a sharedBatchReaderforTypedBatchesandTableBatches.CI: a new non-gating
next-runtimejob runs the net10.0 test build on the .NET 11 RC (--roll-forward LatestMajor).Evidence
Before: 0.80 s per sheet, slower than serial (0.50 s)
After: 0.33 s per sheet; serial unchanged
Before: peak 2.8 GB (column copies waited for the cyclic GC)
After: peak 620 MB
CellStoredDecimalTests):Merge Danger
Door: two-way
Merging publishes nothing; releases are cut from
v*tags. The new C ABI exports become a compatibility commitment only once a tag ships them.Blast Radius: values
In XLSB and XLS, a stored number with 16–17 significant digits read as
decimalnow keeps every digit (0.30000000000000004where it was0.3), matching XLSX. This belongs in the 7.3 release notes. Everything else is additive or internal.