Repository navigation
feat!: FixedWidthParser 2.0 — bytes-only API, native library and Python package - #79
Merged
Merged
Conversation
2.0 keeps one parse path. The char parser, FixedWidthUtf8, FixedWidthByteReader, Utf8FixedWidthParser, IUtf8FixedWidthModel and IUtf8FixedWidthConverter are gone: there is one parser, one reader, one writer and one generated TryParse/TryFormat, all over bytes. Column positions are bytes; text columns, dates parsed with a format and types that only parse from characters are decoded with an ASCII-compatible encoding, UTF-8 by default. The writer formats straight to the stream, batched into one pooled buffer, and never splits a multi-byte character when it truncates. The package targets net10.0 only. Reads take FixedWidthReaderOptions (format provider, encoding, string pool, buffer size, MaxLineLength, OnParseError) and the writer takes FixedWidthWriterOptions (format provider, encoding, new line). Generated code and IFixedWidthConverter<T> receive a FixedWidthContext by in reference, so a later setting arrives as a property instead of a new parameter. A line that fails to parse throws FixedWidthParseException, a FormatException that names the line, the column, its position and its raw text, or says how short the line is. The reader learns the column by parsing the failed line again with a failure sink, so lines that parse pay nothing. OnParseError can skip the line instead. MaxLineLength bounds the line buffer on stream, async and pipe reads. A source without line breaks used to double the buffer until memory ran out. The generator targets netstandard2.0, reports FWP012 for a non-ASCII Padding or TrimChar, and its tests now compile the generated code. The library tracks its public API (PublicAPI.*.txt), validates the package against 1.1.1 with the deliberate breaks listed in CompatibilitySuppressions.xml, and builds with warnings as errors.
Packing runs package validation, so a removed or changed public member fails the pull request instead of the release.
The README describes the byte-only reader and writer, the options, the errors, encodings and the versioning rules. The changelog separates what shipped in 1.1.0 and 1.1.1 from 2.0, with a migration table. CONTRIBUTING explains the public API tracking and package validation.
… doubles as scaled integers IUtf8SpanFormattable columns format directly into the line's slice and are aligned in place, with no char buffer, transcode or copy. The UTF-8 bytes are kept when the encoding is UTF-8 or they are ASCII; otherwise the value goes through the configured encoding as before. A double with no Format, at most four decimals and a magnitude under 1e9 is written as a scaled integer, byte-identical to double.TryFormat, for any culture whose decimal separator is one ASCII character and negative sign is "-". Left-aligned strings under UTF-8 transcode straight into the slice.
Extract was over MA0051's 90-line limit. The per-member work moves to ExtractColumn, which returns null for a member that is not a mapped column.
Expression-bodied methods, constructors, operators and local functions and if without braces warn in the IDE; properties and their accessors are exempt (IDE0027 off). Private consts are PascalCase. MA0051 is enabled at 90 lines / 50 statements instead of suppressed. .gitattributes pins eol=lf so a blob's line endings never depend on who committed it.
STYLEGUIDE.md and ARCHITECTURE.md follow ExcelReader's, adapted to this library's twins (runtime and generated, stream/async/pipe readers) and its untrusted-input rules, and add the rule to avoid else. docs/ holds the measurements behind design decisions, starting with the writer. CONTRIBUTING and SECURITY are rewritten in the same shape, and a code of conduct is added.
FixedWidthReader<T>.AggregateFileAsync and FixedWidth.AggregateFileAsync cut a file into byte ranges, fold each on a worker thread into an IRecordAccumulator<TSelf, TModel>, and merge the accumulators in file order. A line belongs to the range its first byte falls in. Fixed-width lines have no quoting, so a worker that starts one byte before its range and skips the line it lands inside finds the exact boundary: every line is parsed exactly once. Line numbers in errors match a sequential read; they are counted only when a line fails. One range is the sequential read, so DegreeOfParallelism = 1 and small files run the same code.
FixedWidthReader<T>.AggregateAsync(Stream) and FixedWidth.AggregateAsync fold a stream across several threads. A stream cannot be read at an offset, so one producer reads it into pooled 256 KB chunks cut after the last line break and hands them through a bounded queue to the workers; the accumulators are merged in chunk order. The producer counts the line breaks it hands over, so every chunk knows the number of its first line and errors carry the sequential line number. A producer failure lies past every chunk it emitted, so a worker's failure is thrown first. DegreeOfParallelism = 1 reads with the async enumerator. LineBufferState.LineNumber becomes a long: a chunk's first line number can pass int.MaxValue, and so could the sequential reader's count. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The dedicated PipeRecordEnumerator read a 104 MB file in 472 ms; the same pipe through PipeReader.AsStream() into the Stream reader took 187-230 ms. It also duplicated every line rule LineBufferState implements, so removing it leaves one line splitter for every reader. Measurements are in docs/pipe-reader.md. BREAKING CHANGE: ReadAsync(PipeReader) is removed from FixedWidthReader<T> and FixedWidth, with FixedWidthPipeRecordEnumerable and PipeRecordEnumerator. Pass pipeReader.AsStream() to the Stream overloads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
FixedWidthLayout.FromAttributes<T>() builds a model's column parsers and formatters once and caches them; FixedWidth.TryParse and TryFormat take a layout. The runtime parser and writer now sit on top of it. A model that only one side can handle keeps that side's error until the side is first used.
FixedWidth.Read, ReadFile, ReadAsync, ReadFileAsync, AggregateFileAsync and AggregateAsync gain a layout overload next to each generated one. A LayoutLineParser<T> strategy replaces ReflectionLineParser<T>, so the enumerators and both parallel aggregators are unchanged.
FixedWidth.Write, WriteMany, WriteAsync and WriteManyAsync each come as a generated overload and a layout overload. Batching moves out of FixedWidthWriter<T> into RecordWriter, generic over an IRecordFormatter<T> struct like the read side. A generated model has no known line length, so a record that does not fit flushes the batch and, if still too long, grows it. Generated models can now write to a stream.
FixedWidthLayoutBuilder<T> binds each column with a ColumnGetter and a ColumnSetter: Column (string or UTF-8 parsable), DecodedColumn, NullableColumn, ConvertedColumn and Factory. Per-column settings come in FixedWidthColumnOptions, with FixedColumnAttribute's defaults. The builder reaches the same typed parse and format builders as the attribute path, by generics instead of reflection, so the two produce identical results and Build stays trimming- and AOT-safe.
FixedWidthParser<T>, FixedWidthReader<T> and FixedWidthWriter<T> are removed. Attribute-mapped models are read and written through FixedWidth.*(..., FixedWidthLayout.FromAttributes<T>(), ...), the same overloads fluent layouts use. Tests and benchmarks are migrated; the package validation suppressions now list the removed types. BREAKING CHANGE: replace new FixedWidthReader<T>(options).ReadFile(path) with FixedWidth.ReadFile(path, FixedWidthLayout.FromAttributes<T>(), options), and likewise for the parser and the writer.
…write timings tests/AotSmoke builds a layout with FixedWidthLayout.Build and round-trips 1,000 records through FixedWidth; publishing it with NativeAOT must report no trimming or AOT warning. docs/fluent-layout.md compares reading and writing through layouts with the removed classes: every figure is within noise.
README shows reading and writing through FixedWidth with a layout and adds a section on fluent layouts. ARCHITECTURE describes the layout object, its two sources and RecordWriter. STYLEGUIDE and CHANGELOG follow the removal of the runtime classes.
Under the parallel test run, the per-line allocation guard sometimes saw a one-off ~8 KB allocation on its thread during one read (2.02 B/line over 4,000 extra lines); the baseline commit shows the same kind of failure. The guard now keeps the smallest of three measurements: a real per-line allocation still shows in all three, as a deliberately injected 24 B/line one does.
NullableDecodedColumn<T> covers DateOnly?, DateTime? and other types that parse only from characters; NullableConvertedColumn<T> covers T? through an IFixedWidthConverter<T>. Both were possible with attributes and not with the fluent builder. Also adds sync/async parity tests for records longer than a write batch, for generated and layout models.
…to Infrastructure Alignment, TrimMode, OverflowBehavior, FixedColumnAttribute and IFixedWidthConverter<T> now live in FixedWidthParser, so a fluent layout or a converter needs no extra using. The types that are public only for generated code or allocation-free enumeration (FixedWidthRuntime, ColumnFormatOptions, the record enumerables and enumerators, IRecordLineParser and the line-parser structs) move to FixedWidthParser.Infrastructure. The generator emits the new names. BREAKING CHANGE: replace using FixedWidthParser.Attributes and using FixedWidthParser.Processors with using FixedWidthParser. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate methods take cancellationToken, like the write methods, instead of ct. IFixedWidthConverter<T> takes column and bytesWritten, like the rest of the library, instead of field and written. Named arguments make parameter names part of the contract, so they are settled before 2.0 ships. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DegreeOfParallelism is now a property of FixedWidthReaderOptions, ignored by
reads that are not aggregations. The aggregate methods lose their parallel
parameter, and a later parallelism setting is one more property instead of a
new type or parameter.
BREAKING CHANGE: pass new FixedWidthReaderOptions { DegreeOfParallelism = n }
instead of new FixedWidthParallelOptions { DegreeOfParallelism = n }.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ructs ColumnFormatOptions is a plain readonly struct with an explicit constructor instead of a positional record struct, and WriteInto is internal. The new ColumnParseOptions (trim character, trim mode, format) replaces the optional trimChar/trimMode parameters and the format argument of every FixedWidthRuntime parse helper. Generated code stores one of each per column and passes them by in reference, so a later setting is a new constructor overload and property, not a change to helpers that compiled consumer code already binds to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every type in FixedWidthParser.Infrastructure and FixedWidthContext's ReportFailedColumn/ReportShortLine are public only for generated code and allocation-free enumeration; [EditorBrowsable(Never)] keeps them out of completion lists so callers do not build on them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ules README's versioning section no longer claims the enumerator and strategy types can change in a minor version; it states that everything public, the Infrastructure namespace and generated-code helpers included, stays binary compatible across 2.x, and how settings are added. Samples use the root namespace and FixedWidthReaderOptions.DegreeOfParallelism. ARCHITECTURE gains a public-surface section; CHANGELOG covers the namespace move and renames. Package validation suppressions follow the moved types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read, ReadFile, ReadAsync, ReadFileAsync, AggregateFileAsync and AggregateAsync, and IRecordAccumulator, now take `allows ref struct` models, generated and through a layout. An enumerator cannot hold a ref struct field, so, following ExcelReader, MoveNext only finds the next line and Current parses it on access. A foreach still parses each line once. With OnParseError set, MoveNext also parses the line so a bad one can be skipped before it is yielded.
ParallelAggregator now reads through IRandomAccessSource (a file handle today, native memory next) from a data start offset, and can hand lines to an internal ILineSink instead of a model accumulator. The native library uses both; the public API does not change. The model path keeps its own loop: routing it through a generic sink made one-call reads ~40% slower under dynamic PGO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FixedWidthParser.Native publishes as a C-ABI shared library with the same settings as ExcelReader.Native: invariant globalization, an AVX2 floor on x64, a handle table and a per-thread last error. It exports fw_abi_version, fw_last_error and fw_last_parse_error so far. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fw_layout_create validates fw_column_spec arrays into an immutable layout handle shared by every read and write: positions through the core's ColumnLayoutValidator, types, scale, formats, ASCII trim and padding, overflow, and the struct_size contract. fw_layout_free releases it.
fw_read_file and fw_read_memory parse every line straight into native column buffers (int64, float64 with implied decimals, decimal, date, time, timestamp, UTF-8 strings with int32 offsets, validity bitmaps on the first null) through the core's partitioned reader, and publish them as an fw_table freed with fw_free_table. fw_read_options adds header_lines, skip_invalid, encoding, max_line_length and the byte order mark switch; fw_read_result reports lines read and skipped; a parse error sets the line and the layout column. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
added 24 commits
October 24, 2026 03:07
Non-decimal input to a Decimal column went through float64: large integers lost digits and out-of-range values wrapped to int64 min. Integers now scale exactly in int64; NaN, infinity and values outside int64 once scaled raise FixedWidthError before anything is written. Floats keep rounding half to even to the scale, now documented.
…ing work Workers kept every chunk's accumulator until EOF, so memory grew with the source despite the bounded queue. Results now merge left to right as soon as every earlier chunk has merged, and each chunk holds one of four slots per worker until then, so a slow early chunk stalls the producer instead of letting later results pile up. An OperationCanceledException that is not the caller's cancellation now fails its chunk instead of leaving the producer waiting on its slot.
The generator read only members declared on the model, so a derived model silently skipped base-class columns when parsing and wrote filler over them. It now walks the base types as reflection does: the first public member of a name wins, so overrides and 'new' members replace what they hide, and an override inherits [FixedColumn] from the property it overrides. Inherited columns take part in overlap validation.
Decode rented a char array for wide columns before Encoding.GetChars, and its callers' try/finally began only after it returned, so invalid bytes under a strict encoding lost the array to the pool.
lib.exe was always given /machine:x64, so ARM64 builds got an x64 import library. The machine now comes from the target compiler's architecture ID, which holds when cross-compiling. The MSVC tests also define _CRT_SECURE_NO_WARNINGS: /WX turned the std::fopen deprecation warning into an error.
csFastFloat ran only when the decimal separator was '.'; every other culture decoded the column and used the BCL parser, ~40 ns more per value. It now runs with any one-ASCII-character separator when the negative sign is '-', reusing FixedWidthContext.DecimalSeparator from the writer instead of a separate cache, and still falls back unless every byte was consumed. A pt-BR read goes from 98.4 to 58.5 ns/line.
A StringPool lookup cost several times a fresh GetString, so a pooled read ran at 137.8 ns/line against 57.1 without. A per-thread, 4-way set-associative cache, as in ExcelReader's Utf8StringCache, now answers repeated UTF-8 values; every hit is checked against the bytes and a miss still goes to the pool. Pooled reads run at ~70 ns/line. docs/reader-performance.md records both changes and why larger reads in the file engine were not kept.
The README was the NuGet landing page, the whole usage manual and a benchmark report at once. Following ExcelReader, it keeps the overview, installation and a quick start, and links to docs/guide/ (models and layouts, reading, writing, versioning), whose text moved unchanged. The comparison tables were 1.1 numbers from another machine; docs/performance/benchmarks.md has 2.0 numbers, the machine, and why the comparison is not like-for-like.
The changelog said the double fast path ran only for '.', and missed the comma-culture and StringPool speedups, C++ struct mapping and the generator's inherited-columns fix. ARCHITECTURE listed retired diagnostics, STYLEGUIDE a runtime constructor 2.0 no longer has and three of four twin paths, and CONTRIBUTING two of three warnings-as-errors projects and only part of CI, with no steps for the native, Python, C++ or AOT checks. The native README now names fixedwidth_arrow.h, SECURITY.md covers the native library and bindings, and the PR template asks for the changelog and PublicAPI entries. Inherited columns, Python's LayoutOptions, WriteOptions and Table members, and C++'s layout_for, at, table and TypedReader are documented; parallel-aggregate.md says its stream rows predate incremental merging.
2.0 breaks the 1.x API on purpose, so validating against the 1.1.1 baseline only produced a 372-line suppression list regenerated on every break. Validation goes back on with a 2.0.0 baseline once 2.0.0 ships; CONTRIBUTING and the CI step name say so.
…FastDecimal double, float and decimal columns now parse through built-in fast paths that take the culture's decimal separator as a parameter, so comma cultures stay fast. Anything a fast path refuses (thousands separators, trailing text, too many digits) goes to the BCL over the decoded text, which decides the result; the BCL's UTF-8 parser would reject a space standing in for fr-FR's no-break-space group separator. float gets fast_float's binary32 fast path (a mantissa up to 2^24 and a scale up to 10, one rounding) rather than a rounded double, which can be one ulp off. decimal keeps the scale as written. CultureHelpers folds into FixedWidthRuntime, where one decode helper now serves the number fallback, the exact date/time parsers and TryParseDecoded. New tests check float and decimal against the BCL in invariant, de-DE and fr-FR. A 2M-line read of string, int, double, decimal and float columns drops from ~163 to ~122 ns/line.
- ColumnParserFactory: nine near-identical parse-then-assign builders become one Bind over a ColumnValueParser, fed the FixedWidthRuntime helpers as method groups; ForString and ForConverter stop forwarding. - ColumnFormatOptions.WriteInto reuses AlignInPlace instead of repeating the alignment and sign-leftmost logic. - LineBufferState.Fill(Stream) replaces the refill loop copied in the sync enumerator and the native table reader. - ResolveOverflow becomes one switch expression; the unused ParserBuilder.ComputeRequiredLineLength is gone. - FixedColumnAttribute docs no longer mention the removed IUtf8FixedWidthConverter<T>.
EpochDay and Pow10 were declared in both ColumnBuilder and TableFormatter; they move to NativeType. NativeLayout and WriteSettings each had the same Fail helper; NativeApi.InvalidArgument replaces both. GrowableNativeBuffer loses Last, CopyTo and ByteLength, which nothing called.
The hand-written Equals and GetHashCode over fourteen fields become the compiler's, with an IsExternalInit polyfill for netstandard2.0. ModelInfo keeps its own equality, since it leaves Location out. Drops a stale comment on ByteLiteral.
A date or time format made only of fixed-width digit fields (yyyy, MM, dd, HH, mm, ss) and the literals - . T / : is compiled once per column into a FastDate that reads the bytes directly, as ExcelReader's FastDate reads ISO 8601. It steps aside for a non-Gregorian calendar, and for "/" or ":" when the culture's separators differ. DateTime, DateOnly, TimeOnly and native TIMESTAMP columns use it: 85 -> 18 ns per value. int and long columns of plain digits, and the native INT64, implied- decimal FLOAT64 and DECIMAL columns, go through FastInteger, which reuses FastDouble's eight-digit step: 14 -> 9 ns per value. A leading "-" is taken only when the culture's negative sign is "-". Anything either refuses goes to the BCL. New tests check both against it in cultures where the fast paths must not apply (th-TH, de-DE, sv-SE). A single-threaded COTAHIST read through Python drops from 0.920 to 0.678 s; the parallel read barely moves (0.261 -> 0.250 s).
Timed inside the native read, every partition stayed busy but slowed as threads were added (1.0x at 2, 1.7x at 8, 3x at 16), with an in-memory source no better than a file. A read writes ~200 MB of columns: 50,000 demand-zero page faults on one thread, 100,000 in parallel because the concatenation copies everything into fresh memory again. - A partition sizes its column buffers once from its byte count, which fixed-width lines turn into a row count, instead of doubling them from 256 elements. - skip_invalid skips a bad line through an internal OnSkip hook instead of building its exception: numbering B3's trailer counted line breaks from the start of the file on one thread (~27 ms). - Publish concatenates one column per thread. COTAHIST through Python, 1M lines: fw.read 0.250 -> 0.169 s, one thread 0.678 -> 0.580 s.
bench_write.py reads a synthetic COTAHIST file once into pandas, polars and pyarrow tables, then times each writer turning the table back into its record lines, asserting every one reproduces the file's body byte for byte. pandas and polars have no fixed-width writer, so the baselines are numpy.savetxt with printf fields and polars pad + concat_str. The generator now caps a day's quantity at 10^9 shares: its 16-digit volumes did not survive a float64, so no writer could reproduce them. 1M lines: fw.write (pyarrow) 0.259 s, polars 0.722 s, savetxt 4.285 s.
…n parallel At 16 threads fw_write_file took 240 ms where fw_write_memory took 141, with the two equal on one thread: Windows runs one synchronous I/O at a time on a handle, so the parallel batches queued. The file is now opened for asynchronous I/O; each batch still waits for its write, but they run together (240 -> 170 ms). A right-aligned int or long with no format writes its digits from the right straight into the column, then the padding, with "-" kept leftmost (19 -> 9 ns): formatting at the start and aligning after cost a copy. It applies only when the provider's negative sign is "-". A date or time whose format is only digit fields reuses the reader's compiled FastDate to write its digits (47 -> 23 ns). A test checks both against the BCL's char path in four cultures, both alignments, and zero and space padding. COTAHIST through Python, 1M lines: fw.write 0.259 -> 0.162 s, one thread 0.838 -> 0.550 s.
A whole-source read allocates the final columns once and lends each partition the slice its byte offset gives, with the first line's length as the stride. Numeric, date and time values and STRING bytes are parsed in place; publishing moves each partition down to close the gaps skipped lines and trimmed text leave, and hands the buffers over. A partition that outgrows its slice copies its column out, and that column is concatenated as before. STRING offsets are still copied. A parallel COTAHIST read through Python: fw.read 0.184 -> 0.153 s, fw.read_polars 0.198 -> 0.165 s.
Follows ExcelReader's release workflow. A v* tag now builds the native library on Windows x64, macOS arm64 and x64, and Linux glibc and musl on x64 and arm64; tests and builds one wheel per platform with the tag's version; publishes the NuGet package, the wheels to PyPI through Trusted Publishing, and the native libraries and an SBOM to the GitHub Release. A tag with a "-" is a prerelease. check_wheel_tag.py and build_musl_wheel.sh come from ExcelReader. pyarrow has no musllinux wheels, so the pandas, polars, Arrow and write tests skip without it and the musl job tests the NumPy read.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #79 +/- ##
==========================================
Coverage ? 94.35%
==========================================
Files ? 44
Lines ? 2779
Branches ? 464
==========================================
Hits ? 2622
Misses ? 100
Partials ? 57
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Benchmark ResultsMeasured on Benchmarks.Perf.AsyncReaderBenchmarks
Benchmarks.Perf.ComparisonBenchmarks
Benchmarks.Perf.ComparisonWriterBenchmarks
Benchmarks.Perf.ParserBenchmarks
Benchmarks.Perf.ReaderBenchmarks
Benchmarks.Perf.ReaderPoolingBenchmarks
Benchmarks.Perf.ReaderStartupBenchmarks
Benchmarks.Perf.SourceGenParserBenchmarks
Benchmarks.Perf.WriterBenchmarks
|
Follows ExcelReader's workflows. - benchmark-pr is opt-in through the run-benchmarks label, reads its groups from tests/Benchmarks/benchmark-groups.json (shared with benchmark-pages), guards that every benchmark belongs to a group, and comments a delta against the published master series. benchmark-pages runs only when its paths change. The stale IntHeavyParserBenchmarks pattern is gone. - ci filters by path and skips drafts. The required Build & Test jobs always start and skip their steps. Linux builds the solution with the analyzers, checks whitespace, packs and runs every test with coverage; Windows and macOS build and run the library tests without analyzers. The dependency audit moves to audit.yml, weekly and on dependency changes. - codeql filters by path the same way, adds Python, and builds only the shipped projects without analyzers. - native-bindings splits into Python, C++ and musl jobs with their own filters, tests every platform a release ships, and builds the wheel. - codecov.yml flags C# and Python coverage with carryforward; coverage.runsettings leaves out generated code and the C ABI exports. - release promotes PublicAPI.Unshipped to Shipped in a PR after a stable tag. - global.json pins SDK 10.0.402, which every workflow now installs and caches. - Dependabot groups follow the test and benchmark packages in use.
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 2.0 release. The library now reads and writes bytes only and goes through one entry point,
FixedWidth. On top of it sits a NativeAOT C library, with Python and C++ bindings over that. Settings live in options objects andFixedWidthContext, so later 2.x releases can add settings without another breaking change.The full list of changes, and a 1.x → 2.0 migration table, is in the CHANGELOG.
Breaking changes
charand UTF-8 twins of every reader, parser and writer are gone, and column positions are bytes. Text columns are decoded with an ASCII-compatible encoding, UTF-8 by default.FixedWidthmethod takes either a generated model or aFixedWidthLayout<T>.FixedWidthParser<T>,FixedWidthReader<T>andFixedWidthWriter<T>are removed.FixedWidthParser. Plumbing that only generated code uses moved toFixedWidthParser.Infrastructureand is hidden from IntelliSense.FixedWidthReaderOptions/FixedWidthWriterOptionsreplace the positional provider, pool and buffer-size parameters, andFixedWidthParallelOptionsis folded intoFixedWidthReaderOptions.nullnow means the invariant culture for every column type, not onlydouble/float.Currentis read, not fromMoveNext.PipeReaderis removed; usepipeReader.AsStream(), which is about 2x faster.net10.0only. .NET 8 support ends on 10 November 2026.FastDouble/FastDecimal.New
FixedWidthLayout<T>: build a layout from the attributes, or describe it in code withFixedWidthLayout.Build<T>(...). No reflection, so it works under trimming and NativeAOT.AggregateFileAsyncsplits a file into byte ranges, andAggregateAsyncsplits a stream into chunks. Line numbers in errors match a sequential read.FixedWidthParseExceptiongives the line, column and raw value.OnParseErrorlets a caller skip bad lines.MaxLineLengthbounds untrusted input.ref structmodels, nullable value-type columns, per-property converters, and the source-generated writer.FixedWidthParser.Native(fixedwidth.h):fixedwidthPython package: results as NumPy views, pyarrow, pandas or polars.FixedWidthMapper<T>.v*tag publishes the NuGet package, Python wheels to PyPI asfixedwidthparser(Windows x64, macOS arm64/x64, Linux glibc and musl on x64/arm64), and each platform's native library to the GitHub Release.Performance
Measured on a synthetic 1,000,000-line B3 COTAHIST file (235 MiB) through Python, on a Ryzen 7 5700X (16 threads). Details are in docs/python-benchmarks.md.
fw.read, parallelfw.read, 1 threadfw.write(pyarrow), parallelpandas.read_fwftakes 14.8 s on the same file.What made the difference:
Testing
lib/net10.0and the generator underanalyzers/dotnet/cs.Before tagging 2.0.0
PublicAPI.Unshipped.txt(197 entries). After 2.0 ships, any change to these entries is breaking.fixedwidthparser,GabrielMarquezMatte/FixedWidthParser,release.yml.v2.0.0-rc.1first; a tag containing-is published as a prerelease.PublicAPI.Shipped.txt, and re-enable package validation with baseline 2.0.0 (see the note inFixedWidthParser.csproj).