Support CTF 2 LTTng traces by Nemati · Pull Request #141 · microsoft/Microsoft-Performance-Tools-Linux-Android

Hani Nemati (Nemati) · 2026-10-01T16:38:51Z

LTTng 2.15+ records CTF 2 (JSON metadata) by default, which the TSDL parser could not read. Add a CTF 2 metadata parser that maps CTF 2 field classes onto the existing descriptor model, and select the parser from the metadata content.

Also fix issues that CTF 2 / multi-channel LTTng traces exposed:

Accept any _ stream file, not only chan*
Event ids are unique per stream: look up event descriptors and generic event kinds by (stream, id)
Look up streams by id and read the packet context after determining the stream id
Decode explicitly big-endian whole-byte integers
Read the event-specific context when one is defined

LTTng 2.15+ records CTF 2 (JSON metadata) by default, which the TSDL parser could not read. Add a CTF 2 metadata parser that maps CTF 2 field classes onto the existing descriptor model, and select the parser from the metadata content. Also fix issues that CTF 2 / multi-channel LTTng traces exposed: - Accept any <channel>_<cpu> stream file, not only chan* - Event ids are unique per stream: look up event descriptors and generic event kinds by (stream, id) - Look up streams by id and read the packet context after determining the stream id - Decode explicitly big-endian whole-byte integers - Read the event-specific context when one is defined Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

John Pohovey (jhpohovey)

Most of these comments are small style and clarity things; though there are two or three that revolve around logic correctness that I think you should take a quick look at first and let me know if I am thinking nonsense or not

John Pohovey (jhpohovey) · 2026-10-01T17:19:31Z

            ReadPacketHeader();
-            ReadPacketContext();

+            // The packet context layout depends on the stream, which is identified in the packet header.
            this.DetermineStreamIndex();

+            ReadPacketContext();


So thinking out this logic -- this likely would have been previously a latent bug when reading packet metadata? given that the when that streamId is updated controls data read within ReadPacketContext().

Mostly just confirming here that the streamId needs to be updated to the current packet here, and there isn't some desired behavior for off-by-one or similar.

John Pohovey (jhpohovey) · 2026-10-01T18:03:14Z

+            if (GetValue(clockClass, "offset-from-origin") is Dictionary<string, object> offset)
+            {
+                // The descriptor expects the full offset in clock cycles.
+                decimal cycles = 0;
+                if (TryGetNumber(offset, "seconds", out var seconds))
+                {
+                    cycles += decimal.Parse(seconds.Text, CultureInfo.InvariantCulture) * frequency;
+                }
+
+                if (TryGetNumber(offset, "cycles", out var offsetCycles))
+                {
+                    cycles += decimal.Parse(offsetCycles.Text, CultureInfo.InvariantCulture);
+                }
+
+                if (cycles > 0)
+                {
+                    bag.AddValue("offset", decimal.ToUInt64(cycles).ToString(CultureInfo.InvariantCulture));
+                }
+            }


So in the data block, while cycles is required to be non-negative, seconds doesn't have such a constraint (e.g., to be an offset chronologically preceding the origin clock).

Is the intention here that we throw out clock offsets where the origin is more than 1 second forward in time relative to the offset?

nit: i also think it would be nice to link https://diamon.org/ctf/#clock-offset here

John Pohovey (jhpohovey) · 2026-10-01T18:34:02Z

+        private static CtfClockDescriptor CreateClock(Dictionary<string, object> clockClass)
+        {
+            // CTF 2 data stream classes refer to clock classes by id; older drafts only had a name.
+            string clockName = GetString(clockClass, "id") ?? GetRequiredString(clockClass, "name");


nit: consider linking in https://diamon.org/ctf/#cc-frag

John Pohovey (jhpohovey) · 2026-10-01T18:36:13Z

+            return this.metadataBuilder;
+        }
+
+        private void AddTraceClass(Dictionary<string, object> traceClass, Dictionary<string, object> preamble)


nit: consider link in https://diamon.org/ctf/#tc-frag

John Pohovey (jhpohovey) · 2026-10-02T00:41:41Z

+            return new CtfClockDescriptor(bag);
+        }
+
+        private void AddDataStreamClass(Dictionary<string, object> dataStreamClass)


nit: consider linking https://diamon.org/ctf/#dsc-frag

John Pohovey (jhpohovey) · 2026-10-02T01:25:01Z

+    /// <summary>
+    /// CTF 2 variable-length (unsigned or signed LEB128) integer.
+    /// </summary>


nit: add link to https://diamon.org/ctf/#vl-int-fc

John Pohovey (jhpohovey) · 2026-10-02T01:30:15Z

+        public override int Align => 8;
+
+        /// <inheritdoc />
+        public override CtfFieldValue Read(IPacketReader reader, CtfFieldValue parent = null)


nit: While read is true, i'd lean towards having a public Read call an internal DecodeLEB128, which would self-explain what the below algorithm is doing.

John Pohovey (jhpohovey) · 2026-10-02T01:34:55Z

+namespace CtfPlayback.Metadata
+{
+    /// <summary>
+    /// Helpers for reading a metadata stream, which may be plain text or packetized (see https://diamon.org/ctf/#spec7.1).


https://diamon.org/ctf/#spec7.1 isn't a real link

Likely meant https://diamon.org/ctf/#metadata-stream?

John Pohovey (jhpohovey) · 2026-10-02T01:50:42Z

+    /// </summary>
+    internal static class CtfMetadataText
+    {
+        private const uint PacketMagic = 0x75d11d57;


this needs a comment i think to just note that this is not an abritrary value here, but defined in the spec: https://diamon.org/ctf/files/CTF2-PMETA-1.0.html

There is a different magic number for event an data packets as opposed to this metadata packet.

John Pohovey (jhpohovey) · 2026-10-02T02:05:03Z

                int h2 = id.GetHashCode();
-                return ((h1 << 5) + h1) ^ h2;
+                int h3 = streamId.GetHashCode();
+                return ((((h1 << 5) + h1) ^ h2) * 31) ^ h3;


why this specific computation? arbitrary? why not use a SHA256 or MD5?

Or if to simply retain + expand past behavior, that seems generally reasonable.

John Pohovey (jhpohovey)

Approving pending resolutions of whichever above

ivberg · 2026-10-02T19:24:57Z

+        private const uint PacketMagic = 0x75d11d57;
+
+        // magic(4) + uuid(16) + checksum(4) + content_size(4) + packet_size(4) + 5 single-byte fields
+        private const int PacketHeaderSize = 37;


CTF2-PMETA-1.0 defines a minimum 44-byte metadata packet header, not 37 bytes: after the five legacy single-byte fields it adds 3 reserved bytes and a 32-bit header-size field. With a compliant packet, starting the JSON at byte 37 leaves seven binary header bytes before the record separator, so CTF 2 detection/parsing fails. The new packetized-metadata test currently constructs the same nonconforming 37-byte header and therefore masks this. Could this read the CTF 2 declared header size (while retaining the 37-byte CTF 1 path)? See https://diamon.org/ctf/files/CTF2-PMETA-1.0.html

ivberg · 2026-10-02T19:24:59Z

+        {
+            if (metadata.Length < PacketHeaderSize || BitConverter.ToUInt32(metadata, 0) != PacketMagic)
+            {
+                return Encoding.UTF8.GetString(metadata);


The packet metadata specification permits the magic and following integer fields in either byte order; the magic's encoding indicates which order to use. On our little-endian runtime, BitConverter.ToUInt32() will not recognize a valid big-endian magic and this falls through to decoding the binary packet as UTF-8. Please detect both magic encodings and use the detected byte order for content_size, packet_size, and the CTF 2 header-size field.

ivberg · 2026-10-02T19:25:01Z

+                default:
+                    throw new CtfMetadataException("Only 32-bit and 64-bit CTF 2 floating point numbers are supported.");
+            }
+


This accepts big-endian floating-point field classes, but CtfFloatingPointDescriptor.Read() passes the source bytes directly to BitConverter without applying its ByteOrder property. A valid big-endian float therefore produces a silently incorrect value. Could we either implement the byte swap in the descriptor (and test it) or reject big-endian floating-point classes here until they are supported?

Copilot

Copilot review overview

🟡 Changes recommended

Valid traces can be misdecoded due to trace-level event collisions, metadata endianness, field-location origins, and signed 64-bit handling.

Review effort: Balanced
Findings: 3 · 1

Open (4)

Preserve field location origin during array length resolution · New
Decode packetized metadata headers using detected byte order · New
Scope registered event kinds per trace or processing run · New
Fix signed big-endian decoding for integers wider than 32 bits · New

What changed in this PR

Adds CTF 2 metadata and multi-channel LTTng trace support while retaining CTF 1.8 compatibility.

Changes:

Adds JSON metadata parsing with automatic CTF version detection.
Makes stream and event lookup stream-aware.
Expands stream discovery, decoding, tests, and documentation.

File	Description
`LTTngDataExtUnitTest/LTTngCtf2UnitTest.cs`	Adds end-to-end CTF 2 tests.
`LTTngDataExtensions/DataOutputTypes/LTTngGenericEvent.cs`	Keys event kinds by stream.
`LTTngCds/CtfExtensions/ZipArchiveInput/LTTngZipArchiveInput.cs`	Expands ZIP stream discovery.
`LTTngCds/CtfExtensions/LTTngStreamFiles.cs`	Recognizes channel/CPU filenames.
`LTTngCds/CtfExtensions/LTTngPlaybackCustomization.cs`	Selects parsers and performs stream-aware lookup.
`LTTngCds/CtfExtensions/LTTngMetadata.cs`	Indexes events by stream and ID.
`LTTngCds/CtfExtensions/FolderInput/LTTngFolderInput.cs`	Expands folder stream discovery.
`LTTngCds/CtfExtensions/Descriptors/EventDescriptor.cs`	Supports event-specific contexts.
`LTTngCds/CookerData/LTTngEvent.cs`	Exposes stream IDs.
`LinuxTraceLogCapture.md`	Documents CTF 2 and performance counters.
`CtfUnitTest/Ctf2MetadataTests.cs`	Tests CTF 2 parsing and decoding.
`CtfPlayback/Metadata/Types/CtfIntegerDescriptor.cs`	Adds big-endian integer handling.
`CtfPlayback/Metadata/CtfVersionDetectingMetadataParser.cs`	Detects CTF metadata versions.
`CtfPlayback/Metadata/CtfMetadataText.cs`	Reads plain or packetized metadata.
`CtfPlayback/Metadata/CtfMetadataExtensions.cs`	Looks up streams by ID.
`CtfPlayback/Metadata/Ctf2/Ctf2VariableLengthIntegerDescriptor.cs`	Decodes LEB128 integers.
`CtfPlayback/Metadata/Ctf2/Ctf2MetadataParser.cs`	Maps CTF 2 metadata to descriptors.
`CtfPlayback/Metadata/Ctf2/Ctf2Json.cs`	Adds a metadata JSON reader.
`CtfPlayback/EventStreams/Interfaces/ICtfEvent.cs`	Adds stream identity to events.
`CtfPlayback/EventStreams/CtfPacket.cs`	Selects streams before reading contexts.
`CtfPlayback/EventStreams/CtfEvent.cs`	Uses stream-aware descriptors and contexts.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        private string GetFieldLocation(Dictionary<string, object> fieldClass, string property, BuildContext context)
+        {
+            return string.Join(".", GetFieldLocationPath(fieldClass, property).Select(context.MapName));
+        }


+        /// </summary>
+        internal static string Read(byte[] metadata)
+        {
+            if (metadata.Length < PacketHeaderSize || BitConverter.ToUInt32(metadata, 0) != PacketMagic)


        private class Key
        {
            private string domain;
+            private uint streamId;


            // if the byte order is "be" or "network", then it is big endian
            // if the byte order is "le", then it is little endian
+            // Only explicitly big endian, whole-byte integers are handled for now (e.g. IPv4 addresses in network order).
+            if (this.IsExplicitlyBigEndian && (this.Size % 8) == 0)


Hani Nemati (Nemati) assigned John Pohovey (jhpohovey) Oct 1, 2026

John Pohovey (jhpohovey) requested changes Oct 2, 2026

View reviewed changes

John Pohovey (jhpohovey) approved these changes Oct 2, 2026

View reviewed changes

ivberg reviewed Oct 2, 2026

View reviewed changes

John Pohovey (jhpohovey) requested a balanced review from Copilot October 4, 2026 16:00

Copilot started reviewing on behalf of John Pohovey (jhpohovey) October 4, 2026 16:01 View session

Copilot AI reviewed Oct 4, 2026

View reviewed changes

Conversation

Hani Nemati (Nemati) commented Oct 1, 2026

Uh oh!

John Pohovey (jhpohovey) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

John Pohovey (jhpohovey) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants