From 33389f3add941a9927ee0a179c423655e298bb36 Mon Sep 17 00:00:00 2001 From: Rob Johnson Date: Sun, 11 Oct 2026 22:13:23 -0700 Subject: [PATCH 1/7] Persist structure versions and per-instance read configuration --- docs/on-disk-format.md | 77 +++++++++++++++ include/splinterdb/data.h | 5 + src/allocator.h | 5 +- src/blob.c | 2 + src/btree.h | 3 + src/log_data.h | 16 +++- src/mini_allocator.c | 5 + src/mini_allocator.h | 3 +- src/routing_filter.c | 152 +++++++++++++++++++++-------- src/routing_filter.h | 23 +++-- src/shard_log.c | 25 ++++- src/shard_log.h | 1 + src/superblock.c | 107 +++++++++++++++++---- src/superblock.h | 11 ++- src/trunk.c | 47 +++++++-- src/trunk.h | 1 + tests/functional/filter_test.c | 37 ++++++- tests/functional/log_test.c | 11 ++- tests/unit/splinter_test.c | 170 +++++++++++++++++++++++++++++++++ tests/unit/superblock_test.c | 88 ++++++++++++++--- 20 files changed, 686 insertions(+), 103 deletions(-) create mode 100644 docs/on-disk-format.md diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md new file mode 100644 index 000000000..895b98c55 --- /dev/null +++ b/docs/on-disk-format.md @@ -0,0 +1,77 @@ +# On-disk versions and read configuration + +Format 8 rejects files written by earlier formats. There is no migration reader. +Creation writes both superblock slots. Mount and bootstrap select the newest +checksummed record before checking its structure versions, so an unsupported +newest version cannot silently select an older checkpoint. A damaged slot still +permits recovery from its valid peer. Recognition of a future superblock layout +is limited to the current checksum layout. + +| Structure | Version location | Persisted configuration required to read | +| --- | --- | --- | +| Superblock | Superblock header | Device size, page size, extent size | +| Trunk | Superblock | None beyond device geometry and application callbacks | +| Branch B-tree | Each branch reference in its trunk bundle | None beyond device geometry and application callbacks | +| Mini allocator | Each metadata page header | Entries already carry extent number, page type and batch; geometry is global | +| Routing filter | Descriptor in its trunk bundle | Fingerprint width, index width, hash seed, value width, counts and metadata address | +| Shard log | Log head in the superblock | Checksum seed and stream nonce | +| Reference-count allocator | Superblock | Device geometry; map address is already persisted | +| Blob | Existing blob descriptor version | Length and checksum already persisted; descriptor length and geometry determine the extent list | + +Memtables are reconstructed and have no persistent configuration record. +Fanout, incorporation thresholds, scheduling, cache size, prefetching and +statistics settings control construction or execution, not interpretation of +existing pages. Blob allocation rounding rules are part of its versioned +encoding, rather than a mutable reader setting. + +Existing routing filters use their descriptor's widths and seed for synchronous +and asynchronous reads. Extending a maplet retains its encoding and hash seed; +compaction hashes replacement branches with that seed. New maplets use current +construction settings. Split sizing respects the smallest capacity among the +existing maplets it may extend. Cardinality estimation across different seeds +uses a conservative count because their fingerprint samples cannot be compared. + +When combined with optional hash-filtered ranges, an old maplet whose seed +differs from the range predicate's seed must conservatively include branches; +the exact range predicate then uses one common query seed. Hashing the query +under each old seed alone can exclude keys that collide only under the query +seed. The independent range-filtering branch requires this integration change. + +Each recovered log reads with its saved checksum seed. Newly constructed logs +may use the current creation configuration. + +The application must supply matching key comparison, key hash algorithm, merge +semantics and key/value encodings on reopen. Function pointers and application +schemas cannot be persisted as portable identities. The persisted hash seed is +passed to the supplied hash callback. This change does not introduce a portable +byte-order encoding or promise compatibility between arbitrary architectures. + +# Validation and measurements + +Debug validation: 16 superblock tests, 33 internal Splinter tests, 47 public quick +tests, full functional routing-filter and log suites. Added regressions cover +unknown versions, changed routing widths/seeds across reopen and further +compaction, synchronous/asynchronous filter reads, and changed log checksum +creation defaults. + +Initial resident ABBA measurements found a 2.8–4.0% point-read regression. +After removing redundant validation, a second campaign used a single loader, +200,000 keys in a 128 MiB cache and 7-second read phases. Point throughput was +still 3.4–3.6% lower; range results ranged from -2.5% to +2.6%. The 2-million-key, +64 MiB-cache campaign reported +2.0–2.4% point throughput and +2.7–6.6% range +throughput, with comparable I/O per operation. Device/run variability and +independently constructed trees prevent attributing those gains to metadata. +All samples had zero correctness violations and errors. Brief load samples do +not establish write-performance neutrality. + +A subsequent reader change removes a full routing-config copy from each filter +lookup, passing the saved index width and hash seed directly. Version and bounds +validation remains at the same boundaries. The full functional filter suite and +changed-configuration reopen/compaction regression pass after that change. +Its profiling and fixed-image old/new reader measurements are pending; this +implementation milestone does not claim performance acceptance. + +Campaign artifacts: `.cache/projects-20261011/benchmarks/disk-format/` in the +main checkout. The common harness and build provenance are retained there; +benchmark database files were on a verified disk filesystem, with the shared +benchmark lock excluding other benchmarks, builds and tests. diff --git a/include/splinterdb/data.h b/include/splinterdb/data.h index 6e08f961a..fbf507557 100644 --- a/include/splinterdb/data.h +++ b/include/splinterdb/data.h @@ -175,6 +175,11 @@ typedef void (*message_to_str_fn)(const data_config *cfg, * 4. How to convert between messages and values (encode and decode functions) * 4. Few other debugging aids on how-to print & diagnose messages. * + * Callback code and application key/value schemas are not serialized. Reopening + * an existing database requires callbacks with the same key ordering, hash + * algorithm, and message interpretation used when it was written. Persisted + * routing filters supply their own hash seed to key_hash. + * * default_data_config.c is a simple reference implementation, provided * as a "batteries included" solution. If an application wishes * to do something differently, it has to provide these implementations. diff --git a/src/allocator.h b/src/allocator.h index f1cbb2052..5eec61b40 100644 --- a/src/allocator.h +++ b/src/allocator.h @@ -22,8 +22,9 @@ typedef uint64 allocator_root_id; /* * ---------------------------------------------------------------------------- * Different types of pages managed by SplinterDB: - * This is currently not a Disk-resident value, but different modules that - * access a type of a page "know" to expect a page of a specific type / format. + * These numeric values are persisted in mini-allocator extent entries. Changing + * their meaning requires a mini-allocator format version change. Modules that + * access a page also know which structure format to expect. * * Brief overview of the data structures involved in different page types: * diff --git a/src/blob.c b/src/blob.c index 4c420b9ee..d90931fa1 100644 --- a/src/blob.c +++ b/src/blob.c @@ -5,6 +5,8 @@ #include "platform_sleep.h" #include "poison.h" +/* This rounding rule is part of BLOB_FORMAT: the decoder uses it to infer + * address-entry lengths. Changing it requires a new blob format reader. */ #define MIN_LIVE_PERCENTAGE (90ULL) static platform_status diff --git a/src/btree.h b/src/btree.h index 6afdcb55f..96b495ef1 100644 --- a/src/btree.h +++ b/src/btree.h @@ -33,6 +33,9 @@ */ #define BTREE_MAX_HEIGHT (8) +/* Immutable branch version is stored in its trunk branch_ref. */ +#define BTREE_BRANCH_FORMAT_VERSION (1) + /* * Mini-allocator uses separate batches for each height of the BTree. * Therefore, the max # of mini-batches that the mini-allocator can track diff --git a/src/log_data.h b/src/log_data.h index a3a8a2f6a..c5c35adab 100644 --- a/src/log_data.h +++ b/src/log_data.h @@ -9,6 +9,8 @@ #include "splinterdb/platform_linux/public_platform.h" +#define SHARD_LOG_FORMAT_VERSION (1) + typedef struct log_nonce { uint64 high; uint64 low; @@ -19,13 +21,17 @@ _Static_assert(sizeof(log_nonce) == 16, "on-disk log nonce layout changed"); /* * The on-disk head of one mini-allocator-backed log stream: the data head * (where replay begins), the metadata head (which owns the stream's extents), - * and a per-stream nonce that validates its pages. Fixed at creation and + * and a per-stream nonce that validates its pages. The format version and + * checksum seed select the reader independently of new-log configuration. + * Fixed at creation and * shared by the log interface and higher-level durable records. */ -typedef struct log_head { +typedef struct ONDISK log_head { uint64 addr; uint64 meta_addr; log_nonce nonce; + uint64 checksum_seed; + uint16 format_version; } log_head; _Static_assert(offsetof(log_head, addr) == 0, @@ -33,7 +39,7 @@ _Static_assert(offsetof(log_head, addr) == 0, _Static_assert(offsetof(log_head, meta_addr) == 8, "log metadata address layout changed"); _Static_assert(offsetof(log_head, nonce) == 16, "log nonce offset changed"); -_Static_assert(sizeof(log_head) == 32, "on-disk log head layout changed"); +_Static_assert(sizeof(log_head) == 42, "on-disk log head layout changed"); static inline bool32 log_nonce_is_equal(log_nonce left, log_nonce right) @@ -45,5 +51,7 @@ static inline bool32 log_head_is_equal(log_head left, log_head right) { return left.addr == right.addr && left.meta_addr == right.meta_addr - && log_nonce_is_equal(left.nonce, right.nonce); + && log_nonce_is_equal(left.nonce, right.nonce) + && left.checksum_seed == right.checksum_seed + && left.format_version == right.format_version; } diff --git a/src/mini_allocator.c b/src/mini_allocator.c index cf31c5102..a650888c8 100644 --- a/src/mini_allocator.c +++ b/src/mini_allocator.c @@ -37,6 +37,7 @@ typedef struct ONDISK mini_meta_hdr { uint64 meta_page_no; uint64 pos; uint32 num_entries; + uint16 format_version; char entry_buffer[]; } mini_meta_hdr; @@ -179,6 +180,7 @@ mini_init_meta_page(mini_allocator *mini, hdr->meta_page_no = meta_page_no; hdr->pos = offsetof(typeof(*hdr), entry_buffer); hdr->num_entries = 0; + hdr->format_version = MINI_ALLOCATOR_FORMAT_VERSION; } /* @@ -1129,6 +1131,9 @@ mini_validate_meta_header(const mini_meta_hdr *hdr, bool32 check_page_no, uint64 meta_addr) { + if (hdr->format_version != MINI_ALLOCATOR_FORMAT_VERSION) { + return mini_metadata_corruption("unsupported format version", meta_addr); + } uint64 first_entry_offset = offsetof(mini_meta_hdr, entry_buffer); uint64 max_entries = (page_size - first_entry_offset) / sizeof(meta_entry); if (check_prev && hdr->prev_meta_addr != expected_prev) { diff --git a/src/mini_allocator.h b/src/mini_allocator.h index ed02c1fac..2cd9822ba 100644 --- a/src/mini_allocator.h +++ b/src/mini_allocator.h @@ -26,7 +26,8 @@ * extents. This batch-size is somewhat of an artificial limit to manage this * contiguity. */ -#define MINI_MAX_BATCHES 16 +#define MINI_MAX_BATCHES 16 +#define MINI_ALLOCATOR_FORMAT_VERSION (1) /* * mini_allocator: Mini-allocator context. diff --git a/src/routing_filter.c b/src/routing_filter.c index c7d5dddbd..4eb8a4bce 100644 --- a/src/routing_filter.c +++ b/src/routing_filter.c @@ -179,25 +179,39 @@ routing_get_index(uint32 fp, size_t index_remainder_and_value_size) } static inline void -routing_filter_get_remainder_and_value(const routing_config *cfg, - uint32 *data, - uint32 pos, +routing_filter_get_remainder_and_value(uint32 *data, + uint32 pos, uint32 *remainder_and_value, size_t remainder_value_size) { *remainder_and_value = PackedArray_get(data, pos, remainder_value_size); } +/* Construction settings never reinterpret an existing filter. Geometry and + * callbacks still come from the mounted database and application. Call only + * after get_params has checked the descriptor (including its shift widths). */ +static routing_config +routing_filter_read_config(const routing_config *cfg, + const routing_filter *filter) +{ + routing_config result = *cfg; + debug_assert(routing_filter_descriptor_is_valid(filter)); + if (filter->addr != 0) { + result.fingerprint_size = filter->fingerprint_size; + result.log_index_size = filter->log_index_size; + result.index_size = 1U << filter->log_index_size; + result.seed = filter->hash_seed; + } + return result; +} + static platform_status routing_filter_get_params(const routing_config *cfg, const routing_filter *filter, routing_filter_params *params) { - if (cfg == NULL || cfg->cache_cfg == NULL || filter == NULL || params == NULL - || cfg->fingerprint_size == 0 || cfg->fingerprint_size > 32 - || cfg->log_index_size >= 32 - || cfg->log_index_size > cfg->fingerprint_size - || cfg->index_size != (1U << cfg->log_index_size)) + if (cfg == NULL || cfg->cache_cfg == NULL || filter == NULL + || params == NULL) { return STATUS_INVALID_STATE; } @@ -211,21 +225,21 @@ routing_filter_get_params(const routing_config *cfg, } params->log_num_buckets = 31 - __builtin_clz(filter->num_fingerprints); - if (params->log_num_buckets < cfg->log_index_size) { - params->log_num_buckets = cfg->log_index_size; + if (params->log_num_buckets < filter->log_index_size) { + params->log_num_buckets = filter->log_index_size; } if (params->log_num_buckets > filter->fingerprint_size) { return STATUS_INVALID_STATE; } - params->log_num_indices = params->log_num_buckets - cfg->log_index_size; + params->log_num_indices = params->log_num_buckets - filter->log_index_size; params->num_indices = 1U << params->log_num_indices; params->remainder_size = filter->fingerprint_size - params->log_num_buckets; params->value_size = filter->value_size; params->remainder_and_value_size = params->remainder_size + params->value_size; params->index_remainder_and_value_size = - params->remainder_and_value_size + cfg->log_index_size; + params->remainder_and_value_size + filter->log_index_size; uint64 extent_size = cfg->extent_size; if (params->remainder_and_value_size == 0 @@ -294,6 +308,7 @@ routing_remainder_block_size(uint64 num_remainders, uint32 bits_per_item) static platform_status routing_header_view_init(const routing_config *cfg, + uint32 index_size, page_handle *filter_page, uint64 hdr_raw_addr, uint32 bits_per_item, @@ -317,8 +332,9 @@ routing_header_view_init(const routing_config *cfg, return STATUS_INVALID_STATE; } - uint64 encoding_size = (num_remainders + cfg->index_size - 1) / 8 + 4; - uint64 header_length = routing_header_size(cfg, num_remainders); + uint64 encoding_size = (num_remainders + index_size - 1) / 8 + 4; + uint64 header_length = + ROUNDUP(sizeof(routing_hdr) + encoding_size, sizeof(uint32)); uint64 remainder_bytes = routing_remainder_block_size(num_remainders, bits_per_item); uint64 bytes_available = page_size - header_off; @@ -354,13 +370,13 @@ routing_unlock_and_unget_page(cache *cc, page_handle *page) *---------------------------------------------------------------------- */ static platform_status -routing_get_bucket_bounds(const routing_config *cfg, +routing_get_bucket_bounds(uint32 index_size, const routing_header_view *view, uint64 bucket_offset, uint64 *start, uint64 *end) { - if (bucket_offset >= cfg->index_size) { + if (bucket_offset >= index_size) { return STATUS_INVALID_STATE; } @@ -467,7 +483,7 @@ routing_get_bucket_counts(const routing_config *cfg, } static platform_status -routing_filter_lookup_view(const routing_config *cfg, +routing_filter_lookup_view(uint32 index_size, const routing_filter_params *params, uint32 fp, const routing_header_view *view, @@ -476,10 +492,10 @@ routing_filter_lookup_view(const routing_config *cfg, uint32 encoded_fp = fp << params->value_size; uint32 bucket = routing_get_bucket(encoded_fp, params->remainder_and_value_size); - uint32 bucket_off = bucket % cfg->index_size; + uint32 bucket_off = bucket % index_size; uint64 start, end; platform_status rc = - routing_get_bucket_bounds(cfg, view, bucket_off, &start, &end); + routing_get_bucket_bounds(index_size, view, bucket_off, &start, &end); if (!SUCCESS(rc)) { return rc; } @@ -495,8 +511,7 @@ routing_filter_lookup_view(const routing_config *cfg, for (uint64 pos = end; pos > start;) { pos--; uint32 found_remainder_and_value; - routing_filter_get_remainder_and_value(cfg, - (uint32 *)view->remainders, + routing_filter_get_remainder_and_value((uint32 *)view->remainders, pos, &found_remainder_and_value, params->remainder_and_value_size); @@ -560,11 +575,22 @@ routing_filter_add(cache *cc, return STATUS_BAD_PARAM; } + if (cfg == NULL || cfg->cache_cfg == NULL + || (old_filter->addr == 0 + && (cfg->fingerprint_size == 0 || cfg->fingerprint_size > 32 + || cfg->log_index_size >= 32 + || cfg->log_index_size > cfg->fingerprint_size + || cfg->index_size != (1U << cfg->log_index_size)))) + { + return STATUS_BAD_PARAM; + } routing_filter_params old_params; platform_status rc = routing_filter_get_params(cfg, old_filter, &old_params); if (!SUCCESS(rc)) { return rc; } + routing_config read_cfg = routing_filter_read_config(cfg, old_filter); + cfg = &read_cfg; // old filter uint32 old_num_indices = 1; @@ -592,6 +618,8 @@ routing_filter_add(cache *cc, : old_filter->fingerprint_size; filter->format_version = ROUTING_FILTER_FORMAT_VERSION; filter->fingerprint_size = fingerprint_size; + filter->hash_seed = cfg->seed; + filter->log_index_size = cfg->log_index_size; uint32 log_num_buckets = 31 - __builtin_clz(filter->num_fingerprints); if (log_num_buckets < cfg->log_index_size) { @@ -797,6 +825,7 @@ routing_filter_add(cache *cc, } routing_header_view old_view; rc = routing_header_view_init(cfg, + cfg->index_size, old_filter_node, hdr_raw_addr, old_remainder_and_value_size, @@ -1116,6 +1145,10 @@ routing_filter_estimate_unique_fp(cache *cc, uint64 total_num_fp = 0; uint32 common_fingerprint_size = 32; bool32 has_sampled_filter = FALSE; + uint32 max_index_size = 0; + bool32 have_seed = FALSE; + uint32 hash_seed = 0; + bool32 mixed_seeds = FALSE; for (uint64 i = 0; i != num_filters; i++) { routing_filter_params params; platform_status rc = routing_filter_get_params(cfg, &filter[i], ¶ms); @@ -1123,24 +1156,38 @@ routing_filter_estimate_unique_fp(cache *cc, return rc; } total_num_fp += filter[i].num_fingerprints; + if (filter[i].addr != 0) { + max_index_size = MAX(max_index_size, 1U << filter[i].log_index_size); + mixed_seeds |= have_seed && hash_seed != filter[i].hash_seed; + hash_seed = filter[i].hash_seed; + have_seed = TRUE; + } if (filter[i].addr != 0 && params.num_indices >= 16) { has_sampled_filter = TRUE; common_fingerprint_size = MIN(common_fingerprint_size, filter[i].fingerprint_size); } } + /* Hashes from different seeds cannot be compared for overlap. A conservative + * tuple count keeps leaf sizing safe until compaction rebuilds the filters. + */ + if (mixed_seeds) { + *fingerprint_size = 32; + *num_unique_fp = MIN(total_num_fp, UINT32_MAX); + return STATUS_OK; + } if (!has_sampled_filter) { return STATUS_OK; } *fingerprint_size = common_fingerprint_size; uint64 buffer_size64 = total_num_fp / 12; if (buffer_size64 > UINT32_MAX - || cfg->index_size > UINT32_MAX - buffer_size64) + || max_index_size > UINT32_MAX - buffer_size64) { return STATUS_LIMIT_EXCEEDED; } uint32 buffer_size = buffer_size64; - uint32 alloc_size = buffer_size + cfg->index_size; + uint32 alloc_size = buffer_size + max_index_size; uint32 *local = TYPED_ARRAY_ZALLOC(hid, local, alloc_size); if (local == NULL) { platform_error_log("routing_filter_estimate_unique_fp: failed to " @@ -1166,16 +1213,18 @@ routing_filter_estimate_unique_fp(cache *cc, if (!SUCCESS(rc)) { goto out; } - uint32 num_indices = params.num_indices; - uint32 index_size = cfg->index_size; - uint32 filter_fingerprint_size = filter[i].fingerprint_size; + routing_config read_cfg = routing_filter_read_config(cfg, &filter[i]); + const routing_config *filter_cfg = &read_cfg; + uint32 num_indices = params.num_indices; + uint32 index_size = filter_cfg->index_size; + uint32 filter_fingerprint_size = filter[i].fingerprint_size; if (num_indices >= 16) { // the filter is too small forget it platform_assert(num_indices % 16 == 0); num_indices /= 16; - rc = routing_filter_prefetch(cc, cfg, &filter[i], num_indices); + rc = routing_filter_prefetch(cc, filter_cfg, &filter[i], num_indices); if (!SUCCESS(rc)) { goto out; } @@ -1186,13 +1235,18 @@ routing_filter_estimate_unique_fp(cache *cc, uint16 index_count = 0; page_handle *filter_node; uint64 hdr_raw_addr; - rc = routing_get_header( - cc, cfg, filter[i].addr, index_no, &filter_node, &hdr_raw_addr); + rc = routing_get_header(cc, + filter_cfg, + filter[i].addr, + index_no, + &filter_node, + &hdr_raw_addr); if (!SUCCESS(rc)) { goto out; } routing_header_view view; - rc = routing_header_view_init(cfg, + rc = routing_header_view_init(filter_cfg, + filter_cfg->index_size, filter_node, hdr_raw_addr, params.remainder_and_value_size, @@ -1203,7 +1257,7 @@ routing_filter_estimate_unique_fp(cache *cc, } block_start = view.remainders; index_count = view.hdr->num_remainders; - rc = routing_get_bucket_counts(cfg, &view, count); + rc = routing_get_bucket_counts(filter_cfg, &view, count); if (!SUCCESS(rc)) { routing_unget_header(cc, filter_node); goto out; @@ -1379,8 +1433,8 @@ routing_filter_lookup_async(routing_filter_lookup_async_state *state) async_return(state, rc); } - state->fp = - data_key_hash(state->cfg->data_cfg, state->target, state->cfg->seed); + state->fp = data_key_hash( + state->cfg->data_cfg, state->target, state->filter.hash_seed); state->fp >>= 32 - state->filter.fingerprint_size; state->remainder_size = params.remainder_size; state->index = routing_get_index(state->fp << state->filter.value_size, @@ -1403,13 +1457,17 @@ routing_filter_lookup_async(routing_filter_lookup_async_state *state) } routing_header_view view; rc = routing_header_view_init(state->cfg, + 1U << state->filter.log_index_size, state->filter_page, state->hdr_raw_addr, params.remainder_and_value_size, &view); if (SUCCESS(rc)) { - rc = routing_filter_lookup_view( - state->cfg, ¶ms, state->fp, &view, state->found_values); + rc = routing_filter_lookup_view(1U << state->filter.log_index_size, + ¶ms, + state->fp, + &view, + state->found_values); } routing_unget_header(state->cc, state->filter_page); async_return(state, rc); @@ -1447,6 +1505,9 @@ routing_filter_lookup(cache *cc, if (filter == NULL || found_values == NULL) { return STATUS_BAD_PARAM; } + if (cfg == NULL) { + return STATUS_BAD_PARAM; + } if (filter->addr == 0) { *found_values = 0; routing_filter_params params; @@ -1459,7 +1520,7 @@ routing_filter_lookup(cache *cc, return rc; } - uint32 fp = data_key_hash(cfg->data_cfg, target, cfg->seed); + uint32 fp = data_key_hash(cfg->data_cfg, target, filter->hash_seed); fp >>= 32 - filter->fingerprint_size; uint32 index = routing_get_index(fp << params.value_size, params.index_remainder_and_value_size); @@ -1475,10 +1536,15 @@ routing_filter_lookup(cache *cc, return rc; } routing_header_view view; - rc = routing_header_view_init( - cfg, filter_node, hdr_raw_addr, params.remainder_and_value_size, &view); + rc = routing_header_view_init(cfg, + 1U << filter->log_index_size, + filter_node, + hdr_raw_addr, + params.remainder_and_value_size, + &view); if (SUCCESS(rc)) { - rc = routing_filter_lookup_view(cfg, ¶ms, fp, &view, found_values); + rc = routing_filter_lookup_view( + 1U << filter->log_index_size, ¶ms, fp, &view, found_values); } routing_unget_header(cc, filter_node); return rc; @@ -1686,7 +1752,7 @@ routing_filter_print_remainders(routing_config *cfg, size_t remainder_and_value_size = value_size + remainder_size; for (i = 0; i < cfg->index_size; i++) { platform_status rc = - routing_get_bucket_bounds(cfg, view, i, &start, &end); + routing_get_bucket_bounds(cfg->index_size, view, i, &start, &end); if (!SUCCESS(rc)) { platform_default_log("invalid routing encoding\n"); return; @@ -1694,8 +1760,7 @@ routing_filter_print_remainders(routing_config *cfg, platform_default_log("0x%lx remainders:", i); for (j = start; j < end; j++) { uint32 remainder, value, remainder_and_value; - routing_filter_get_remainder_and_value(cfg, - (uint32 *)view->remainders, + routing_filter_get_remainder_and_value((uint32 *)view->remainders, j, &remainder_and_value, remainder_and_value_size); @@ -1717,6 +1782,8 @@ routing_filter_print(cache *cc, routing_config *cfg, routing_filter *filter) platform_default_log("invalid routing filter descriptor\n"); return; } + routing_config read_cfg = routing_filter_read_config(cfg, filter); + cfg = &read_cfg; if (filter->addr == 0) { platform_default_log("empty routing filter\n"); return; @@ -1741,6 +1808,7 @@ routing_filter_print(cache *cc, routing_config *cfg, routing_filter *filter) } routing_header_view view; rc = routing_header_view_init(cfg, + cfg->index_size, filter_page, hdr_raw_addr, params.remainder_and_value_size, diff --git a/src/routing_filter.h b/src/routing_filter.h index 660fc4335..cac429fa7 100644 --- a/src/routing_filter.h +++ b/src/routing_filter.h @@ -24,7 +24,7 @@ */ #define MAX_FILTERS 32 #define ROUTING_NOT_FOUND (UINT16_MAX) -#define ROUTING_FILTER_FORMAT_VERSION (1) +#define ROUTING_FILTER_FORMAT_VERSION (2) /* @@ -84,15 +84,19 @@ typedef struct ONDISK routing_filter { uint32 value_size; uint16 format_version; uint16 fingerprint_size; + uint32 hash_seed; + uint16 log_index_size; } routing_filter; -_Static_assert(sizeof(routing_filter) == 32, +_Static_assert(sizeof(routing_filter) == 38, "routing filter descriptor layout changed"); #define NULL_ROUTING_FILTER ((routing_filter){0}) typedef struct ONDISK routing_hdr routing_hdr; +/* Extending a filter retains its encoding configuration. Incoming hashes must + * use old_filter->hash_seed (or cfg->seed for a new filter). */ platform_status routing_filter_add(cache *cc, const routing_config *cfg, @@ -140,11 +144,13 @@ routing_filters_equal(const routing_filter *f1, const routing_filter *f2) && f1->num_unique == f2->num_unique && f1->value_size == f2->value_size && f1->format_version == f2->format_version - && f1->fingerprint_size == f2->fingerprint_size; + && f1->fingerprint_size == f2->fingerprint_size + && f1->hash_seed == f2->hash_seed + && f1->log_index_size == f2->log_index_size; } /* Validate fields that are self-contained in the descriptor. The remaining - * encoding geometry depends on the database-wide index size and is checked by + * encoding geometry depends on the descriptor index size and is checked by * routing_filter_get_params() before the filter is accessed. */ static inline bool32 routing_filter_descriptor_is_valid(const routing_filter *filter) @@ -161,7 +167,9 @@ routing_filter_descriptor_is_valid(const routing_filter *filter) && filter->value_size <= 5 && filter->format_version == ROUTING_FILTER_FORMAT_VERSION && 0 < filter->fingerprint_size && filter->fingerprint_size <= 32 - && filter->fingerprint_size + filter->value_size <= 32; + && filter->fingerprint_size + filter->value_size <= 32 + && filter->log_index_size < 32 + && filter->log_index_size <= filter->fingerprint_size; } static inline uint64 @@ -170,7 +178,10 @@ routing_filter_max_fingerprints(cache_config *cache_cfg, { uint64 extent_size = cache_config_extent_size(cache_cfg); uint64 addrs_per_extent = extent_size / sizeof(uint64); - return 2ULL * addrs_per_extent * (1ULL << cfg->log_index_size) - 1; + uint64 index_limit = + 2ULL * addrs_per_extent * (1ULL << cfg->log_index_size) - 1; + uint64 fingerprint_limit = (1ULL << (cfg->fingerprint_size + 1)) - 1; + return MIN(UINT32_MAX, MIN(index_limit, fingerprint_limit)); } // clang-format off diff --git a/src/shard_log.c b/src/shard_log.c index 43981fdc3..1983b2a66 100644 --- a/src/shard_log.c +++ b/src/shard_log.c @@ -2053,9 +2053,11 @@ shard_log_get_head(log_handle *logh) { shard_log *log = (shard_log *)logh; return (log_head){ - .addr = log->addr, - .meta_addr = log->meta_head, - .nonce = log->nonce, + .addr = log->addr, + .meta_addr = log->meta_head, + .nonce = log->nonce, + .checksum_seed = log->cfg->seed, + .format_version = SHARD_LOG_FORMAT_VERSION, }; } @@ -2187,6 +2189,12 @@ shard_log_recover_allocations(cache *cc, shard_log_config *cfg, log_head head) if (head.addr == 0) { return STATUS_OK; // no such log } + if (head.format_version != SHARD_LOG_FORMAT_VERSION) { + return STATUS_INVALID_STATE; + } + shard_log_config read_cfg = *cfg; + read_cfg.seed = head.checksum_seed; + cfg = &read_cfg; /* * The metadata head sits in an extent of its own, allocated before the mini @@ -2663,7 +2671,7 @@ shard_log_iterator_init(cache *cc, shard_log_config *cfg, platform_heap_id hid, uint64 addr, - log_nonce nonce, + log_head head, uint64 first_needed_generation, shard_log_iterator *itor) { @@ -2683,7 +2691,11 @@ shard_log_iterator_init(cache *cc, itor->super.ops = &shard_log_log_iterator_ops; // log_iterator itor->heap_id = hid; itor->cc = cc; + itor->read_cfg = *cfg; + itor->read_cfg.seed = head.checksum_seed; + cfg = &itor->read_cfg; itor->cfg = cfg; + log_nonce nonce = head.nonce; allocator *al = cache_get_allocator(cc); /* @@ -3004,6 +3016,9 @@ shard_log_iterator_create(cache *cc, return STATUS_BAD_PARAM; } *itor_out = NULL; + if (head.format_version != SHARD_LOG_FORMAT_VERSION) { + return STATUS_INVALID_STATE; + } shard_log_iterator *itor = TYPED_MALLOC(hid, itor); if (itor == NULL) { @@ -3012,7 +3027,7 @@ shard_log_iterator_create(cache *cc, return STATUS_NO_MEMORY; } platform_status rc = shard_log_iterator_init( - cc, cfg, hid, head.addr, head.nonce, first_needed_generation, itor); + cc, cfg, hid, head.addr, head, first_needed_generation, itor); if (!SUCCESS(rc)) { platform_error_log("shard_log_iterator_create: shard_log_iterator_init " "failed: %s\n", diff --git a/src/shard_log.h b/src/shard_log.h index eb4b50fc1..b0a61e46b 100644 --- a/src/shard_log.h +++ b/src/shard_log.h @@ -238,6 +238,7 @@ typedef struct shard_log_iterator { platform_heap_id heap_id; cache *cc; shard_log_config *cfg; + shard_log_config read_cfg; // derived from the persisted log head char *contents; log_entry **entries; uint64 num_entries; diff --git a/src/superblock.c b/src/superblock.c index f6f102a18..da36f73a8 100644 --- a/src/superblock.c +++ b/src/superblock.c @@ -52,7 +52,8 @@ superblock_log_head_is_valid(superblock_log_head log, disk_geometry geometry) { if (SUPERBLOCK_NO_LOG(log)) { return log.head.addr == 0 && log.head.nonce.high == 0 - && log.head.nonce.low == 0 && log.start_generation == 0; + && log.head.nonce.low == 0 && log.start_generation == 0 + && log.head.format_version == 0 && log.head.checksum_seed == 0; } return superblock_extent_addr_is_valid(log.head.addr, geometry) @@ -95,7 +96,6 @@ superblock_has_valid_bootstrap(const superblock *sb) { disk_geometry geometry = sb->geometry; return sb->format_magic == SUPERBLOCK_FORMAT_MAGIC - && sb->format_version == SUPERBLOCK_FORMAT_VERSION && geometry.page_size == IO_DEFAULT_PAGE_SIZE && geometry.extent_size == IO_DEFAULT_EXTENT_SIZE && geometry.disk_size >= SUPERBLOCK_NUM_SLOTS * geometry.page_size @@ -111,7 +111,52 @@ superblock_is_valid(const superblock *sb, const allocator_config *cfg) disk_geometry geometry = sb->geometry; return superblock_has_valid_bootstrap(sb) && superblock_geometry_matches(geometry, cfg) - && superblock_metadata_is_valid(sb); + && (sb->format_version != SUPERBLOCK_FORMAT_VERSION + || superblock_metadata_is_valid(sb)); +} + +static platform_status +superblock_unsupported_version(uint64 version); + +/* Check supported readers only after choosing the newest valid record. An + * unknown component version must never roll the database back to an older + * checkpoint whose versions happen to be understood. */ +static platform_status +superblock_check_versions(const superblock *sb) +{ + /* An unknown version with a checksum valid under this layout is a + * recognizable record, not a torn write. Reject it after generation + * selection instead of silently returning an older checkpoint. */ + if (sb->format_version != SUPERBLOCK_FORMAT_VERSION) { + return superblock_unsupported_version(sb->format_version); + } + if (sb->trunk_format_version != TRUNK_FORMAT_VERSION + || sb->allocator_format_version != RC_ALLOCATOR_FORMAT_VERSION + || (!SUPERBLOCK_NO_LOG(sb->tree.live_log) + && sb->tree.live_log.head.format_version != SHARD_LOG_FORMAT_VERSION) + || (!SUPERBLOCK_NO_LOG(sb->tree.sealed_log) + && sb->tree.sealed_log.head.format_version + != SHARD_LOG_FORMAT_VERSION)) + { + platform_error_log("Unsupported on-disk structure version: " + "trunk=%u allocator=%u live-log=%u sealed-log=%u\n", + sb->trunk_format_version, + sb->allocator_format_version, + sb->tree.live_log.head.format_version, + sb->tree.sealed_log.head.format_version); + return STATUS_INVALID_STATE; + } + return STATUS_OK; +} + +static platform_status +superblock_unsupported_version(uint64 version) +{ + platform_error_log("Unsupported superblock format version %lu; " + "this build requires version %u.\n", + version, + SUPERBLOCK_FORMAT_VERSION); + return STATUS_INVALID_STATE; } /* @@ -134,8 +179,10 @@ superblock_read_bootstrap(const char *filename, superblock *image) return STATUS_BAD_PARAM; } - bool32 have_valid = FALSE; - superblock newest = {0}; + bool32 have_valid = FALSE; + superblock newest = {0}; + bool32 have_unsupported = FALSE; + uint64 unsupported_version = 0; for (uint64 slot = 0; slot < SUPERBLOCK_NUM_SLOTS; slot++) { superblock candidate; platform_status rc = io_read_bootstrap( @@ -143,6 +190,12 @@ superblock_read_bootstrap(const char *filename, superblock *image) if (!SUCCESS(rc)) { continue; } + if (candidate.format_magic == SUPERBLOCK_FORMAT_MAGIC + && candidate.format_version != SUPERBLOCK_FORMAT_VERSION) + { + have_unsupported = TRUE; + unsupported_version = candidate.format_version; + } if (superblock_has_valid_bootstrap(&candidate) && (!have_valid || superblock_generation_is_newer(candidate.generation, @@ -153,12 +206,18 @@ superblock_read_bootstrap(const char *filename, superblock *image) } } if (!have_valid) { + if (have_unsupported) { + return superblock_unsupported_version(unsupported_version); + } platform_error_log( "superblock_read_bootstrap: no valid superblock found\n"); return STATUS_NOT_FOUND; } - *image = newest; - return STATUS_OK; + platform_status rc = superblock_check_versions(&newest); + if (SUCCESS(rc)) { + *image = newest; + } + return rc; } platform_status @@ -220,8 +279,10 @@ superblock_mount(superblock_context *ctx, const allocator_config *cfg) } superblock *slot = platform_buffer_getaddr(&slot_buffer); - bool32 have_valid = FALSE; - platform_status first_read_error = STATUS_OK; + bool32 have_valid = FALSE; + platform_status first_read_error = STATUS_OK; + bool32 have_unsupported = FALSE; + uint64 unsupported_version = 0; for (uint64 s = 0; s < SUPERBLOCK_NUM_SLOTS; s++) { rc = io_read(ctx->io, slot, ctx->page_size, superblock_slot_addr(ctx, s)); if (!SUCCESS(rc)) { @@ -230,6 +291,12 @@ superblock_mount(superblock_context *ctx, const allocator_config *cfg) } continue; } + if (slot->format_magic == SUPERBLOCK_FORMAT_MAGIC + && slot->format_version != SUPERBLOCK_FORMAT_VERSION) + { + have_unsupported = TRUE; + unsupported_version = slot->format_version; + } if (!superblock_is_valid(slot, cfg)) { continue; } @@ -244,7 +311,9 @@ superblock_mount(superblock_context *ctx, const allocator_config *cfg) } if (!have_valid) { - if (!SUCCESS(first_read_error)) { + if (have_unsupported) { + rc = superblock_unsupported_version(unsupported_version); + } else if (!SUCCESS(first_read_error)) { platform_error_log("superblock_mount: no readable valid superblock " "found: %s\n", platform_status_to_string(first_read_error)); @@ -255,7 +324,7 @@ superblock_mount(superblock_context *ctx, const allocator_config *cfg) } goto out; } - rc = STATUS_OK; + rc = superblock_check_versions(ctx->image); out: { @@ -305,13 +374,15 @@ superblock_format(superblock_context *ctx, const allocator_config *cfg) platform_assert(ctx->image != NULL); memset(ctx->image, 0, ctx->page_size); - ctx->image->geometry.disk_size = cfg->capacity; - ctx->image->geometry.page_size = cfg->io_cfg->page_size; - ctx->image->geometry.extent_size = cfg->io_cfg->extent_size; - ctx->image->format_magic = SUPERBLOCK_FORMAT_MAGIC; - ctx->image->format_version = SUPERBLOCK_FORMAT_VERSION; - ctx->image->generation = 0; - ctx->image->allocation_state_addr = 0; // fresh DB: rebuild on crash + ctx->image->geometry.disk_size = cfg->capacity; + ctx->image->geometry.page_size = cfg->io_cfg->page_size; + ctx->image->geometry.extent_size = cfg->io_cfg->extent_size; + ctx->image->format_magic = SUPERBLOCK_FORMAT_MAGIC; + ctx->image->format_version = SUPERBLOCK_FORMAT_VERSION; + ctx->image->trunk_format_version = TRUNK_FORMAT_VERSION; + ctx->image->allocator_format_version = RC_ALLOCATOR_FORMAT_VERSION; + ctx->image->generation = 0; + ctx->image->allocation_state_addr = 0; // fresh DB: rebuild on crash // A fresh, empty tree: no root, and nothing incorporated yet -- replay from // generation 0. ctx->image->tree.first_unincorporated_generation = 0; diff --git a/src/superblock.h b/src/superblock.h index a5bee05f5..3343e91b9 100644 --- a/src/superblock.h +++ b/src/superblock.h @@ -45,9 +45,12 @@ * moved the checksum and uint16 format into the fixed blob header; v5 made * log group ids start at 1 so they equal their durability tickets; v6 added * per-level page ordinals to branch btree pages; v7 made routing-filter - * descriptors carry their format version and fingerprint size. + * descriptors carry their format version and fingerprint size; v8 adds + * independent structure versions and complete routing-filter/log read config. */ -#define SUPERBLOCK_FORMAT_VERSION (7) +#define SUPERBLOCK_FORMAT_VERSION (8) +#define TRUNK_FORMAT_VERSION (1) +#define RC_ALLOCATOR_FORMAT_VERSION (1) /* The two physical superblock copies live at pages 0 and 1. */ #define SUPERBLOCK_NUM_SLOTS (2) @@ -123,6 +126,8 @@ typedef struct ONDISK superblock { * checkpoint writes this as 0 in the same atomic update that advances a * root; a clean unmount writes it nonzero after the map is durable. */ + uint16 trunk_format_version; + uint16 allocator_format_version; uint64 allocation_state_addr; superblock_tree_record tree; checksum128 checksum; @@ -181,6 +186,8 @@ superblock_context_deinit(superblock_context *ctx); * image. Returns STATUS_NOT_FOUND if every readable copy is invalid, or the * first read error if no valid copy can be read. * + * Unsupported structure versions return STATUS_INVALID_STATE with a diagnostic. + * * Must call superblock_init() first. */ platform_status diff --git a/src/trunk.c b/src/trunk.c index f7bd457ab..e4c8b56e3 100644 --- a/src/trunk.c +++ b/src/trunk.c @@ -155,6 +155,7 @@ struct trunk_pivot_state { key_buffer key; key_buffer ubkey; uint64 height; + uint32 hash_seed; // immutable seed of this pivot maplet uint64 refcount; @@ -215,7 +216,9 @@ struct pending_gc { static branch_ref create_branch_ref(uint64 addr) { - return (branch_ref){.addr = addr}; + return (branch_ref){.addr = addr, + .format_version = + addr == 0 ? 0 : BTREE_BRANCH_FORMAT_VERSION}; } static uint64 @@ -1525,6 +1528,13 @@ trunk_ondisk_bundle_readable(trunk_ondisk_bundle *bundle, { return FALSE; } + for (uint64 i = 0; i < bundle->num_branches; i++) { + if (bundle->branches[i].addr == 0 + || bundle->branches[i].format_version != BTREE_BRANCH_FORMAT_VERSION) + { + return FALSE; + } + } if (!routing_filter_descriptor_is_valid(&bundle->maplet)) { return FALSE; } @@ -3875,6 +3885,9 @@ trunk_pivot_state_init_protected(trunk_pivot_state *state, trunk_pivot_state_lock(state); state->protected.maplet_compaction_rc = STATUS_OK; state->protected.maplet = pivot_bundle->maplet; + state->hash_seed = pivot_bundle->maplet.addr == 0 + ? context->cfg->filter_cfg->seed + : pivot_bundle->maplet.hash_seed; routing_filter_inc_ref(context->cc, &state->protected.maplet); state->protected.num_branches = bundle_num_branches(pivot_bundle); trunk_pivot_state_unlock(state); @@ -4813,7 +4826,7 @@ bundle_compaction_task(task *arg) context->cfg->btree_cfg, &merger.merge_itor->super, bc->input.stats.num_tuples, - context->cfg->filter_cfg->seed, + state->hash_seed, TRUE, context->hid); if (!SUCCESS(rc)) { @@ -5357,6 +5370,27 @@ leaf_estimate_unique_keys(trunk_context *context, return rc; } +/* An existing pivot maplet keeps its encoding while branches are appended. + * Respect that instance's capacity when new creation defaults are larger. */ +static uint64 +trunk_node_filter_limit(const trunk_context *context, const trunk_node *node) +{ + cache_config *cache_cfg = cache_get_config(context->cc); + uint64 limit = + routing_filter_max_fingerprints(cache_cfg, context->cfg->filter_cfg); + for (uint64 i = 0; i < vector_length(&node->pivot_bundles); i++) { + const routing_filter *filter = + &vector_get_ptr(&node->pivot_bundles, i)->maplet; + if (filter->addr != 0) { + routing_config cfg = *context->cfg->filter_cfg; + cfg.fingerprint_size = filter->fingerprint_size; + cfg.log_index_size = filter->log_index_size; + limit = MIN(limit, routing_filter_max_fingerprints(cache_cfg, &cfg)); + } + } + return limit; +} + static platform_status leaf_split_target_num_leaves(trunk_context *context, trunk_node *leaf, @@ -5364,8 +5398,7 @@ leaf_split_target_num_leaves(trunk_context *context, { debug_assert(trunk_node_is_well_formed_leaf(context->cfg->data_cfg, leaf)); - uint64 rflimit = routing_filter_max_fingerprints( - cache_get_config(context->cc), context->cfg->filter_cfg); + uint64 rflimit = trunk_node_filter_limit(context, leaf); if (!leaf_might_need_to_split(context->cfg, rflimit, leaf)) { *target = 1; @@ -5478,8 +5511,7 @@ leaf_split_select_pivots(trunk_context *context, goto cleanup; } - uint64 rflimit = routing_filter_max_fingerprints( - cache_get_config(context->cc), context->cfg->filter_cfg); + uint64 rflimit = trunk_node_filter_limit(context, leaf); uint64 leaf_num = 1; uint64 cumulative_kv_bytes = 0; uint64 current_tuples = 0; @@ -6255,8 +6287,7 @@ restore_balance_index(trunk_context *context, { platform_status rc; threadid tid = platform_get_tid(); - uint64 rflimit = routing_filter_max_fingerprints( - cache_get_config(context->cc), context->cfg->filter_cfg); + uint64 rflimit = trunk_node_filter_limit(context, index); policy = trunk_flush_policy_or_default(policy); diff --git a/src/trunk.h b/src/trunk.h index 95ac67b0f..19abf286f 100644 --- a/src/trunk.h +++ b/src/trunk.h @@ -124,6 +124,7 @@ typedef struct trunk_ondisk_node_ref { typedef struct ONDISK branch_ref { uint64 addr; + uint16 format_version; } branch_ref; typedef VECTOR(branch_ref) branch_ref_vector; diff --git a/tests/functional/filter_test.c b/tests/functional/filter_test.c index 83423c1e9..beece4f16 100644 --- a/tests/functional/filter_test.c +++ b/tests/functional/filter_test.c @@ -30,6 +30,7 @@ typedef struct ONDISK test_mini_meta_hdr { uint64 meta_page_no; uint64 pos; uint32 num_entries; + uint16 format_version; char entry_buffer[]; } test_mini_meta_hdr; @@ -222,6 +223,18 @@ test_filter_malicious_metadata(cache *cc, goto out; } + invalid_filter = filter; + invalid_filter.format_version = ROUTING_FILTER_FORMAT_VERSION + 1; + if (!test_filter_lookup_rejects(cc, cfg, &invalid_filter, target)) { + rc = STATUS_TEST_FAILED; + goto out; + } + invalid_filter = filter; + invalid_filter.log_index_size = 32; + if (!test_filter_lookup_rejects(cc, cfg, &invalid_filter, target)) { + rc = STATUS_TEST_FAILED; + goto out; + } invalid_filter = filter; invalid_filter.fingerprint_size = 0; if (!test_filter_lookup_rejects(cc, cfg, &invalid_filter, target)) { @@ -325,6 +338,21 @@ test_filter_malicious_metadata(cache *cc, rc = STATUS_TEST_FAILED; goto out; } + meta_hdr->format_version = MINI_ALLOCATOR_FORMAT_VERSION + 1; + test_filter_unclaim_page(cc, meta_page); + mini_meta_cursor version_cursor; + if (mini_meta_cursor_init( + &version_cursor, cc, PAGE_TYPE_FILTER, filter.meta_head, filter.addr) + != MINI_META_CURSOR_INVALID) + { + mini_meta_cursor_deinit(&version_cursor); + rc = STATUS_TEST_FAILED; + goto out; + } + mini_meta_cursor_deinit(&version_cursor); + meta_page = test_filter_claim_page(cc, filter.meta_head, PAGE_TYPE_FILTER); + meta_hdr = (test_mini_meta_hdr *)meta_page->data; + meta_hdr->format_version = MINI_ALLOCATOR_FORMAT_VERSION; test_mini_meta_entry *entry = (test_mini_meta_entry *)meta_hdr->entry_buffer; uint64 target_extent = (entry->packed >> 16) * cache_extent_size(cc); if (meta_hdr->meta_page_no != 0) { @@ -386,6 +414,7 @@ test_filter_malicious_metadata(cache *cc, memset(second_meta_page->data, 0, page_size); test_mini_meta_hdr *second_meta_hdr = (test_mini_meta_hdr *)second_meta_page->data; + second_meta_hdr->format_version = MINI_ALLOCATOR_FORMAT_VERSION; second_meta_hdr->prev_meta_addr = filter.meta_head; second_meta_hdr->meta_page_no = 1; second_meta_hdr->num_entries = 1; @@ -610,6 +639,9 @@ test_filter_mixed_fingerprint_sizes(cache *cc, routing_config narrow_cfg = *cfg; narrow_cfg.fingerprint_size--; + narrow_cfg.log_index_size--; + narrow_cfg.index_size >>= 1; + narrow_cfg.seed ^= 0x12345678; routing_filter null_filter = NULL_ROUTING_FILTER; routing_filter narrow_filter = NULL_ROUTING_FILTER; @@ -623,7 +655,7 @@ test_filter_mixed_fingerprint_sizes(cache *cc, uint64 key_data = 1; key target = key_create(TRUE, sizeof(key_data), &key_data); - uint32 fp = data_key_hash(cfg->data_cfg, target, cfg->seed); + uint32 fp = data_key_hash(cfg->data_cfg, target, narrow_cfg.seed); rc = routing_filter_add( cc, &narrow_cfg, &null_filter, &narrow_filter, &fp, 1, 0); if (!SUCCESS(rc)) { @@ -660,7 +692,7 @@ test_filter_mixed_fingerprint_sizes(cache *cc, key_data = 2; target = key_create(TRUE, sizeof(key_data), &key_data); - fp = data_key_hash(cfg->data_cfg, target, cfg->seed); + fp = data_key_hash(cfg->data_cfg, target, narrow_cfg.seed); stage = "extend narrow filter"; rc = routing_filter_add(cc, cfg, &narrow_filter, &extended_filter, &fp, 1, 1); @@ -681,6 +713,7 @@ test_filter_mixed_fingerprint_sizes(cache *cc, /* The cross-filter estimator samples one sixteenth of each filter. Build * two filters containing the same evenly distributed hashes at different * widths; after normalization they must still count as one set. */ + narrow_cfg.seed = cfg->seed; uint32 log_num_fingerprints = cfg->log_index_size + 4; uint32 num_fingerprints = 1U << log_num_fingerprints; uint32 stride_shift = 32 - log_num_fingerprints; diff --git a/tests/functional/log_test.c b/tests/functional/log_test.c index d54c242f5..e82b3aa5c 100644 --- a/tests/functional/log_test.c +++ b/tests/functional/log_test.c @@ -1635,8 +1635,17 @@ test_log_large_message(cache *cc, shard_log_config *cfg, platform_heap_id hid) rc = cache_durable_barrier(cc); platform_assert_status_ok(rc); + log_head unknown = sealed; + unknown.format_version++; + platform_assert( + !SUCCESS(shard_log_iterator_create(cc, cfg, hid, unknown, 0, &itor))); + platform_assert(itor == NULL); + platform_assert(!SUCCESS(shard_log_recover_allocations(cc, cfg, unknown))); + + shard_log_config read_cfg = *cfg; + read_cfg.seed ^= UINT64_C(0x123456789abcdef0); platform_assert_status_ok( - shard_log_iterator_create(cc, cfg, hid, sealed, 0, &itor)); + shard_log_iterator_create(cc, &read_cfg, hid, sealed, 0, &itor)); platform_assert(log_iterator_stream_complete(itor)); platform_assert(log_iterator_can_next(itor)); diff --git a/tests/unit/splinter_test.c b/tests/unit/splinter_test.c index b92dafd9d..85272c994 100644 --- a/tests/unit/splinter_test.c +++ b/tests/unit/splinter_test.c @@ -1494,6 +1494,81 @@ CTEST2(splinter, test_malformed_trunk_pivot_bundle) ASSERT_TRUE(range_rejected); } +CTEST2(splinter, test_unknown_branch_version_rejected) +{ + cache *cc = (cache *)data->clock_cache; + core_handle spl; + ASSERT_TRUE(SUCCESS(core_mkfs(&spl, + &data->system_cfg->splinter_cfg, + (allocator *)&data->al, + cc, + data->io, + &data->tasks, + test_generate_allocator_root_id(), + data->hid))); + DECLARE_AUTO_KEY_BUFFER(keybuf, data->hid); + test_key(&keybuf, TEST_RANDOM, 1, 0, 0, data->workload_cfg->key_size, 0); + merge_accumulator msg; + merge_accumulator_init(&msg, data->hid); + generate_test_message(&data->gen, 1, &msg); + ASSERT_TRUE(SUCCESS(core_insert(&spl, + key_buffer_key(&keybuf), + merge_accumulator_to_message(&msg), + NULL))); + merge_accumulator_deinit(&msg); + ASSERT_TRUE(SUCCESS(core_checkpoint(&spl, 0))); + + uint64 root_addr = spl.trunk_context.root->ref.addr; + uint64 page_size = cache_page_size(cc); + page_handle *root_page = + test_splinter_claim_page(cc, root_addr, PAGE_TYPE_TRUNK); + test_trunk_node_hdr *node = (test_trunk_node_hdr *)root_page->data; + /* The single freshly incorporated branch is still an inflight bundle. */ + ASSERT_TRUE(node->num_inflight_bundles != 0); + uint64 offset = node->inflight_bundles_offset; + test_splinter_unclaim_page(cc, root_page); + uint64 page_addr = root_addr + offset - offset % page_size; + page_handle *page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); + test_trunk_bundle *bundle = + (test_trunk_bundle *)(page->data + offset % page_size); + ASSERT_TRUE(bundle->num_branches != 0); + uint16 saved_version = bundle->branches[0].format_version; + bundle->branches[0].format_version++; + test_splinter_unclaim_page(cc, page); + + lookup_result result; + lookup_result_init( + &result, spl.cfg.data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); + platform_status sync_rc = + core_lookup(&spl, key_buffer_key(&keybuf), &result); + platform_status async_rc = async_call_sync_callback(cache_cleanup(cc), + core_lookup_async, + &spl, + key_buffer_key(&keybuf), + &result); + lookup_result_deinit(&result); + core_range_iterator itor; + platform_status range_rc = core_range_iterator_init(&spl, + &itor, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, + less_than, + POSITIVE_INFINITY_KEY, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY); + if (SUCCESS(range_rc)) { + core_range_iterator_deinit(&itor); + } + page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); + bundle = (test_trunk_bundle *)(page->data + offset % page_size); + bundle->branches[0].format_version = saved_version; + test_splinter_unclaim_page(cc, page); + core_destroy(&spl); + ASSERT_FALSE(SUCCESS(sync_rc)); + ASSERT_FALSE(SUCCESS(async_rc)); + ASSERT_FALSE(SUCCESS(range_rc)); +} + /* Persisted pivot statistics are sizing hints, but they still feed arithmetic * during writes. Reject the impossible zero-tuple/nonzero-byte state before * it can reach leaf splitting (where it used to cause a division by zero). */ @@ -1801,6 +1876,101 @@ CTEST2(splinter, test_two_log_checkpoint) core_destroy(&spl); } +/* Old filters must survive changed creation defaults, including later + * compactions that append branches to their existing maplets. */ +CTEST2(splinter, test_reopen_with_changed_filter_config) +{ + allocator *al = (allocator *)&data->al; + cache *cc = (cache *)data->clock_cache; + system_config *cfg = data->system_cfg; + allocator_root_id root_id = test_generate_allocator_root_id(); + cfg->trunk_node_cfg.incorporation_size_kv_bytes = 64 * KiB; + cfg->trunk_node_cfg.target_fanout = 4; + core_handle spl; + ASSERT_TRUE(SUCCESS(core_mkfs(&spl, + &cfg->splinter_cfg, + al, + cc, + data->io, + &data->tasks, + root_id, + data->hid))); + DECLARE_AUTO_KEY_BUFFER(keybuf, data->hid); + merge_accumulator msg; + merge_accumulator_init(&msg, data->hid); + const uint64 per_round = 2048; + const uint64 rounds = 8; + for (uint64 round = 0; round < rounds; round++) { + if (round == rounds / 2) { + ASSERT_TRUE(SUCCESS(core_unmount(&spl, FALSE))); + cfg->filter_cfg.seed ^= 0x87654321; + cfg->filter_cfg.log_index_size--; + cfg->filter_cfg.index_size >>= 1; + cfg->filter_cfg.fingerprint_size--; + ASSERT_TRUE(SUCCESS(core_mount(&spl, + &cfg->splinter_cfg, + al, + cc, + data->io, + &data->tasks, + root_id, + data->hid))); + } + for (uint64 i = round * per_round; i < (round + 1) * per_round; i++) { + test_key( + &keybuf, TEST_RANDOM, i, 0, 0, data->workload_cfg->key_size, 0); + generate_test_message(&data->gen, i, &msg); + ASSERT_TRUE(SUCCESS(core_insert(&spl, + key_buffer_key(&keybuf), + merge_accumulator_to_message(&msg), + NULL))); + } + ASSERT_TRUE(SUCCESS(core_checkpoint(&spl, 0))); + } + merge_accumulator_deinit(&msg); + ASSERT_TRUE(SUCCESS(core_unmount(&spl, FALSE))); + ASSERT_TRUE(SUCCESS(core_mount(&spl, + &cfg->splinter_cfg, + al, + cc, + data->io, + &data->tasks, + root_id, + data->hid))); + lookup_result result; + lookup_result_init( + &result, spl.cfg.data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); + for (uint64 i = 0; i < per_round * rounds; i++) { + test_key(&keybuf, TEST_RANDOM, i, 0, 0, data->workload_cfg->key_size, 0); + ASSERT_TRUE(SUCCESS(core_lookup(&spl, key_buffer_key(&keybuf), &result))); + verify_tuple( + &spl, + &data->gen, + i, + key_buffer_key(&keybuf), + merge_accumulator_to_message(lookup_result_accumulator(&result)), + TRUE); + } + lookup_result_deinit(&result); + core_range_iterator itor; + ASSERT_TRUE(SUCCESS(core_range_iterator_init(&spl, + &itor, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, + less_than, + POSITIVE_INFINITY_KEY, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY))); + uint64 count = 0; + while (iterator_can_next(&itor.super)) { + count++; + ASSERT_TRUE(SUCCESS(iterator_next(&itor.super))); + } + core_range_iterator_deinit(&itor); + ASSERT_EQUAL(per_round * rounds, count); + core_destroy(&spl); +} + /* * Preparing the log record blobifies a large inline value before the B-tree * sees it. The B-tree must attach the complete extents from that blob rather diff --git a/tests/unit/superblock_test.c b/tests/unit/superblock_test.c index f31e215dc..6649f87be 100644 --- a/tests/unit/superblock_test.c +++ b/tests/unit/superblock_test.c @@ -183,9 +183,11 @@ CTEST2(superblock, test_snapshot_persists_state) rc = superblock_format(&ctx, &data->allocator_cfg); ASSERT_TRUE(SUCCESS(rc)); - superblock_log_head live = {.head = {.addr = TEST_EXTENT_ADDR(6), - .meta_addr = TEST_EXTENT_ADDR(8), - .nonce = {.low = 0x11}}}; + superblock_log_head live = { + .head = {.addr = TEST_EXTENT_ADDR(6), + .meta_addr = TEST_EXTENT_ADDR(8), + .format_version = SHARD_LOG_FORMAT_VERSION, + .nonce = {.low = 0x11}}}; superblock_log_cut(&ctx, live); superblock_snapshot_tree(&ctx, TEST_EXTENT_ADDR(4), 0); rc = superblock_make_durable(&ctx); @@ -221,14 +223,16 @@ CTEST2(superblock, test_snapshot_persists_state) * covers generations 0..5 and is cut at 5, so L2 takes over at 6. */ static const superblock_log_head TEST_LOG_L1 = { - .head = {.addr = TEST_EXTENT_ADDR(6), - .meta_addr = TEST_EXTENT_ADDR(8), - .nonce = {.low = 0x11}}, + .head = {.addr = TEST_EXTENT_ADDR(6), + .meta_addr = TEST_EXTENT_ADDR(8), + .format_version = SHARD_LOG_FORMAT_VERSION, + .nonce = {.low = 0x11}}, .start_generation = 0}; static const superblock_log_head TEST_LOG_L2 = { - .head = {.addr = TEST_EXTENT_ADDR(10), - .meta_addr = TEST_EXTENT_ADDR(12), - .nonce = {.low = 0x22}}, + .head = {.addr = TEST_EXTENT_ADDR(10), + .meta_addr = TEST_EXTENT_ADDR(12), + .format_version = SHARD_LOG_FORMAT_VERSION, + .nonce = {.low = 0x22}}, .start_generation = 6}; /* @@ -519,9 +523,10 @@ CTEST2(superblock, test_mount_rejects_invalid_metadata) ASSERT_TRUE(SUCCESS(rc)); superblock_log_head valid_log = { - .head = {.addr = TEST_EXTENT_ADDR(6), - .meta_addr = TEST_EXTENT_ADDR(8), - .nonce = {.low = 1}}}; + .head = {.addr = TEST_EXTENT_ADDR(6), + .meta_addr = TEST_EXTENT_ADDR(8), + .format_version = SHARD_LOG_FORMAT_VERSION, + .nonce = {.low = 1}}}; switch (test_case) { case INVALID_MAP_ADDRESS: ctx.image->allocation_state_addr = TEST_EXTENT_ADDR(2); @@ -706,3 +711,62 @@ CTEST2(superblock, test_bootstrap_geometry_metadata_budget) ASSERT_NOT_EQUAL(0, splinterdb_open(&cfg, &kvs)); ASSERT_TRUE(kvs == NULL); } + +/* A checksummed newer version is not permission to roll back to its peer. */ +CTEST2(superblock, test_unknown_structure_version_rejected) +{ + for (uint64 component = 0; component < 3; component++) { + superblock_context ctx; + platform_status rc = superblock_context_init( + &ctx, data->ioh, &data->allocator_cfg, data->hid); + ASSERT_TRUE(SUCCESS(rc)); + ASSERT_TRUE(SUCCESS(superblock_format(&ctx, &data->allocator_cfg))); + if (component == 0) { + ctx.image->trunk_format_version++; + } else if (component == 1) { + ctx.image->allocator_format_version++; + } else { + superblock_log_cut(&ctx, TEST_LOG_L1); + ctx.image->tree.live_log.head.format_version++; + } + ASSERT_TRUE(SUCCESS(superblock_make_durable(&ctx))); + superblock image; + ASSERT_TRUE( + STATUS_IS_EQ(superblock_read_bootstrap(data->io_cfg.filename, &image), + STATUS_INVALID_STATE)); + ASSERT_TRUE(STATUS_IS_EQ(superblock_mount(&ctx, &data->allocator_cfg), + STATUS_INVALID_STATE)); + superblock_context_deinit(&ctx); + } +} + +CTEST2(superblock, test_unsupported_superblock_version_rejected) +{ + uint64 versions[] = {SUPERBLOCK_FORMAT_VERSION - 1, + SUPERBLOCK_FORMAT_VERSION + 1}; + for (uint64 i = 0; i < ARRAY_SIZE(versions); i++) { + superblock_context ctx; + ASSERT_TRUE(SUCCESS(superblock_context_init( + &ctx, data->ioh, &data->allocator_cfg, data->hid))); + ASSERT_TRUE(SUCCESS(superblock_format(&ctx, &data->allocator_cfg))); + ctx.image->format_version = versions[i]; + ASSERT_TRUE(SUCCESS(superblock_make_durable(&ctx))); + superblock image; + /* A valid older peer must not hide an unsupported newest record. */ + ASSERT_TRUE( + STATUS_IS_EQ(superblock_read_bootstrap(data->io_cfg.filename, &image), + STATUS_INVALID_STATE)); + ASSERT_TRUE(STATUS_IS_EQ(superblock_mount(&ctx, &data->allocator_cfg), + STATUS_INVALID_STATE)); + ASSERT_TRUE(SUCCESS(superblock_make_durable(&ctx))); + ASSERT_TRUE( + STATUS_IS_EQ(superblock_read_bootstrap(data->io_cfg.filename, &image), + STATUS_INVALID_STATE)); + /* Reformat replaces both slots, including the unsupported old peer. */ + ASSERT_TRUE(SUCCESS(superblock_format(&ctx, &data->allocator_cfg))); + ASSERT_TRUE( + SUCCESS(superblock_read_bootstrap(data->io_cfg.filename, &image))); + ASSERT_TRUE(SUCCESS(superblock_mount(&ctx, &data->allocator_cfg))); + superblock_context_deinit(&ctx); + } +} From 3721ae5311f06f0b5fb1889796437de30c055c13 Mon Sep 17 00:00:00 2001 From: Rob Johnson Date: Sun, 11 Oct 2026 22:18:50 -0700 Subject: [PATCH 2/7] Document metadata benchmarks and remaining point-read regression --- docs/on-disk-format.md | 55 ++++++++++++++++++++++++++++++------------ 1 file changed, 39 insertions(+), 16 deletions(-) diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md index 895b98c55..4abca1573 100644 --- a/docs/on-disk-format.md +++ b/docs/on-disk-format.md @@ -54,22 +54,45 @@ unknown versions, changed routing widths/seeds across reopen and further compaction, synchronous/asynchronous filter reads, and changed log checksum creation defaults. -Initial resident ABBA measurements found a 2.8–4.0% point-read regression. -After removing redundant validation, a second campaign used a single loader, -200,000 keys in a 128 MiB cache and 7-second read phases. Point throughput was -still 3.4–3.6% lower; range results ranged from -2.5% to +2.6%. The 2-million-key, -64 MiB-cache campaign reported +2.0–2.4% point throughput and +2.7–6.6% range -throughput, with comparable I/O per operation. Device/run variability and -independently constructed trees prevent attributing those gains to metadata. -All samples had zero correctness violations and errors. Brief load samples do -not establish write-performance neutrality. - -A subsequent reader change removes a full routing-config copy from each filter -lookup, passing the saved index width and hash seed directly. Version and bounds -validation remains at the same boundaries. The full functional filter suite and -changed-configuration reopen/compaction regression pass after that change. -Its profiling and fixed-image old/new reader measurements are pending; this -implementation milestone does not claim performance acceptance. +The benchmark used identical generated kvbench source, O_DIRECT files on ext4, +CPU affinity 0–11, 24-byte keys, 100-byte values and WAL disabled. A single +loader constructed each image; one memtable and three normal workers were +enabled, so incompatible baseline/candidate formats still have a concurrent +tree-shape caveat. Resident: 200,000 keys / 128 MiB cache. Disk regime: 2 million +keys / 64 MiB cache. ABBA read phases lasted seven seconds. + +| Pre-scalar-reader campaign | Point, 1 reader | Point, 8 readers | Range throughput change | +| --- | ---: | ---: | ---: | +| Resident | -3.4% | -3.6% | -2.5% to +2.6% | +| Disk | +2.0% | +2.4% | +2.7% to +6.6% | + +The disk samples had comparable I/O per operation. Device/run variation and +independently constructed trees prevent attributing their apparent gains to +metadata. Initial shorter resident samples also showed a 2.8–4.0% point-read +regression. All runs had zero correctness violations and errors; brief load +samples do not establish write-performance neutrality. + +The final reader removes a full routing-config copy from each filter lookup, +passing the saved index width and hash seed directly. Version and bounds +validation stays at the same boundaries. A six-run, ten-second fixed-image +comparison used baseline / old reader / final reader / final reader / old +reader / baseline order. Both candidate readers used copies of exactly the same +format-8 image; baseline used its own format-7 image. + +| Final reader, resident | Versus old reader | Versus baseline | +| --- | ---: | ---: | +| Point, 1 reader | +1.2% | -1.5% | +| Point, 8 readers | +2.0% | -6.5% | +| Length-1 range, 1 reader | -2.2% | -1.5% | +| Length-1 range, 8 readers | -1.7% | -1.0% | + +The final functional filter suite and changed-configuration reopen/compaction +regression pass. Profiling the earlier reader places most samples in cache +acquisition/release and B-tree/trunk lookup. It does not isolate a single cause +for the residual point-read cost. Each bundle decode now checks its branch +versions; that added work remains a possible cost. Performance neutrality has +**not** been established. The final scalar reader has not repeated the full +disk or write matrix; the disk figures above describe the earlier reader. Campaign artifacts: `.cache/projects-20261011/benchmarks/disk-format/` in the main checkout. The common harness and build provenance are retained there; From f6c76a996ce5048806fa39da12bfb2745750df5d Mon Sep 17 00:00:00 2001 From: Rob Johnson Date: Sun, 11 Oct 2026 22:20:51 -0700 Subject: [PATCH 3/7] Retain metadata benchmark samples and provenance --- benchmarks/results-20261011/README.md | 13 + .../results-20261011/baseline-build.json | 30 ++ benchmarks/results-20261011/build_kvbench.py | 27 ++ benchmarks/results-20261011/compare.py | 72 ++++ .../disk-format-pre-scalar.json | 5 + .../disk-log0-final-singleload/manifest.json | 283 ++++++++++++++ .../disk-log0-final-singleload/results.csv | 57 +++ .../results-20261011/final-reader-build.json | 30 ++ benchmarks/results-20261011/format_profile.py | 32 ++ .../results-20261011/format_scalar_compare.py | 32 ++ .../manifest.json | 283 ++++++++++++++ .../results.csv | 57 +++ .../resident-log0/manifest.json | 283 ++++++++++++++ .../resident-log0/results.csv | 57 +++ .../resident-profile/baseline-t1.txt | 61 +++ .../resident-profile/baseline-t8.txt | 58 +++ .../resident-profile/candidate-t1.txt | 52 +++ .../resident-profile/candidate-t8.txt | 56 +++ .../resident-profile/manifest.json | 214 +++++++++++ .../resident-scalar-fixed/manifest.json | 360 ++++++++++++++++++ .../resident-scalar-fixed/results.csv | 31 ++ docs/on-disk-format.md | 5 +- 22 files changed, 2096 insertions(+), 2 deletions(-) create mode 100644 benchmarks/results-20261011/README.md create mode 100644 benchmarks/results-20261011/baseline-build.json create mode 100644 benchmarks/results-20261011/build_kvbench.py create mode 100644 benchmarks/results-20261011/compare.py create mode 100644 benchmarks/results-20261011/disk-format-pre-scalar.json create mode 100644 benchmarks/results-20261011/disk-log0-final-singleload/manifest.json create mode 100644 benchmarks/results-20261011/disk-log0-final-singleload/results.csv create mode 100644 benchmarks/results-20261011/final-reader-build.json create mode 100644 benchmarks/results-20261011/format_profile.py create mode 100644 benchmarks/results-20261011/format_scalar_compare.py create mode 100644 benchmarks/results-20261011/resident-log0-final-singleload/manifest.json create mode 100644 benchmarks/results-20261011/resident-log0-final-singleload/results.csv create mode 100644 benchmarks/results-20261011/resident-log0/manifest.json create mode 100644 benchmarks/results-20261011/resident-log0/results.csv create mode 100644 benchmarks/results-20261011/resident-profile/baseline-t1.txt create mode 100644 benchmarks/results-20261011/resident-profile/baseline-t8.txt create mode 100644 benchmarks/results-20261011/resident-profile/candidate-t1.txt create mode 100644 benchmarks/results-20261011/resident-profile/candidate-t8.txt create mode 100644 benchmarks/results-20261011/resident-profile/manifest.json create mode 100644 benchmarks/results-20261011/resident-scalar-fixed/manifest.json create mode 100644 benchmarks/results-20261011/resident-scalar-fixed/results.csv diff --git a/benchmarks/results-20261011/README.md b/benchmarks/results-20261011/README.md new file mode 100644 index 000000000..a75dea67e --- /dev/null +++ b/benchmarks/results-20261011/README.md @@ -0,0 +1,13 @@ +# Metadata benchmark evidence, 2026-10-11 + +See ../../docs/on-disk-format.md for interpretation and validation. +CSV files and manifests preserve the raw samples, commands, timestamps and +binary/source provenance. Absolute paths in manifests identify the original +workspace. Database images and perf recordings are deliberately omitted. + +The final implementation uses the scalar reader. `resident-log0` is the initial +reader; the two `*-final-singleload` campaigns and `resident-profile` precede +the scalar change. `resident-scalar-fixed` compares the old and final readers +on copies of exactly the same candidate image, with its own incompatible +baseline image. The retained runners document fixture sources and run order. +Baseline source is db3047a; the final implementation milestone is 33389f3. diff --git a/benchmarks/results-20261011/baseline-build.json b/benchmarks/results-20261011/baseline-build.json new file mode 100644 index 000000000..8b6c4da40 --- /dev/null +++ b/benchmarks/results-20261011/baseline-build.json @@ -0,0 +1,30 @@ +{ + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/include", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/benchmarks/kvbench.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench" + ], + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e" +} diff --git a/benchmarks/results-20261011/build_kvbench.py b/benchmarks/results-20261011/build_kvbench.py new file mode 100644 index 000000000..aa5cc0e15 --- /dev/null +++ b/benchmarks/results-20261011/build_kvbench.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python3 +import argparse +import hashlib +import json +from pathlib import Path +import subprocess + +p = argparse.ArgumentParser() +p.add_argument('tree', type=Path) +p.add_argument('--build-root', default='build') +a = p.parse_args() +tree = a.tree.resolve() +root = Path(__file__).resolve().parent.parent +source = root / 'baseline/benchmarks/kvbench.c' +binary = tree / a.build_root / 'release/bin/kvbench' +binary.parent.mkdir(parents=True, exist_ok=True) +command = ['cc', '-D_GNU_SOURCE', '-ggdb', '-Wall', '-pthread', '-Werror', + '-DXXH_STATIC_LINKING_ONLY', '-fPIC', '-DSPLINTERDB_PLATFORM_DIR=platform_linux', + '-O3', '-ffast-math', '-flto', '-march=native', '-DKVBENCH_CLOSE_HAS_FORCE', + '-I', str(tree / 'include'), str(source), + str(tree / a.build_root / 'release/lib/libsplinterdb.a'), '-laio', '-lxxhash', '-lm', + '-lpthread', '-o', str(binary)] +subprocess.run(command, check=True) +provenance = dict(command=command, source_sha256=hashlib.sha256(source.read_bytes()).hexdigest(), + binary_sha256=hashlib.sha256(binary.read_bytes()).hexdigest()) +binary.with_suffix('.json').write_text(json.dumps(provenance, indent=2) + '\n') +print(binary) diff --git a/benchmarks/results-20261011/compare.py b/benchmarks/results-20261011/compare.py new file mode 100644 index 000000000..f62c65cf0 --- /dev/null +++ b/benchmarks/results-20261011/compare.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""Paired O_DIRECT public API runs; invoke through run_job.py benchmark.""" +import argparse +import csv +import hashlib +import json +from pathlib import Path +import subprocess +import time + +p = argparse.ArgumentParser() +p.add_argument('--project', required=True) +p.add_argument('--regime', choices=['resident', 'disk'], required=True) +p.add_argument('--seconds', type=int, default=5) +p.add_argument('--log', type=int, default=0) +p.add_argument('--baseline', type=Path) +p.add_argument('--campaign', default='') +p.add_argument('--fixture', type=Path) +p.add_argument('--candidate-binary', type=Path) +p.add_argument('--baseline-binary', type=Path) +p.add_argument('--phases', help='Comma-separated phase override') +a = p.parse_args() +root = Path(__file__).resolve().parent.parent +out = root / 'benchmarks' / a.project / (a.regime + '-log' + str(a.log) + ('-' + a.campaign if a.campaign else '')) +out.mkdir(parents=True, exist_ok=False) +trees = {'baseline': a.baseline or root / 'baseline', 'candidate': root / 'worktrees' / a.project} +phases = ['load:random', 'warm', 'point:uniform@1', 'point:uniform@8'] +phases += [f'range:uniform:{length}@{threads}' for threads in [1, 8] for length in [1, 2, 10, 100, 10000]] +if a.phases: + phases = a.phases.split(',') +manifest = {'project': a.project, 'regime': a.regime, 'io': 'O_DIRECT', 'runs': []} +if a.fixture: + a.fixture = a.fixture.resolve(strict=True) + if 'load:random' in phases: + phases.remove('load:random') + # Reuse the identical immutable tree in each sample. Concurrent loading can + # produce different branch shapes, obscuring CPU-only query changes. + manifest['fixture'] = str(a.fixture) + manifest['fixture_sha256'] = subprocess.check_output(['sha256sum', str(a.fixture)], text=True).split()[0] +for index, version in enumerate(['baseline', 'candidate', 'candidate', 'baseline']): + binary = (a.baseline_binary if version == 'baseline' else a.candidate_binary) or trees[version] / 'build/release/bin/kvbench' + db = out / f'{index}-{version}.db' + if a.fixture: + subprocess.run(['cp', '--reflink=auto', '--sparse=always', str(a.fixture), str(db)], check=True) + argv = ['taskset', '-c', '0-11', str(binary), '--db', str(db), '--open' if a.fixture else '--create', + '--n', '200000' if a.regime == 'resident' else '2000000', + '--cache-mib', '128' if a.regime == 'resident' else '64', + '--disk-gib', '8', '--log', str(a.log), '--threads', '8', + '--mt-bg', '1', '--normal-bg', '3', '--memtable-mib', '4', + '--secs', str(a.seconds), '--warmup', '1', '--csv', str(out / 'results.csv'), + '--vtag', version, '--stag', a.regime, '--otag', str(index), + '--rep', str(index // 2), '--seed', '42', '--hang-secs', '60', + '--hang-dir', str(out), *phases] + record = dict(index=index, version=version, command=argv, started=time.time(), + binary_sha256=hashlib.sha256(binary.read_bytes()).hexdigest()) + print(f'{a.project} {a.regime}: run {index + 1}/4 {version}', flush=True) + with (out / f'{index}-{version}.log').open('w') as log: + result = subprocess.run(argv, stdout=log, stderr=subprocess.STDOUT) + record.update(returncode=result.returncode, finished=time.time()) + if db.exists(): + record.update(db_allocated_bytes=db.stat().st_blocks * 512, db_apparent_bytes=db.stat().st_size) + manifest['runs'].append(record) + (out / 'manifest.json').write_text(json.dumps(manifest, indent=2) + '\n') + if result.returncode: + raise SystemExit(result.returncode) +with (out / 'results.csv').open() as stream: + rows = list(csv.DictReader(stream)) +for row in rows: + for name in ['violations', 'errors']: + if name in row and int(row[name]) != 0: + raise RuntimeError('Correctness failures: ' + repr(row)) +print('Completed', out, flush=True) diff --git a/benchmarks/results-20261011/disk-format-pre-scalar.json b/benchmarks/results-20261011/disk-format-pre-scalar.json new file mode 100644 index 000000000..3d4e081c3 --- /dev/null +++ b/benchmarks/results-20261011/disk-format-pre-scalar.json @@ -0,0 +1,5 @@ +{ + "src/routing_filter.c": "9de195af62910b303d59e09f3ecaf0006d1ce1867bbb0048b1eb29c0c04df4db", + "build/release/bin/kvbench": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "build/release/lib/libsplinterdb.a": "cd4dd673c7e27e0f3356e4fdc67522a14b38c475dc4770be7324ab81470821f3" +} diff --git a/benchmarks/results-20261011/disk-log0-final-singleload/manifest.json b/benchmarks/results-20261011/disk-log0-final-singleload/manifest.json new file mode 100644 index 000000000..3a555b240 --- /dev/null +++ b/benchmarks/results-20261011/disk-log0-final-singleload/manifest.json @@ -0,0 +1,283 @@ +{ + "project": "disk-format", + "regime": "disk", + "io": "O_DIRECT", + "runs": [ + { + "index": 0, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/0-baseline.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/results.csv", + "--vtag", + "baseline", + "--stag", + "disk", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791756998.3438628, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791757096.6433449, + "db_allocated_bytes": 523608064, + "db_apparent_bytes": 2510688256 + }, + { + "index": 1, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/1-candidate.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/results.csv", + "--vtag", + "candidate", + "--stag", + "disk", + "--otag", + "1", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791757096.6435323, + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "returncode": 0, + "finished": 1791757194.9192994, + "db_allocated_bytes": 515788800, + "db_apparent_bytes": 2510688256 + }, + { + "index": 2, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/2-candidate.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/results.csv", + "--vtag", + "candidate", + "--stag", + "disk", + "--otag", + "2", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791757194.9194977, + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "returncode": 0, + "finished": 1791757293.1430647, + "db_allocated_bytes": 514957312, + "db_apparent_bytes": 2510688256 + }, + { + "index": 3, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/3-baseline.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload/results.csv", + "--vtag", + "baseline", + "--stag", + "disk", + "--otag", + "3", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/disk-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791757293.143284, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791757391.4352462, + "db_allocated_bytes": 511049728, + "db_apparent_bytes": 2510688256 + } + ] +} diff --git a/benchmarks/results-20261011/disk-log0-final-singleload/results.csv b/benchmarks/results-20261011/disk-log0-final-singleload/results.csv new file mode 100644 index 000000000..2859cb701 --- /dev/null +++ b/benchmarks/results-20261011/disk-log0-final-singleload/results.csv @@ -0,0 +1,57 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +baseline,disk,0,0,load:random@1,1,0,2000000,2000000,1.950,1025854.8,0.552,1.648,4.160,42346.061,0.912,2.518,794349.6,185470976,486088704,92.74,243.04,nan,nan,nan,nan,nan,0,0.0,1.950,0.075,2.024,0.017,0,37306368,2.1105,2510688256,523608064,0.053,0,0,0,0,0 +baseline,disk,0,0,warm,8,0,2000000,2000000,0.099,20274803.3,nan,nan,nan,0.000,nan,0.430,4655038.4,292454400,32768,146.23,0.02,nan,nan,nan,nan,nan,2000000,20274803.3,0.099,0.000,0.099,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,point:uniform@1,1,0,2000000,176621,7.000,25230.5,45.568,91.136,471.040,2064.648,39.599,0.821,215145.6,570363904,0,3229.31,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,point:uniform@8,8,0,2000000,1331730,7.000,190242.6,47.616,76.800,1097.728,2385.498,42.011,6.864,194024.7,4300054528,0,3228.92,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:1@1,1,0,2000000,96574,7.000,13795.9,64.000,133.120,1523.712,2467.459,72.451,7.487,12899.6,1910173696,0,19779.38,0.00,64.000,133.120,72.107,nan,0.312,96574,13795.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:2@1,1,0,2000000,84634,7.000,12090.3,66.560,430.080,1490.944,5494.968,82.674,7.358,11502.7,1988276224,4096,23492.64,0.05,65.024,430.080,80.005,2338.58,0.316,169268,24180.6,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:10@1,1,0,2000000,53864,7.000,7694.7,87.040,794.624,1884.160,15838.040,129.919,6.621,8135.0,2060484608,4096,38253.46,0.08,68.608,696.320,109.681,2208.54,0.347,538638,76946.6,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:100@1,1,0,2000000,24620,7.000,3517.0,235.520,1007.616,3047.424,5986.935,284.256,3.625,6790.8,1975783424,0,80251.15,0.00,76.800,532.480,101.653,1840.62,0.368,2462000,351704.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:10000@1,1,0,2000000,1195,7.000,170.7,5963.776,22282.240,44564.480,53342.969,5861.585,2.587,462.0,2850979840,0,2385757.19,0.00,145.408,876.544,180.336,569.18,0.866,11913972,1701949.1,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:1@8,8,0,2000000,493306,7.000,70469.9,99.328,323.584,1916.928,7852.723,113.458,59.225,8329.4,9781293056,4096,19828.04,0.01,99.328,323.584,112.664,nan,0.747,493306,70469.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:2@8,8,0,2000000,378855,7.000,54120.4,107.520,794.624,2195.456,5504.872,147.744,58.115,6519.0,8963907584,4096,23660.52,0.01,105.472,778.240,143.157,3794.83,0.769,757710,108240.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:10@8,8,0,2000000,235105,7.000,33585.5,165.888,1228.800,2785.280,18405.041,238.116,53.059,4431.0,9057742848,4096,38526.37,0.02,137.216,1130.496,204.983,3590.21,0.799,2351041,335853.3,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:100@8,8,0,2000000,113040,7.000,16148.2,405.504,1884.160,5046.272,19066.701,495.319,30.644,3688.9,9213263872,0,81504.46,0.00,141.312,1040.384,201.330,2962.15,0.729,11303730,1614777.7,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,0,0,range:uniform:10000@8,8,0,2000000,6247,7.000,892.4,9043.968,34078.720,58195.968,77072.626,8963.544,20.672,302.2,14959923200,0,2394737.19,0.00,227.328,1753.088,292.332,868.51,1.234,62376565,8910697.2,7.000,0.063,7.063,0.007,0,278528,nan,2510688256,523608064,0.007,0,0,0,0,0 +candidate,disk,1,0,load:random@1,1,0,2000000,2000000,1.945,1028524.0,0.616,1.776,4.000,20066.187,0.909,2.697,741432.2,177815552,477597696,88.91,238.80,nan,nan,nan,nan,nan,0,0.0,1.945,0.091,2.035,0.031,3858432,37978112,2.0789,2510688256,515788800,0.008,0,0,0,0,0 +candidate,disk,1,0,warm,8,0,2000000,2000000,0.115,17371094.4,nan,nan,nan,0.000,nan,0.495,4038878.2,293220352,32768,146.61,0.02,nan,nan,nan,nan,nan,2000000,17371094.4,0.115,0.000,0.115,0.000,0,0,nan,0,0,0.005,0,0,0,0,2 +candidate,disk,1,0,point:uniform@1,1,0,2000000,183288,7.000,26183.4,45.568,74.752,198.656,2795.440,38.156,0.851,215293.3,591736832,0,3228.45,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,point:uniform@8,8,0,2000000,1395646,7.000,199372.7,48.640,72.704,99.328,5157.110,40.081,7.628,182955.1,4507254784,4096,3229.51,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:1@1,1,0,2000000,104040,7.000,14862.5,64.000,119.808,174.080,932.218,67.249,7.546,13788.1,2057637888,4096,19777.37,0.04,64.000,119.808,66.921,nan,0.294,104040,14862.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:2@1,1,0,2000000,88959,7.000,12708.1,66.560,380.928,614.400,1754.744,78.653,7.402,12017.5,2089132032,4096,23484.21,0.05,65.024,372.736,76.000,2328.55,0.309,177918,25416.3,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:10@1,1,0,2000000,60549,7.000,8649.6,84.992,548.864,761.856,4009.685,115.573,6.634,9127.2,2318872576,0,38297.45,0.00,68.608,495.616,96.328,2100.03,0.330,605488,86496.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:100@1,1,0,2000000,25874,7.000,3696.2,235.520,843.776,1196.032,1636.230,270.502,3.653,7082.7,2077765632,0,80303.22,0.00,74.752,462.848,95.322,1765.90,0.340,2587400,369620.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:10000@1,1,0,2000000,1258,7.000,179.7,5963.776,12976.128,14024.704,14876.512,5563.065,2.744,458.5,3006136320,4096,2389615.52,3.26,145.408,471.040,164.227,540.76,0.921,12543972,1791950.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:1@8,8,0,2000000,551883,7.000,78837.8,99.328,190.464,247.808,8185.242,101.407,59.517,9272.7,10937942016,4096,19819.31,0.01,97.280,190.464,100.600,nan,0.759,551883,78837.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:2@8,8,0,2000000,422491,7.000,60353.9,107.520,581.632,827.392,1673.080,132.478,58.411,7233.0,10000805888,4096,23671.05,0.01,103.424,565.248,128.063,3618.40,0.773,844982,120707.7,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:10@8,8,0,2000000,257341,7.000,36761.8,161.792,794.624,1163.264,4363.002,217.533,53.139,4842.8,9942478848,0,38635.42,0.00,133.120,729.088,185.511,3465.28,0.810,2573401,367616.7,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:100@8,8,0,2000000,120814,7.000,17258.6,405.504,1261.568,1720.320,7129.879,463.450,30.787,3924.2,9829883904,0,81363.78,0.00,137.216,712.704,185.868,2796.67,0.703,12081085,1725818.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,1,0,range:uniform:10000@8,8,0,2000000,6639,7.000,948.4,8781.824,18612.224,22806.528,25332.445,8434.402,21.900,303.2,15765860352,0,2374734.20,0.00,223.232,729.088,254.015,819.31,1.190,66291618,9469943.2,7.000,0.068,7.069,0.008,0,278528,nan,2510688256,515788800,0.005,0,0,0,0,0 +candidate,disk,2,0,load:random@1,1,1,2000000,2000000,1.919,1042043.6,0.600,1.680,4.064,28558.508,0.898,2.611,765958.9,180187136,476430336,90.09,238.22,nan,nan,nan,nan,nan,0,0.0,1.919,0.078,1.997,0.026,4186112,38305792,2.0755,2510688256,514957312,0.010,0,0,0,0,0 +candidate,disk,2,0,warm,8,1,2000000,2000000,0.114,17501225.3,nan,nan,nan,0.000,nan,0.529,3784073.2,293048320,32768,146.52,0.02,nan,nan,nan,nan,nan,2000000,17501225.3,0.114,0.000,0.114,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,point:uniform@1,1,1,2000000,183880,7.000,26267.9,45.568,74.752,198.656,676.502,38.033,0.837,219694.6,593772544,0,3229.13,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,point:uniform@8,8,1,2000000,1398239,7.000,199743.8,48.640,72.704,97.280,1518.121,40.006,7.637,183093.1,4516302848,4096,3229.99,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:1@1,1,1,2000000,103096,7.000,14727.5,65.024,121.856,178.176,6859.647,67.864,7.545,13663.6,2039238656,4096,19780.00,0.04,65.024,121.856,67.527,nan,0.303,103096,14727.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:2@1,1,1,2000000,88737,7.000,12676.4,66.560,380.928,630.784,1909.966,78.851,7.392,12005.1,2083926016,4096,23484.30,0.05,65.024,380.928,76.214,2302.61,0.319,177474,25352.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:10@1,1,1,2000000,60498,7.000,8642.3,84.992,548.864,827.392,2163.931,115.663,6.654,9091.7,2316374016,0,38288.44,0.00,68.608,503.808,96.607,2078.27,0.334,604978,86422.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:100@1,1,1,2000000,25915,7.000,3702.1,235.520,843.776,1196.032,2587.557,270.058,3.641,7118.0,2079649792,0,80248.88,0.00,74.752,462.848,95.246,1762.20,0.337,2591500,370205.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:10000@1,1,1,2000000,1262,7.000,180.3,5963.776,12713.984,14286.848,14338.813,5545.656,2.721,463.8,3017924608,4096,2391382.42,3.25,145.408,446.464,159.600,539.54,0.922,12583972,1797663.6,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:1@8,8,1,2000000,550143,7.000,78588.9,99.328,194.560,251.904,6972.107,101.729,59.556,9237.4,10894966784,4096,19803.88,0.01,99.328,190.464,100.901,nan,0.779,550143,78588.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:2@8,8,1,2000000,419661,7.000,59949.7,107.520,581.632,860.160,3041.906,133.371,58.418,7183.8,9931198464,4096,23664.81,0.01,105.472,565.248,128.801,3748.02,0.798,839322,119899.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:10@8,8,1,2000000,258128,7.000,36874.2,161.792,794.624,1130.496,4736.650,216.872,53.406,4833.3,9966845952,0,38612.03,0.00,133.120,712.704,185.435,3400.02,0.813,2581271,368740.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:100@8,8,1,2000000,120896,7.000,17270.3,397.312,1261.568,1753.088,4073.654,463.119,30.866,3916.8,9840189440,0,81393.84,0.00,137.216,729.088,185.816,2793.76,0.710,12089285,1726986.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,disk,2,0,range:uniform:10000@8,8,1,2000000,6606,7.000,943.7,8781.824,19136.512,23330.816,25225.905,8477.470,21.996,300.3,15691059200,0,2375273.87,0.00,223.232,794.624,258.789,823.24,1.209,65952689,9421544.8,7.000,0.054,7.054,0.008,0,278528,nan,2510688256,514957312,0.005,0,0,0,0,0 +baseline,disk,3,0,load:random@1,1,1,2000000,2000000,1.933,1034763.0,0.616,1.776,4.064,18946.487,0.903,2.716,736424.5,174329856,472858624,87.16,236.43,nan,nan,nan,nan,nan,0,0.0,1.933,0.103,2.036,0.027,3928064,37969920,2.0598,2510688256,511049728,0.008,0,0,0,0,0 +baseline,disk,3,0,warm,8,1,2000000,2000000,0.115,17451219.8,nan,nan,nan,0.000,nan,0.538,3720431.9,292868096,32768,146.43,0.02,nan,nan,nan,nan,nan,2000000,17451219.8,0.115,0.000,0.115,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,point:uniform@1,1,1,2000000,183452,7.000,26206.7,45.568,78.848,202.752,617.867,38.123,0.845,217101.9,592338944,0,3228.85,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,point:uniform@8,8,1,2000000,1397469,7.000,199633.6,48.640,72.704,97.280,1682.196,40.031,7.629,183183.0,4514091008,4096,3230.19,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:1@1,1,1,2000000,102943,7.000,14705.7,65.024,121.856,182.272,593.118,67.965,7.542,13649.6,2035879936,0,19776.77,0.00,65.024,121.856,67.606,nan,0.326,102943,14705.7,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:2@1,1,1,2000000,88325,7.000,12617.6,66.560,389.120,647.168,7106.227,79.216,7.387,11956.3,2074435584,4096,23486.39,0.05,65.024,389.120,76.555,2318.70,0.328,176650,25235.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:10@1,1,1,2000000,59732,7.000,8532.9,84.992,565.248,827.392,1312.646,117.149,6.649,8983.9,2287091712,0,38289.22,0.00,68.608,512.000,97.782,2110.04,0.362,597318,85329.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:100@1,1,1,2000000,25808,7.000,3686.8,231.424,860.160,1163.264,4260.667,271.182,3.637,7096.1,2072236032,0,80294.33,0.00,74.752,462.848,95.549,1770.16,0.372,2580800,368676.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:10000@1,1,1,2000000,1259,7.000,179.9,5963.776,12976.128,14548.992,15664.463,5560.611,2.723,462.4,3007279104,4096,2388625.18,3.25,145.408,413.696,161.236,540.85,0.926,12553972,1793378.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:1@8,8,1,2000000,545093,7.000,77868.0,101.376,194.560,256.000,3996.061,102.669,59.478,9164.6,10804973568,0,19822.26,0.00,99.328,194.560,101.833,nan,0.787,545093,77868.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:2@8,8,1,2000000,413443,7.000,59061.6,107.520,598.016,892.928,7506.657,135.376,58.293,7092.4,9777311744,4096,23648.51,0.01,105.472,598.016,130.775,3779.42,0.797,826886,118123.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:10@8,8,1,2000000,254613,7.000,36372.1,161.792,811.008,1163.264,3875.101,219.863,53.311,4776.0,9838321664,0,38640.30,0.00,133.120,745.472,188.056,3440.99,0.815,2546121,363719.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:100@8,8,1,2000000,120386,7.000,17197.5,397.312,1294.336,1753.088,4690.126,465.082,31.016,3881.4,9812140032,0,81505.66,0.00,141.312,729.088,187.552,2795.83,0.735,12038285,1719707.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,disk,3,0,range:uniform:10000@8,8,1,2000000,6591,7.000,941.5,8781.824,19136.512,23855.104,24556.088,8494.273,22.134,297.8,15733878784,0,2387176.27,0.00,227.328,712.704,256.962,825.08,1.291,65811618,9401353.5,7.000,0.080,7.080,0.007,0,278528,nan,2510688256,511049728,0.007,0,0,0,0,0 diff --git a/benchmarks/results-20261011/final-reader-build.json b/benchmarks/results-20261011/final-reader-build.json new file mode 100644 index 000000000..993b75f57 --- /dev/null +++ b/benchmarks/results-20261011/final-reader-build.json @@ -0,0 +1,30 @@ +{ + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/include", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/benchmarks/kvbench.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/bin/kvbench" + ], + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "binary_sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d" +} diff --git a/benchmarks/results-20261011/format_profile.py b/benchmarks/results-20261011/format_profile.py new file mode 100644 index 000000000..597a70a30 --- /dev/null +++ b/benchmarks/results-20261011/format_profile.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +import hashlib,json,subprocess,time +from pathlib import Path +root=Path(__file__).resolve().parent.parent +out=root/'benchmarks/disk-format/resident-profile' +out.mkdir(parents=True,exist_ok=False) +manifest=[] +for version,slot in [('baseline',0),('candidate',1)]: + tree=root/('baseline' if version=='baseline' else 'worktrees/disk-format') + binary=tree/'build/release/bin/kvbench' + fixture=root/f'benchmarks/disk-format/resident-log0-final-singleload/{slot}-{version}.db' + for threads in [1,8]: + stem=out/f'{version}-t{threads}' + db=stem.with_suffix('.db') + subprocess.run(['cp','--reflink=auto','--sparse=always',str(fixture),str(db)],check=True) + perf=stem.with_suffix('.perf') + argv=['perf','record','-e','cycles:u','-F','997','-o',str(perf),'--', + 'taskset','-c','0-11',str(binary),'--db',str(db),'--open','--n','200000', + '--cache-mib','128','--disk-gib','8','--log','0','--threads',str(threads), + '--mt-bg','1','--normal-bg','3','--memtable-mib','4','--secs','15', + '--warmup','1','--seed','42','--hang-secs','60','warm',f'point:uniform@{threads}'] + record=dict(version=version,threads=threads,fixture=str(fixture),command=argv, + binary_sha256=hashlib.sha256(binary.read_bytes()).hexdigest(),started=time.time()) + print('Format profile',version,threads,flush=True) + with stem.with_suffix('.log').open('w') as f: + result=subprocess.run(argv,stdout=f,stderr=subprocess.STDOUT,timeout=120) + record.update(returncode=result.returncode,finished=time.time());manifest.append(record) + (out/'manifest.json').write_text(json.dumps(manifest,indent=2)+'\n') + if result.returncode:raise SystemExit(result.returncode) + with stem.with_suffix('.txt').open('w') as f: + subprocess.run(['perf','report','--stdio','--no-children','--percent-limit','0.3','--sort','symbol','-i',str(perf)],stdout=f,check=True) +print('Completed',out,flush=True) diff --git a/benchmarks/results-20261011/format_scalar_compare.py b/benchmarks/results-20261011/format_scalar_compare.py new file mode 100644 index 000000000..3b9ff4d02 --- /dev/null +++ b/benchmarks/results-20261011/format_scalar_compare.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +import csv,hashlib,json,subprocess,time +from pathlib import Path +root=Path(__file__).resolve().parent.parent +out=root/'benchmarks/disk-format/resident-scalar-fixed' +out.mkdir(parents=True,exist_ok=False) +profiles={'baseline':(root/'baseline/build/release/bin/kvbench','0-baseline.db'), + 'pre-scalar':(root/'worktrees/disk-format/build/release/bin/kvbench','1-candidate.db'), + 'candidate':(root/'worktrees/disk-format/build-scalar/release/bin/kvbench','1-candidate.db')} +manifest=dict(runs=[],io='O_DIRECT',note='Old and new format reader binaries use copies of the exact same candidate fixture; baseline uses its own incompatible format fixture.') +for index,version in enumerate(['baseline','pre-scalar','candidate','candidate','pre-scalar','baseline']): + binary,fixture=profiles[version] + source=root/'benchmarks/disk-format/resident-log0-final-singleload'/fixture + db=out/f'{index}-{version}.db' + subprocess.run(['cp','--reflink=auto','--sparse=always',str(source),str(db)],check=True) + argv=['taskset','-c','0-11',str(binary),'--db',str(db),'--open','--n','200000', + '--cache-mib','128','--disk-gib','8','--log','0','--threads','8', + '--mt-bg','1','--normal-bg','3','--memtable-mib','4','--secs','10','--warmup','1', + '--csv',str(out/'results.csv'),'--vtag',version,'--stag','resident','--otag',str(index), + '--rep',str(index//3),'--seed','42','--hang-secs','60','--hang-dir',str(out),'warm', + 'point:uniform@1','point:uniform@8','range:uniform:1@1','range:uniform:1@8'] + record=dict(index=index,version=version,command=argv,fixture=str(source),started=time.time(), + binary_sha256=hashlib.sha256(binary.read_bytes()).hexdigest()) + print('Format scalar comparison',index+1,version,flush=True) + with (out/f'{index}-{version}.log').open('w') as f: + result=subprocess.run(argv,stdout=f,stderr=subprocess.STDOUT,timeout=120) + record.update(returncode=result.returncode,finished=time.time());manifest['runs'].append(record) + (out/'manifest.json').write_text(json.dumps(manifest,indent=2)+'\n') + if result.returncode:raise SystemExit(result.returncode) +rows=list(csv.DictReader((out/'results.csv').open())) +assert all(int(r[x])==0 for r in rows for x in ['violations','errors']) +print('Completed',out,flush=True) diff --git a/benchmarks/results-20261011/resident-log0-final-singleload/manifest.json b/benchmarks/results-20261011/resident-log0-final-singleload/manifest.json new file mode 100644 index 000000000..244d7a9fe --- /dev/null +++ b/benchmarks/results-20261011/resident-log0-final-singleload/manifest.json @@ -0,0 +1,283 @@ +{ + "project": "disk-format", + "regime": "resident", + "io": "O_DIRECT", + "runs": [ + { + "index": 0, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/0-baseline.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/results.csv", + "--vtag", + "baseline", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791756112.9108348, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791756209.2992659, + "db_allocated_bytes": 28905472, + "db_apparent_bytes": 162664448 + }, + { + "index": 1, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/results.csv", + "--vtag", + "candidate", + "--stag", + "resident", + "--otag", + "1", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791756209.2994514, + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "returncode": 0, + "finished": 1791756305.707102, + "db_allocated_bytes": 28905472, + "db_apparent_bytes": 162664448 + }, + { + "index": 2, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/2-candidate.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/results.csv", + "--vtag", + "candidate", + "--stag", + "resident", + "--otag", + "2", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791756305.7073188, + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "returncode": 0, + "finished": 1791756402.1552217, + "db_allocated_bytes": 28905472, + "db_apparent_bytes": 162664448 + }, + { + "index": 3, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/3-baseline.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "7", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/results.csv", + "--vtag", + "baseline", + "--stag", + "resident", + "--otag", + "3", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload", + "load:random@1", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791756402.1554475, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791756498.5873287, + "db_allocated_bytes": 28905472, + "db_apparent_bytes": 162664448 + } + ] +} diff --git a/benchmarks/results-20261011/resident-log0-final-singleload/results.csv b/benchmarks/results-20261011/resident-log0-final-singleload/results.csv new file mode 100644 index 000000000..3a6f1593d --- /dev/null +++ b/benchmarks/results-20261011/resident-log0-final-singleload/results.csv @@ -0,0 +1,57 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +baseline,resident,0,0,load:random@1,1,0,200000,200000,0.150,1337426.5,0.520,2.592,4.288,2466.708,0.712,0.183,1091726.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.150,0.094,0.244,0.023,0,28749824,1.1593,162664448,28905472,0.009,0,0,0,0,0 +baseline,resident,0,0,warm,8,0,200000,200000,0.022,8901833.3,nan,nan,nan,0.000,nan,0.057,3534006.0,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,8901833.3,0.022,0.000,0.022,0.000,0,0,nan,0,0,0.006,0,0,0,0,1 +baseline,resident,0,0,point:uniform@1,1,0,200000,8843062,7.000,1263249.7,0.744,1.072,4.064,223.296,0.767,6.997,1263796.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,point:uniform@8,8,0,200000,41987838,7.000,5998080.1,1.264,1.840,2.592,1699.815,1.293,55.987,749953.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:1@1,1,0,200000,805565,7.000,115078.2,8.576,12.160,34.304,63.562,8.664,6.999,115089.8,0,4096,0.00,0.01,8.320,11.904,8.401,nan,0.237,805565,115078.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:2@1,1,0,200000,776932,7.000,110986.2,8.832,12.416,33.280,61.530,8.983,7.000,110991.8,0,0,0.00,0.00,8.320,11.904,8.546,182.69,0.242,1553856,221971.3,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10@1,1,0,200000,691238,7.000,98744.8,10.112,12.416,18.176,154.489,10.100,6.998,98775.1,0,0,0.00,0.00,8.320,10.112,8.501,148.60,0.248,6912182,987419.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:100@1,1,0,200000,318164,7.000,45450.7,21.760,27.392,32.000,93.160,21.974,7.000,45454.5,0,0,0.00,0.00,8.576,11.392,8.633,132.21,0.243,31808302,4543917.7,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10000@1,1,0,200000,5552,7.000,793.1,1294.336,1425.408,1556.480,1824.412,1260.700,7.000,793.1,0,0,0.00,0.00,12.160,18.176,12.246,128.12,0.424,54088170,7726627.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:1@8,8,0,200000,4035802,7.000,576523.3,12.672,18.688,28.928,451.683,13.833,55.985,72087.6,0,4096,0.00,0.00,12.160,18.176,13.239,nan,0.555,4035802,576523.3,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:2@8,8,0,200000,3794555,7.000,542062.1,13.696,20.224,30.464,136.217,14.712,55.990,67771.7,0,4096,0.00,0.00,12.928,18.688,13.825,287.34,0.580,7589084,1084120.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10@8,8,0,200000,3289257,7.000,469879.2,15.744,23.296,32.512,113.436,16.978,55.992,58745.5,0,0,0.00,0.00,13.184,19.200,14.187,241.16,0.599,32891724,4698671.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:100@8,8,0,200000,1597582,7.000,228218.3,33.280,46.592,55.808,375.868,35.006,56.020,28517.9,0,0,0.00,0.00,13.184,19.200,13.905,207.22,0.571,159717802,22816061.9,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10000@8,8,0,200000,27032,7.000,3861.6,1949.696,2523.136,2588.672,2914.946,2071.569,55.987,482.8,0,0,0.00,0.00,17.664,27.904,17.794,210.45,0.860,263711809,37671786.2,7.000,0.089,7.089,0.012,0,311296,nan,162664448,28905472,0.006,0,0,0,0,0 +candidate,resident,1,0,load:random@1,1,0,200000,200000,0.155,1293725.7,0.536,2.656,4.416,2404.477,0.735,0.191,1049389.5,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.155,0.099,0.254,0.017,0,28749824,1.1593,162664448,28905472,0.008,0,0,0,0,0 +candidate,resident,1,0,warm,8,0,200000,200000,0.022,8914035.2,nan,nan,nan,0.000,nan,0.062,3243330.9,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,8914035.2,0.022,0.000,0.022,0.000,0,0,nan,0,0,0.005,0,0,0,0,6 +candidate,resident,1,0,point:uniform@1,1,0,200000,8379703,7.000,1197074.5,0.792,1.104,1.392,48.699,0.809,6.999,1197279.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,point:uniform@8,8,0,200000,40338757,7.000,5762504.3,1.296,1.904,2.528,415.409,1.344,55.949,720990.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:1@1,1,0,200000,785601,7.000,112226.2,8.832,10.624,16.000,140.262,8.884,6.999,112240.0,0,4096,0.00,0.01,8.576,10.368,8.597,nan,0.260,785601,112226.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:2@1,1,0,200000,782984,7.000,111850.8,8.832,10.624,17.664,53.596,8.914,6.999,111872.1,0,4096,0.00,0.01,8.320,10.112,8.463,178.92,0.259,1565960,223700.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10@1,1,0,200000,698431,7.000,99772.4,9.856,12.928,35.328,46.113,9.997,7.000,99773.9,0,0,0.00,0.00,8.320,10.368,8.427,144.37,0.257,6984112,997695.3,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:100@1,1,0,200000,312293,7.000,44611.7,22.272,28.416,39.424,702.819,22.388,6.995,44643.3,0,0,0.00,0.00,8.832,12.160,8.885,133.65,0.261,31221332,4460029.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10000@1,1,0,200000,5566,7.000,795.1,1294.336,1425.408,1654.784,1877.648,1257.675,7.000,795.2,0,0,0.00,0.00,12.416,18.688,12.375,127.78,0.438,54228170,7746606.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:1@8,8,0,200000,3988148,7.000,569717.4,12.928,18.688,27.392,115.724,13.999,55.992,71226.7,0,4096,0.00,0.00,12.416,17.664,13.376,nan,0.583,3988148,569717.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:2@8,8,0,200000,3924658,7.000,560647.8,13.184,19.200,28.416,133.656,14.225,55.989,70097.3,0,4096,0.00,0.00,12.416,17.664,13.342,279.71,0.583,7849291,1121292.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10@8,8,0,200000,3396113,7.000,485144.6,15.232,22.784,31.488,111.844,16.444,55.990,60655.9,0,0,0.00,0.00,12.672,18.176,13.726,232.87,0.602,33960266,4851322.3,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:100@8,8,0,200000,1509424,7.000,215624.7,35.328,49.664,58.880,857.691,37.051,55.983,26962.1,0,0,0.00,0.00,13.952,19.712,14.776,218.48,0.629,150905624,21557212.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10000@8,8,0,200000,27553,7.000,3936.0,1916.928,2457.600,2523.136,2666.951,2032.494,55.991,492.1,0,4096,0.00,0.15,17.664,26.880,18.064,206.33,0.860,268900405,38412960.9,7.000,0.098,7.099,0.012,0,311296,nan,162664448,28905472,0.005,0,0,0,0,0 +candidate,resident,2,0,load:random@1,1,1,200000,200000,0.157,1274957.4,0.536,2.720,4.544,2535.615,0.746,0.193,1036731.4,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.157,0.107,0.264,0.018,0,28749824,1.1593,162664448,28905472,0.009,0,0,0,0,0 +candidate,resident,2,0,warm,8,1,200000,200000,0.021,9501731.6,nan,nan,nan,0.000,nan,0.058,3435894.8,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9501731.6,0.021,0.000,0.021,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,point:uniform@1,1,1,200000,8314028,7.000,1187672.6,0.808,1.136,1.392,54.637,0.816,7.000,1187745.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,point:uniform@8,8,1,200000,38755858,7.000,5536381.3,1.360,1.968,2.848,3153.587,1.401,55.982,692296.7,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:1@1,1,1,200000,793028,7.000,113285.5,8.832,10.368,15.744,80.873,8.801,6.999,113310.9,0,4096,0.00,0.01,8.576,10.112,8.518,nan,0.256,793028,113285.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:2@1,1,1,200000,785882,7.000,112264.8,8.832,10.624,18.176,64.223,8.881,6.999,112279.4,0,4096,0.00,0.01,8.320,10.112,8.433,180.04,0.255,1571756,224528.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:10@1,1,1,200000,695300,7.000,99324.9,9.856,13.440,35.328,72.984,10.041,7.000,99332.5,0,0,0.00,0.00,8.320,11.136,8.467,145.13,0.255,6952802,993220.2,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:100@1,1,1,200000,321475,7.000,45923.8,21.760,26.368,30.464,300.532,21.748,7.000,45926.5,0,0,0.00,0.00,8.576,10.112,8.562,130.50,0.256,32139274,4591201.6,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:10000@1,1,1,200000,5580,7.000,797.1,1294.336,1359.872,1392.640,1712.950,1254.302,7.004,796.7,0,0,0.00,0.00,12.160,16.000,12.135,127.47,0.430,54360316,7765604.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:1@8,8,1,200000,3987220,7.000,569584.4,12.928,18.688,28.416,160.175,14.001,55.988,71216.2,0,4096,0.00,0.00,12.416,18.176,13.371,nan,0.591,3987220,569584.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:2@8,8,1,200000,3779778,7.000,539950.2,13.696,20.224,29.440,138.476,14.769,55.990,67507.9,0,0,0.00,0.00,12.672,18.688,13.840,293.33,0.615,7559531,1079896.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:10@8,8,1,200000,3278143,7.000,468291.9,16.000,23.296,32.512,459.319,17.036,55.985,58554.5,0,0,0.00,0.00,13.184,19.200,14.202,242.20,0.633,32780575,4682797.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:100@8,8,1,200000,1620333,7.000,231468.9,33.280,45.568,52.736,952.579,34.514,55.993,28938.3,0,0,0.00,0.00,12.928,18.176,13.635,204.76,0.593,161992260,23141031.3,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:10000@8,8,1,200000,27536,7.000,3933.6,1916.928,2392.064,2457.600,2526.870,2033.510,56.026,491.5,0,4096,0.00,0.15,17.152,23.808,17.242,206.61,0.822,268673366,38380598.5,7.000,0.130,7.130,0.011,0,311296,nan,162664448,28905472,0.006,0,0,0,0,0 +baseline,resident,3,0,load:random@1,1,1,200000,200000,0.177,1131475.6,0.616,3.040,4.928,2613.707,0.840,0.216,926161.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.177,0.106,0.282,0.020,0,28749824,1.1593,162664448,28905472,0.014,0,0,0,0,0 +baseline,resident,3,0,warm,8,1,200000,200000,0.022,9068615.5,nan,nan,nan,0.000,nan,0.066,3043676.8,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9068615.5,0.022,0.000,0.022,0.000,0,0,nan,0,0,0.007,0,0,0,0,3 +baseline,resident,3,0,point:uniform@1,1,1,200000,8441269,7.000,1205868.6,0.792,1.136,1.584,41.176,0.803,7.000,1205930.4,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,point:uniform@8,8,1,200000,40085277,7.000,5726277.0,1.296,1.936,3.232,186.821,1.355,55.988,715964.7,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:1@1,1,1,200000,756664,7.000,108092.0,9.088,12.160,16.000,51.980,9.224,7.000,108098.3,0,4096,0.00,0.01,8.832,11.904,8.917,nan,0.279,756664,108092.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:2@1,1,1,200000,788024,7.000,112572.4,8.832,10.368,17.152,145.899,8.857,7.000,112581.9,0,4096,0.00,0.01,8.320,9.856,8.399,182.28,0.263,1576040,225143.6,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10@1,1,1,200000,666918,7.000,95270.5,10.368,14.208,18.688,64.203,10.469,6.999,95282.7,0,0,0.00,0.00,8.576,11.648,8.779,154.47,0.286,6668990,952678.0,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:100@1,1,1,200000,315484,7.000,45067.5,21.760,27.392,30.976,65.005,22.161,7.000,45069.9,0,0,0.00,0.00,8.576,11.136,8.649,133.67,0.269,31540394,4505601.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10000@1,1,1,200000,5507,7.000,786.7,1294.336,1392.640,1490.944,1696.039,1271.172,7.000,786.7,0,4096,0.00,0.74,12.416,16.000,12.353,129.17,0.445,53650060,7664013.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:1@8,8,1,200000,4141461,7.000,591619.7,12.416,18.176,26.880,130.394,13.480,55.989,73968.9,0,4096,0.00,0.00,11.904,17.152,12.890,nan,0.552,4141461,591619.7,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:2@8,8,1,200000,3759517,7.000,537057.5,13.696,20.224,30.464,127.799,14.849,55.991,67145.5,0,4096,0.00,0.00,12.928,18.688,13.939,293.07,0.596,7519010,1074111.5,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10@8,8,1,200000,3390112,7.000,484285.7,15.232,22.784,32.000,190.665,16.472,55.986,60552.6,0,0,0.00,0.00,12.672,18.688,13.732,236.34,0.593,33900235,4842730.8,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:100@8,8,1,200000,1551619,7.000,221652.4,34.304,47.616,57.856,164.680,36.042,55.987,27713.8,0,0,0.00,0.00,13.440,19.200,14.248,213.95,0.598,155122903,22159666.4,7.000,0.000,7.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10000@8,8,1,200000,26921,7.000,3845.7,1949.696,2523.136,2588.672,2661.104,2080.014,55.994,480.8,0,0,0.00,0.00,17.152,27.392,17.609,211.32,0.858,262666699,37522457.5,7.000,0.087,7.088,0.012,0,311296,nan,162664448,28905472,0.007,0,0,0,0,0 diff --git a/benchmarks/results-20261011/resident-log0/manifest.json b/benchmarks/results-20261011/resident-log0/manifest.json new file mode 100644 index 000000000..d76a5f635 --- /dev/null +++ b/benchmarks/results-20261011/resident-log0/manifest.json @@ -0,0 +1,283 @@ +{ + "project": "disk-format", + "regime": "resident", + "io": "O_DIRECT", + "runs": [ + { + "index": 0, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/0-baseline.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/results.csv", + "--vtag", + "baseline", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0", + "load:random", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791754071.6557283, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791754143.9231174, + "db_allocated_bytes": 28901376, + "db_apparent_bytes": 162926592 + }, + { + "index": 1, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/1-candidate.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/results.csv", + "--vtag", + "candidate", + "--stag", + "resident", + "--otag", + "1", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0", + "load:random", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791754143.9233093, + "binary_sha256": "c0109a373531cadc438d001e8b1324b5aa4051afed757a22a2e54fa80de2ff81", + "returncode": 0, + "finished": 1791754216.2111654, + "db_allocated_bytes": 28884992, + "db_apparent_bytes": 162926592 + }, + { + "index": 2, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/2-candidate.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/results.csv", + "--vtag", + "candidate", + "--stag", + "resident", + "--otag", + "2", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0", + "load:random", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791754216.2113616, + "binary_sha256": "c0109a373531cadc438d001e8b1324b5aa4051afed757a22a2e54fa80de2ff81", + "returncode": 0, + "finished": 1791754288.4951994, + "db_allocated_bytes": 28901376, + "db_apparent_bytes": 162795520 + }, + { + "index": 3, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/3-baseline.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0/results.csv", + "--vtag", + "baseline", + "--stag", + "resident", + "--otag", + "3", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0", + "load:random", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:2@1", + "range:uniform:10@1", + "range:uniform:100@1", + "range:uniform:10000@1", + "range:uniform:1@8", + "range:uniform:2@8", + "range:uniform:10@8", + "range:uniform:100@8", + "range:uniform:10000@8" + ], + "started": 1791754288.495404, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791754360.8111336, + "db_allocated_bytes": 28884992, + "db_apparent_bytes": 162926592 + } + ] +} diff --git a/benchmarks/results-20261011/resident-log0/results.csv b/benchmarks/results-20261011/resident-log0/results.csv new file mode 100644 index 000000000..8aa11bda4 --- /dev/null +++ b/benchmarks/results-20261011/resident-log0/results.csv @@ -0,0 +1,57 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +baseline,resident,0,0,load:random,8,0,200000,200000,0.037,5436979.3,0.776,5.440,17.152,3581.176,1.171,0.283,706641.4,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.037,0.084,0.121,0.021,0,28745728,1.1591,162926592,28901376,0.016,0,0,0,0,0 +baseline,resident,0,0,warm,8,0,200000,200000,0.020,10026190.4,nan,nan,nan,0.000,nan,0.053,3767400.7,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,10026190.4,0.020,0.000,0.020,0.000,0,0,nan,0,0,0.006,0,0,0,0,1 +baseline,resident,0,0,point:uniform@1,1,0,200000,6356534,5.000,1271242.0,0.728,1.072,4.672,37.649,0.763,5.000,1271359.4,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,point:uniform@8,8,0,200000,30896160,5.000,6178958.7,1.232,1.776,2.464,83.992,1.255,39.992,772556.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:1@1,1,0,200000,569434,5.000,113880.8,8.576,10.368,15.488,39.831,8.754,5.004,113800.8,0,0,0.00,0.00,8.320,10.112,8.485,nan,0.243,569434,113880.8,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:2@1,1,0,200000,561873,5.000,112371.2,8.832,10.624,16.256,39.682,8.872,5.000,112380.6,0,0,0.00,0.00,8.320,9.856,8.437,181.13,0.240,1123739,224741.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10@1,1,0,200000,490903,5.000,98175.6,9.856,14.720,35.328,145.902,10.158,5.000,98189.7,0,0,0.00,0.00,8.320,12.416,8.560,148.77,0.246,4908853,981720.1,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:100@1,1,0,200000,228715,5.000,45740.6,21.760,26.880,32.000,167.478,21.835,5.000,45745.7,0,0,0.00,0.00,8.576,10.112,8.583,131.31,0.242,22865928,4572942.4,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10000@1,1,0,200000,3930,5.000,786.0,1294.336,1458.176,1589.248,1716.746,1272.076,4.999,786.2,0,4096,0.00,1.04,12.416,20.736,12.569,129.25,0.434,38276705,7654870.5,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:1@8,8,0,200000,2908944,5.000,581760.9,12.672,18.688,28.416,94.583,13.708,39.988,72745.0,0,0,0.00,0.00,12.160,17.664,13.106,nan,0.563,2908944,581760.9,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:2@8,8,0,200000,2729036,5.000,545782.2,13.440,19.712,29.440,606.426,14.611,39.990,68242.9,0,0,0.00,0.00,12.672,18.688,13.712,286.24,0.592,5458051,1091560.3,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10@8,8,0,200000,2516226,5.000,503207.0,14.720,22.272,30.976,117.166,15.852,39.992,62918.7,0,0,0.00,0.00,12.160,18.176,13.222,226.57,0.572,25161576,5031933.6,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:100@8,8,0,200000,1112667,5.000,222522.7,34.304,48.640,57.856,130.522,35.901,39.991,27823.3,0,0,0.00,0.00,13.440,19.712,14.194,212.97,0.606,111239698,22246870.2,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,0,0,range:uniform:10000@8,8,0,200000,19206,5.000,3841.0,1949.696,2523.136,2523.136,2646.323,2082.358,39.990,480.3,0,4096,0.00,0.21,17.664,25.344,17.894,211.51,0.858,187386834,37475494.7,5.000,0.073,5.074,0.012,0,307200,nan,162926592,28901376,0.006,0,0,0,0,0 +candidate,resident,1,0,load:random,8,0,200000,200000,0.040,5042460.9,0.888,5.952,18.688,4176.966,1.317,0.314,637700.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.040,0.103,0.142,0.026,0,28729344,1.1584,162926592,28884992,0.010,0,0,0,0,0 +candidate,resident,1,0,warm,8,0,200000,200000,0.022,9130476.0,nan,nan,nan,0.000,nan,0.055,3617029.0,27668480,0,138.34,0.00,nan,nan,nan,nan,nan,200000,9130476.0,0.022,0.000,0.022,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,point:uniform@1,1,0,200000,6032132,5.000,1206385.8,0.792,1.104,1.520,38.035,0.803,5.000,1206473.5,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,point:uniform@8,8,0,200000,28456566,5.000,5691065.6,1.328,1.936,3.936,82.287,1.363,39.990,711589.6,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:1@1,1,0,200000,541022,5.000,108198.6,9.088,12.672,19.200,73.513,9.214,4.999,108217.1,0,0,0.00,0.00,8.832,12.160,8.918,nan,0.268,541022,108198.6,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:2@1,1,0,200000,541222,5.000,108240.0,9.088,11.136,16.640,41.420,9.210,5.000,108247.1,0,0,0.00,0.00,8.576,10.624,8.748,185.35,0.264,1082437,216478.6,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10@1,1,0,200000,477617,5.000,95519.5,10.368,13.440,18.688,56.307,10.440,4.999,95547.3,0,0,0.00,0.00,8.832,11.392,8.808,150.32,0.266,4775993,955159.9,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:100@1,1,0,200000,224176,5.000,44833.8,22.272,27.904,37.376,206.800,22.275,5.004,44803.6,0,0,0.00,0.00,8.832,12.160,8.822,133.09,0.266,22412181,4482293.6,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10000@1,1,0,200000,3965,5.000,793.0,1294.336,1359.872,1392.640,1456.846,1261.016,5.000,793.1,0,4096,0.00,1.03,12.416,16.640,12.475,128.13,0.448,38622684,7724152.8,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:1@8,8,0,200000,2984381,5.000,596849.5,12.416,18.176,26.368,113.189,13.362,39.991,74626.3,0,0,0.00,0.00,11.648,17.152,12.745,nan,0.580,2984381,596849.5,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:2@8,8,0,200000,2742838,5.000,548543.5,13.440,19.712,28.928,159.412,14.538,39.992,68584.4,0,0,0.00,0.00,12.672,18.176,13.610,282.82,0.625,5485654,1097082.6,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10@8,8,0,200000,2353866,5.000,470752.6,16.000,22.784,32.000,112.369,16.945,39.985,58868.1,0,0,0.00,0.00,13.184,18.688,14.100,240.67,0.657,23538012,4707396.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:100@8,8,0,200000,1105498,5.000,221089.7,34.304,47.616,57.856,130.684,36.132,39.988,27646.0,0,0,0.00,0.00,13.696,19.200,14.210,214.75,0.645,110523208,22103646.9,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,1,0,range:uniform:10000@8,8,0,200000,19880,5.000,3975.8,1884.160,2457.600,2457.600,2520.710,2011.897,39.988,497.1,0,4096,0.00,0.21,17.152,24.832,17.282,204.51,0.848,193803860,38758916.5,5.000,0.089,5.089,0.013,0,307200,nan,162926592,28884992,0.006,0,0,0,0,0 +candidate,resident,2,0,load:random,8,1,200000,200000,0.040,5048086.6,0.888,5.824,15.744,4081.362,1.311,0.313,638394.3,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.040,0.108,0.148,0.026,0,28745728,1.1591,162795520,28901376,0.009,0,0,0,0,0 +candidate,resident,2,0,warm,8,1,200000,200000,0.023,8720037.9,nan,nan,nan,0.000,nan,0.062,3201229.3,27668480,0,138.34,0.00,nan,nan,nan,nan,nan,200000,8720037.9,0.023,0.000,0.023,0.000,0,0,nan,0,0,0.006,0,0,0,0,5 +candidate,resident,2,0,point:uniform@1,1,1,200000,5860081,5.000,1171953.7,0.808,1.136,1.424,46.505,0.827,4.999,1172211.3,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,point:uniform@8,8,1,200000,28117064,5.000,5623155.5,1.328,1.968,3.552,95.195,1.380,39.989,703115.7,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:1@1,1,1,200000,555066,5.000,111006.3,8.832,10.624,16.000,104.467,8.981,5.000,111021.0,0,0,0.00,0.00,8.576,10.368,8.695,nan,0.258,555066,111006.3,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:2@1,1,1,200000,544933,5.000,108979.9,9.088,13.952,17.664,39.110,9.148,5.000,108984.5,0,4096,0.00,0.01,8.576,13.440,8.693,183.06,0.259,1089859,217958.5,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,42,0,0 +candidate,resident,2,0,range:uniform:10@1,1,1,200000,474064,5.000,94807.8,10.368,13.952,18.176,40.761,10.519,5.000,94814.1,0,0,0.00,0.00,8.832,11.648,8.892,149.84,0.265,4740463,948043.1,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:100@1,1,1,200000,223247,5.000,44647.0,22.272,26.880,32.000,55.236,22.368,5.000,44649.8,0,0,0.00,0.00,8.832,10.624,8.941,132.86,0.264,22319399,4463643.8,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:10000@1,1,1,200000,3945,5.000,789.0,1294.336,1359.872,1392.640,1423.905,1267.321,5.000,789.0,0,4096,0.00,1.04,12.672,16.640,12.731,128.74,0.426,38422684,7684136.5,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:1@8,8,1,200000,2789155,5.000,557806.0,13.184,19.200,28.416,107.136,14.296,39.989,69747.3,0,0,0.00,0.00,12.672,18.176,13.654,nan,0.602,2789155,557806.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:2@8,8,1,200000,2706090,5.000,541194.3,13.696,19.712,29.952,162.895,14.734,39.992,67665.8,0,0,0.00,0.00,12.928,18.688,13.806,293.31,0.614,5412160,1082384.5,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:10@8,8,1,200000,2397406,5.000,479460.7,15.744,22.784,31.488,149.161,16.637,39.992,59946.9,0,0,0.00,0.00,13.184,18.688,13.872,236.23,0.618,23973419,4794479.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:100@8,8,1,200000,1154168,5.000,230823.2,33.280,45.568,53.760,488.464,34.608,39.989,28862.5,0,0,0.00,0.00,13.184,18.176,13.682,205.19,0.595,115388844,23076732.4,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +candidate,resident,2,0,range:uniform:10000@8,8,1,200000,19981,5.000,3996.0,1884.160,2392.064,2457.600,2592.379,2001.615,39.989,499.7,0,4096,0.00,0.20,17.152,25.344,17.478,203.28,0.818,194937713,38985630.0,5.000,0.081,5.081,0.013,0,307200,nan,162795520,28901376,0.006,0,0,0,0,0 +baseline,resident,3,0,load:random,8,1,200000,200000,0.038,5291898.1,0.888,6.080,18.176,4247.047,1.317,0.314,637251.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.038,0.122,0.160,0.031,0,28729344,1.1584,162926592,28884992,0.009,0,0,0,0,0 +baseline,resident,3,0,warm,8,1,200000,200000,0.022,8897997.7,nan,nan,nan,0.000,nan,0.064,3119394.8,27664384,0,138.32,0.00,nan,nan,nan,nan,nan,200000,8897997.7,0.022,0.000,0.022,0.000,0,0,nan,0,0,0.006,0,0,0,0,1 +baseline,resident,3,0,point:uniform@1,1,1,200000,5877188,5.000,1175377.6,0.808,1.264,1.936,139.617,0.824,5.000,1175510.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,point:uniform@8,8,1,200000,28054468,5.000,5610661.2,1.360,1.936,3.424,1574.669,1.383,39.987,701584.0,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:1@1,1,1,200000,548855,5.000,109765.3,9.088,10.624,15.488,39.987,9.082,5.000,109771.4,0,0,0.00,0.00,8.832,10.368,8.801,nan,0.253,548855,109765.3,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:2@1,1,1,200000,553186,5.000,110631.3,8.832,11.136,16.256,41.775,9.011,5.000,110638.4,0,0,0.00,0.00,8.576,10.368,8.570,184.64,0.243,1106365,221261.2,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10@1,1,1,200000,494036,5.000,98801.9,10.112,12.672,18.176,76.173,10.094,5.000,98811.7,0,0,0.00,0.00,8.320,10.112,8.489,149.20,0.249,4940183,987983.9,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:100@1,1,1,200000,219730,5.000,43943.7,22.272,29.440,40.448,71.491,22.727,5.000,43946.3,0,0,0.00,0.00,8.832,13.184,8.944,136.57,0.253,21967807,4393334.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10000@1,1,1,200000,3883,5.000,776.6,1327.104,1458.176,1556.480,1752.270,1287.744,5.000,776.6,0,4096,0.00,1.05,12.672,19.200,12.721,130.84,0.456,37816658,7563006.9,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:1@8,8,1,200000,2866530,5.000,573280.0,12.928,18.688,28.928,132.299,13.910,39.996,71670.6,0,0,0.00,0.00,12.416,18.176,13.292,nan,0.579,2866530,573280.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:2@8,8,1,200000,2813185,5.000,562610.2,12.928,19.200,27.904,192.396,14.174,40.022,70291.8,0,0,0.00,0.00,12.160,18.176,13.302,278.90,0.573,5626348,1125216.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10@8,8,1,200000,2340104,5.000,468000.6,16.000,23.808,32.512,107.777,17.045,39.993,58512.4,0,0,0.00,0.00,13.440,19.200,14.184,246.17,0.625,23400415,4679880.6,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:100@8,8,1,200000,1131657,5.000,226321.3,33.280,47.616,55.808,109.739,35.298,39.993,28296.6,0,0,0.00,0.00,13.184,18.688,13.846,210.55,0.593,113138282,22626646.9,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +baseline,resident,3,0,range:uniform:10000@8,8,1,200000,19218,5.000,3843.4,1982.464,2523.136,2654.208,4084.980,2081.057,39.990,480.6,0,4096,0.00,0.21,17.664,29.952,18.220,211.41,0.879,187404822,37479154.0,5.000,0.100,5.100,0.012,0,307200,nan,162926592,28884992,0.006,0,0,0,0,0 diff --git a/benchmarks/results-20261011/resident-profile/baseline-t1.txt b/benchmarks/results-20261011/resident-profile/baseline-t1.txt new file mode 100644 index 000000000..f96535d7c --- /dev/null +++ b/benchmarks/results-20261011/resident-profile/baseline-t1.txt @@ -0,0 +1,61 @@ +# To display the perf.data header info, please use --header/--header-only options. +# +# +# Total Lost Samples: 0 +# +# Samples: 1 of event 'cpu_atom/cycles/u' +# Event count (approx.): 501504 +# +# Overhead Symbol IPC [IPC Coverage] +# ........ ...................... .................... +# + 100.00% [.] 0x000000000003cba5 - - + + +# Samples: 16K of event 'cpu_core/cycles/u' +# Event count (approx.): 59202519974 +# +# Overhead Symbol IPC [IPC Coverage] +# ........ ............................................................... .................... +# + 17.08% [.] trunk_ondisk_bundle_merge_lookup.constprop.0 - - + 16.27% [.] btree_find_pivot_checked.lto_priv.0 - - + 11.71% [.] btree_find_tuple_checked.lto_priv.0 - - + 7.93% [.] clockcache_try_get_read.constprop.0 - - + 5.91% [.] clockcache_unget_virtual - - + 4.06% [.] clockcache_get_virtual - - + 3.91% [.] key_compare - - + 3.29% [.] 0x0000000000162419 - - + 3.23% [.] __vdso_clock_gettime - - + 2.48% [.] 0x00000000001621f8 - - + 2.17% [.] core_lookup.constprop.0 - - + 2.13% [.] 0x000000000016241d - - + 1.49% [.] __popcountdi2 - - + 1.38% [.] 0x00000000001624c7 - - + 1.24% [.] point_worker - - + 0.95% [.] 0x00000000001621f4 - - + 0.81% [.] splinterdb_lookup.constprop.0 - - + 0.81% [.] batch_rwlock_get - - + 0.76% [.] routing_get_header.isra.0 - - + 0.74% [.] trunk_ondisk_node_handle_init.lto_priv.0 - - + 0.72% [.] XXH32 - - + 0.70% [.] trunk_init_root_handle - - + 0.68% [.] 0x00000000001624cb - - + 0.64% [.] batch_rwlock_get.constprop.0 - - + 0.53% [.] 0x0000000000162428 - - + 0.48% [.] routing_header_view_init.lto_priv.0 - - + 0.44% [.] btree_node_get_checked.lto_priv.0 - - + 0.43% [.] 0x00000000001621fe - - + 0.43% [.] memcmp@plt - - + 0.43% [.] trunk_merge_lookup.constprop.0 - - + 0.40% [.] routing_filter_get_params.lto_priv.0 - - + 0.37% [.] 0x0000000000162405 - - + 0.35% [.] btree_lookup_node_internal.lto_priv.0 - - + 0.34% [.] 0x0000000000162423 - - + 0.33% [.] trunk_ondisk_node_read_pivot.lto_priv.0 - - + 0.33% [.] 0x000000000016288d - - + + +# +# (Tip: Compare performance results with: perf diff [ ]) +# diff --git a/benchmarks/results-20261011/resident-profile/baseline-t8.txt b/benchmarks/results-20261011/resident-profile/baseline-t8.txt new file mode 100644 index 000000000..c6adfd8d1 --- /dev/null +++ b/benchmarks/results-20261011/resident-profile/baseline-t8.txt @@ -0,0 +1,58 @@ +# To display the perf.data header info, please use --header/--header-only options. +# +# +# Total Lost Samples: 0 +# +# Samples: 1 of event 'cpu_atom/cycles/u' +# Event count (approx.): 501504 +# +# Overhead Symbol IPC [IPC Coverage] +# ........ ...................... .................... +# + 100.00% [.] 0x0000000000163d34 - - + + +# Samples: 127K of event 'cpu_core/cycles/u' +# Event count (approx.): 344814990024 +# +# Overhead Symbol IPC [IPC Coverage] +# ........ ............................................................... .................... +# + 15.10% [.] clockcache_try_get_read.constprop.0 - - + 14.91% [.] trunk_ondisk_bundle_merge_lookup.constprop.0 - - + 13.85% [.] btree_find_pivot_checked.lto_priv.0 - - + 8.11% [.] clockcache_unget_virtual - - + 7.84% [.] btree_find_tuple_checked.lto_priv.0 - - + 3.56% [.] clockcache_get_virtual - - + 3.35% [.] key_compare - - + 3.34% [.] batch_rwlock_get - - + 2.67% [.] 0x0000000000162419 - - + 2.65% [.] __vdso_clock_gettime - - + 2.53% [.] core_lookup.constprop.0 - - + 2.14% [.] 0x00000000001621f8 - - + 1.99% [.] __popcountdi2 - - + 1.71% [.] batch_rwlock_get.constprop.0 - - + 1.50% [.] 0x000000000016241d - - + 1.43% [.] point_worker - - + 1.00% [.] trunk_init_root_handle - - + 0.95% [.] 0x00000000001624c7 - - + 0.71% [.] routing_get_header.isra.0 - - + 0.67% [.] splinterdb_lookup.constprop.0 - - + 0.64% [.] 0x00000000001621f4 - - + 0.59% [.] XXH32 - - + 0.53% [.] 0x00000000001624cb - - + 0.49% [.] trunk_ondisk_node_handle_init.lto_priv.0 - - + 0.45% [.] routing_filter_get_params.lto_priv.0 - - + 0.44% [.] memcmp@plt - - + 0.41% [.] btree_node_get_checked.lto_priv.0 - - + 0.38% [.] 0x00000000001621fe - - + 0.38% [.] trunk_merge_lookup.constprop.0 - - + 0.37% [.] routing_header_view_init.lto_priv.0 - - + 0.32% [.] 0x0000000000162428 - - + 0.30% [.] btree_lookup_node_internal.lto_priv.0 - - + 0.30% [.] trunk_ondisk_node_read_pivot.lto_priv.0 - - + + +# +# (Tip: To show context switches in perf report sample context add --switch-events to perf record.) +# diff --git a/benchmarks/results-20261011/resident-profile/candidate-t1.txt b/benchmarks/results-20261011/resident-profile/candidate-t1.txt new file mode 100644 index 000000000..72608c2c7 --- /dev/null +++ b/benchmarks/results-20261011/resident-profile/candidate-t1.txt @@ -0,0 +1,52 @@ +# To display the perf.data header info, please use --header/--header-only options. +# +# +# Total Lost Samples: 0 +# +# Samples: 16K of event 'cpu_core/cycles/u' +# Event count (approx.): 59322903877 +# +# Overhead Symbol IPC [IPC Coverage] +# ........ ............................................................... .................... +# + 17.13% [.] trunk_ondisk_bundle_merge_lookup.constprop.0 - - + 16.74% [.] btree_find_pivot_checked.lto_priv.0 - - + 11.72% [.] btree_find_tuple_checked.lto_priv.0 - - + 7.81% [.] clockcache_try_get_read.constprop.0 - - + 5.90% [.] clockcache_unget_virtual - - + 4.17% [.] clockcache_get_virtual - - + 3.57% [.] key_compare - - + 3.46% [.] 0x0000000000162419 - - + 3.24% [.] __vdso_clock_gettime - - + 2.25% [.] 0x00000000001621f8 - - + 2.18% [.] core_lookup.constprop.0 - - + 1.89% [.] 0x000000000016241d - - + 1.36% [.] 0x00000000001624c7 - - + 1.33% [.] __popcountdi2 - - + 1.17% [.] point_worker - - + 0.92% [.] 0x00000000001621f4 - - + 0.88% [.] batch_rwlock_get - - + 0.86% [.] XXH32 - - + 0.78% [.] splinterdb_lookup.constprop.0 - - + 0.76% [.] routing_get_header.isra.0 - - + 0.67% [.] batch_rwlock_get.constprop.0 - - + 0.66% [.] trunk_init_root_handle - - + 0.62% [.] trunk_ondisk_node_handle_init.lto_priv.0 - - + 0.59% [.] 0x0000000000162405 - - + 0.56% [.] trunk_ondisk_node_read_pivot_bundle.lto_priv.0 - - + 0.55% [.] memcmp@plt - - + 0.54% [.] 0x00000000001624cb - - + 0.47% [.] routing_header_view_init.lto_priv.0 - - + 0.43% [.] 0x00000000001621fe - - + 0.42% [.] trunk_merge_lookup.constprop.0 - - + 0.39% [.] 0x0000000000162428 - - + 0.33% [.] 0x0000000000162423 - - + 0.32% [.] 0x00000000001621f0 - - + 0.32% [.] btree_node_get_checked.lto_priv.0 - - + 0.31% [.] 0x000000000016288d - - + 0.31% [.] routing_filter_get_params.lto_priv.0 - - + + +# +# (Tip: When collecting LBR backtraces use --stitch-lbr to handle more than 32 deep entries: perf record --call-graph lbr ; perf report --stitch-lbr) +# diff --git a/benchmarks/results-20261011/resident-profile/candidate-t8.txt b/benchmarks/results-20261011/resident-profile/candidate-t8.txt new file mode 100644 index 000000000..a8c8d05e8 --- /dev/null +++ b/benchmarks/results-20261011/resident-profile/candidate-t8.txt @@ -0,0 +1,56 @@ +# To display the perf.data header info, please use --header/--header-only options. +# +# +# Total Lost Samples: 0 +# +# Samples: 1 of event 'cpu_atom/cycles/u' +# Event count (approx.): 501504 +# +# Overhead Symbol IPC [IPC Coverage] +# ........ ...................... .................... +# + 100.00% [.] 0x00000000000a0fef - - + + +# Samples: 127K of event 'cpu_core/cycles/u' +# Event count (approx.): 343757214652 +# +# Overhead Symbol IPC [IPC Coverage] +# ........ ............................................................... .................... +# + 17.00% [.] clockcache_try_get_read.constprop.0 - - + 14.36% [.] trunk_ondisk_bundle_merge_lookup.constprop.0 - - + 13.10% [.] btree_find_pivot_checked.lto_priv.0 - - + 8.43% [.] clockcache_unget_virtual - - + 7.80% [.] btree_find_tuple_checked.lto_priv.0 - - + 3.68% [.] clockcache_get_virtual - - + 3.40% [.] batch_rwlock_get - - + 3.24% [.] key_compare - - + 2.65% [.] __vdso_clock_gettime - - + 2.60% [.] 0x0000000000162419 - - + 2.50% [.] core_lookup.constprop.0 - - + 2.04% [.] 0x00000000001621f8 - - + 1.67% [.] __popcountdi2 - - + 1.50% [.] batch_rwlock_get.constprop.0 - - + 1.41% [.] 0x000000000016241d - - + 1.37% [.] point_worker - - + 1.14% [.] trunk_init_root_handle - - + 0.91% [.] 0x00000000001624c7 - - + 0.71% [.] routing_get_header.isra.0 - - + 0.67% [.] splinterdb_lookup.constprop.0 - - + 0.61% [.] trunk_ondisk_node_read_pivot_bundle.lto_priv.0 - - + 0.59% [.] 0x00000000001621f4 - - + 0.55% [.] XXH32 - - + 0.47% [.] routing_filter_get_params.lto_priv.0 - - + 0.47% [.] trunk_ondisk_node_handle_init.lto_priv.0 - - + 0.45% [.] 0x00000000001624cb - - + 0.43% [.] btree_node_get_checked.lto_priv.0 - - + 0.38% [.] memcmp@plt - - + 0.37% [.] 0x00000000001621fe - - + 0.36% [.] trunk_merge_lookup.constprop.0 - - + 0.32% [.] 0x0000000000162405 - - + + +# +# (Tip: To compare hot regions in two workloads use perf record -b -o file ... ; perf diff --stream file1 file2) +# diff --git a/benchmarks/results-20261011/resident-profile/manifest.json b/benchmarks/results-20261011/resident-profile/manifest.json new file mode 100644 index 000000000..a0faddb8c --- /dev/null +++ b/benchmarks/results-20261011/resident-profile/manifest.json @@ -0,0 +1,214 @@ +[ + { + "version": "baseline", + "threads": 1, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/0-baseline.db", + "command": [ + "perf", + "record", + "-e", + "cycles:u", + "-F", + "997", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/baseline-t1.perf", + "--", + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/baseline-t1.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "warm", + "point:uniform@1" + ], + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "started": 1791761156.1697598, + "returncode": 0, + "finished": 1791761172.688094 + }, + { + "version": "baseline", + "threads": 8, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/0-baseline.db", + "command": [ + "perf", + "record", + "-e", + "cycles:u", + "-F", + "997", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/baseline-t8.perf", + "--", + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/baseline-t8.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "warm", + "point:uniform@8" + ], + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "started": 1791761172.7636127, + "returncode": 0, + "finished": 1791761189.3314357 + }, + { + "version": "candidate", + "threads": 1, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "command": [ + "perf", + "record", + "-e", + "cycles:u", + "-F", + "997", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/candidate-t1.perf", + "--", + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/candidate-t1.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "warm", + "point:uniform@1" + ], + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "started": 1791761189.515156, + "returncode": 0, + "finished": 1791761206.0340154 + }, + { + "version": "candidate", + "threads": 8, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "command": [ + "perf", + "record", + "-e", + "cycles:u", + "-F", + "997", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/candidate-t8.perf", + "--", + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-profile/candidate-t8.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "warm", + "point:uniform@8" + ], + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "started": 1791761206.0708003, + "returncode": 0, + "finished": 1791761222.5906453 + } +] diff --git a/benchmarks/results-20261011/resident-scalar-fixed/manifest.json b/benchmarks/results-20261011/resident-scalar-fixed/manifest.json new file mode 100644 index 000000000..e6acff6b1 --- /dev/null +++ b/benchmarks/results-20261011/resident-scalar-fixed/manifest.json @@ -0,0 +1,360 @@ +{ + "runs": [ + { + "index": 0, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/0-baseline.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "10", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/results.csv", + "--vtag", + "baseline", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/0-baseline.db", + "started": 1791762068.8393192, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791762113.0404754 + }, + { + "index": 1, + "version": "pre-scalar", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/1-pre-scalar.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "10", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/results.csv", + "--vtag", + "pre-scalar", + "--stag", + "resident", + "--otag", + "1", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "started": 1791762113.0554235, + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "returncode": 0, + "finished": 1791762157.3057103 + }, + { + "index": 2, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/2-candidate.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "10", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/results.csv", + "--vtag", + "candidate", + "--stag", + "resident", + "--otag", + "2", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "started": 1791762157.317927, + "binary_sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "returncode": 0, + "finished": 1791762201.4851263 + }, + { + "index": 3, + "version": "candidate", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/3-candidate.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "10", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/results.csv", + "--vtag", + "candidate", + "--stag", + "resident", + "--otag", + "3", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "started": 1791762201.4973776, + "binary_sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "returncode": 0, + "finished": 1791762245.7015123 + }, + { + "index": 4, + "version": "pre-scalar", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/4-pre-scalar.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "10", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/results.csv", + "--vtag", + "pre-scalar", + "--stag", + "resident", + "--otag", + "4", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "started": 1791762245.7122736, + "binary_sha256": "99555661022785e5d3a494935e9b83cc3f6840a158024e6a68c20b370984b85f", + "returncode": 0, + "finished": 1791762289.976912 + }, + { + "index": 5, + "version": "baseline", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/build/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/5-baseline.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "10", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed/results.csv", + "--vtag", + "baseline", + "--stag", + "resident", + "--otag", + "5", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-scalar-fixed", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/0-baseline.db", + "started": 1791762289.9912405, + "binary_sha256": "40a04c4e24b3018cade6ad4a4fd2b3a39a8b4f050a364d1001c82ccc788e0d3e", + "returncode": 0, + "finished": 1791762334.1945896 + } + ], + "io": "O_DIRECT", + "note": "Old and new format reader binaries use copies of the exact same candidate fixture; baseline uses its own incompatible format fixture." +} diff --git a/benchmarks/results-20261011/resident-scalar-fixed/results.csv b/benchmarks/results-20261011/resident-scalar-fixed/results.csv new file mode 100644 index 000000000..9b56141e8 --- /dev/null +++ b/benchmarks/results-20261011/resident-scalar-fixed/results.csv @@ -0,0 +1,31 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +baseline,resident,0,0,warm,8,0,200000,200000,0.025,7921238.2,nan,nan,nan,0.000,nan,0.055,3617683.2,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,7921238.2,0.025,0.000,0.025,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +baseline,resident,0,0,point:uniform@1,1,0,200000,12892308,10.000,1289197.7,0.728,1.072,2.144,46.083,0.751,10.000,1289246.3,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +baseline,resident,0,0,point:uniform@8,8,0,200000,60369002,10.000,6036766.2,1.264,1.840,2.528,95.659,1.285,79.993,754676.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +baseline,resident,0,0,range:uniform:1@1,1,0,200000,1164208,10.000,116417.7,8.576,10.112,14.976,42.383,8.563,10.000,116418.9,0,4096,0.00,0.00,8.320,9.856,8.293,nan,0.243,1164208,116417.7,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +baseline,resident,0,0,range:uniform:1@8,8,0,200000,5903092,10.000,590293.3,12.416,18.176,26.368,100.407,13.510,79.995,73793.2,0,4096,0.00,0.00,11.904,17.664,12.925,nan,0.547,5903092,590293.3,10.000,0.112,10.112,0.011,0,311296,nan,162664448,28655616,0.021,0,0,0,0,0 +pre-scalar,resident,1,0,warm,8,0,200000,200000,0.056,3542249.0,nan,nan,nan,0.000,nan,0.089,2258968.1,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,3542249.0,0.056,0.000,0.056,0.000,0,0,nan,0,0,0.077,0,0,0,0,5 +pre-scalar,resident,1,0,point:uniform@1,1,0,200000,12704449,10.000,1270423.0,0.744,1.072,1.328,88.872,0.762,10.000,1270480.1,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.077,0,0,0,0,0 +pre-scalar,resident,1,0,point:uniform@8,8,0,200000,56263989,10.000,5626276.7,1.360,1.936,2.400,85.461,1.378,79.995,703347.8,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.077,0,0,0,0,0 +pre-scalar,resident,1,0,range:uniform:1@1,1,0,200000,1169848,10.000,116981.8,8.320,11.648,34.304,61.353,8.522,10.000,116987.8,0,4096,0.00,0.00,8.128,11.392,8.234,nan,0.262,1169848,116981.8,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.077,0,0,0,0,0 +pre-scalar,resident,1,0,range:uniform:1@8,8,0,200000,5761530,10.000,576140.5,12.672,19.200,27.392,482.011,13.841,79.994,72024.3,0,4096,0.00,0.00,12.160,18.176,13.216,nan,0.586,5761530,576140.5,10.000,0.100,10.100,0.012,0,311296,nan,162664448,28655616,0.077,0,0,0,0,0 +candidate,resident,2,0,warm,8,0,200000,200000,0.024,8183102.5,nan,nan,nan,0.000,nan,0.059,3366833.3,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,8183102.5,0.024,0.000,0.024,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +candidate,resident,2,0,point:uniform@1,1,0,200000,12639293,10.000,1263905.1,0.744,1.072,1.360,131.784,0.766,10.000,1263952.1,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +candidate,resident,2,0,point:uniform@8,8,0,200000,57509992,10.000,5750876.5,1.296,1.904,2.784,81.288,1.349,79.995,718918.4,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +candidate,resident,2,0,range:uniform:1@1,1,0,200000,1150932,10.000,115091.0,8.576,10.368,15.488,111.028,8.662,10.000,115092.7,0,4096,0.00,0.00,8.320,9.856,8.375,nan,0.260,1150932,115091.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +candidate,resident,2,0,range:uniform:1@8,8,0,200000,5446331,10.000,544620.9,13.440,19.712,28.416,128.332,14.643,79.991,68087.2,0,4096,0.00,0.00,12.928,19.200,13.971,nan,0.631,5446331,544620.9,10.000,0.096,10.096,0.012,0,311296,nan,162664448,28655616,0.021,0,0,0,0,0 +candidate,resident,3,0,warm,8,1,200000,200000,0.024,8203613.7,nan,nan,nan,0.000,nan,0.059,3411979.5,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,8203613.7,0.024,0.000,0.024,0.000,0,0,nan,0,0,0.021,0,0,0,0,5 +candidate,resident,3,0,point:uniform@1,1,1,200000,12460260,10.000,1245993.1,0.760,1.136,1.808,53.347,0.777,10.000,1246041.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +candidate,resident,3,0,point:uniform@8,8,1,200000,55521006,10.000,5551975.8,1.360,1.968,2.528,88.987,1.398,79.998,694033.1,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +candidate,resident,3,0,range:uniform:1@1,1,1,200000,1135290,10.000,113525.9,8.576,10.368,15.232,150.408,8.782,10.000,113533.8,0,4096,0.00,0.00,8.320,10.112,8.499,nan,0.256,1135290,113525.9,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +candidate,resident,3,0,range:uniform:1@8,8,1,200000,5806627,10.000,580650.3,12.672,18.176,26.880,463.741,13.734,79.989,72593.3,0,4096,0.00,0.00,12.160,17.664,13.117,nan,0.578,5806627,580650.3,10.000,0.103,10.103,0.012,0,311296,nan,162664448,28655616,0.021,0,0,0,0,0 +pre-scalar,resident,4,0,warm,8,1,200000,200000,0.061,3303188.3,nan,nan,nan,0.000,nan,0.098,2035934.2,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,3303188.3,0.061,0.000,0.061,0.000,0,0,nan,0,0,0.044,0,0,0,0,6 +pre-scalar,resident,4,0,point:uniform@1,1,1,200000,12102713,10.000,1210247.4,0.776,1.200,1.744,39.661,0.800,10.000,1210288.5,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.044,0,0,0,0,0 +pre-scalar,resident,4,0,point:uniform@8,8,1,200000,54563390,10.000,5456207.4,1.392,1.968,2.592,242.394,1.421,79.994,682095.4,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.044,0,0,0,0,0 +pre-scalar,resident,4,0,range:uniform:1@1,1,1,200000,1167551,10.000,116752.1,8.320,10.368,29.440,179.079,8.539,10.000,116760.0,0,4096,0.00,0.00,8.128,9.856,8.262,nan,0.251,1167551,116752.1,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.044,0,0,0,0,0 +pre-scalar,resident,4,0,range:uniform:1@8,8,1,200000,5685755,10.000,568562.6,12.928,18.688,26.880,1573.161,14.027,79.990,71080.5,0,4096,0.00,0.00,12.416,18.176,13.402,nan,0.585,5685755,568562.6,10.000,0.101,10.102,0.012,0,307200,nan,162664448,28655616,0.044,0,0,0,0,0 +baseline,resident,5,0,warm,8,1,200000,200000,0.026,7781300.6,nan,nan,nan,0.000,nan,0.060,3322700.7,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,7781300.6,0.026,0.000,0.026,0.000,0,0,nan,0,0,0.021,0,0,0,0,6 +baseline,resident,5,0,point:uniform@1,1,1,200000,12580262,10.000,1258003.3,0.760,1.072,1.328,36.952,0.770,10.000,1258019.3,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +baseline,resident,5,0,point:uniform@8,8,1,200000,60569969,10.000,6056855.4,1.264,1.712,2.208,88.519,1.277,79.992,757197.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,0,0,0 +baseline,resident,5,0,range:uniform:1@1,1,1,200000,1156741,10.000,115671.1,8.576,11.904,16.640,79.276,8.618,10.000,115673.6,0,4096,0.00,0.00,8.320,11.392,8.351,nan,0.240,1156741,115671.1,10.000,0.000,10.000,0.000,0,0,nan,0,0,0.021,0,0,1,0,0 +baseline,resident,5,0,range:uniform:1@8,8,1,200000,5461697,10.000,546157.5,13.440,19.712,29.440,517.587,14.602,79.989,68280.2,0,4096,0.00,0.00,12.928,19.200,13.968,nan,0.592,5461697,546157.5,10.000,0.109,10.109,0.012,0,307200,nan,162664448,28655616,0.021,0,0,0,0,0 diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md index 4abca1573..52070f10f 100644 --- a/docs/on-disk-format.md +++ b/docs/on-disk-format.md @@ -94,7 +94,8 @@ versions; that added work remains a possible cost. Performance neutrality has **not** been established. The final scalar reader has not repeated the full disk or write matrix; the disk figures above describe the earlier reader. -Campaign artifacts: `.cache/projects-20261011/benchmarks/disk-format/` in the -main checkout. The common harness and build provenance are retained there; +Compact CSVs, manifests, profiling summaries, runners and build provenance are +committed under `benchmarks/results-20261011/`. Full campaign artifacts remain +at `.cache/projects-20261011/benchmarks/disk-format/` in the main checkout; benchmark database files were on a verified disk filesystem, with the shared benchmark lock excluding other benchmarks, builds and tests. From 07dfa1761182214f18b8269c7b24cc67bd35c0f4 Mon Sep 17 00:00:00 2001 From: Rob Johnson Date: Mon, 12 Oct 2026 00:26:48 -0700 Subject: [PATCH 4/7] Document buffered fixture copy caveat in metadata measurements --- benchmarks/results-20261011/README.md | 6 ++++++ docs/on-disk-format.md | 7 +++++++ 2 files changed, 13 insertions(+) diff --git a/benchmarks/results-20261011/README.md b/benchmarks/results-20261011/README.md index a75dea67e..5a968379b 100644 --- a/benchmarks/results-20261011/README.md +++ b/benchmarks/results-20261011/README.md @@ -11,3 +11,9 @@ the scalar change. `resident-scalar-fixed` compares the old and final readers on copies of exactly the same candidate image, with its own incompatible baseline image. The retained runners document fixture sources and run order. Baseline source is db3047a; the final implementation milestone is 33389f3. + +Method caveat: `resident-scalar-fixed` used buffered fixture copies without an +explicit fsync before O_DIRECT open/warm/timing. Pending copy writeback could +confound those samples; it is not an established explanation of any difference. +The earlier fresh-create matrices did not use copied fixtures. Original evidence +is preserved without modification; there is no fsynced metadata repeat. diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md index 52070f10f..1d08e0526 100644 --- a/docs/on-disk-format.md +++ b/docs/on-disk-format.md @@ -79,6 +79,13 @@ comparison used baseline / old reader / final reader / final reader / old reader / baseline order. Both candidate readers used copies of exactly the same format-8 image; baseline used its own format-7 image. +The final resident fixture copies were made with buffered `cp` and were not +explicitly fsynced before opening with O_DIRECT. Pending copy writeback, including +unreachable historical pages, is a possible measurement confound; this does not +establish that it caused any observed difference. The earlier resident/disk +matrices constructed fresh O_DIRECT databases and did not use this copy path. +No additional metadata benchmark was run after identifying this method caveat. + | Final reader, resident | Versus old reader | Versus baseline | | --- | ---: | ---: | | Point, 1 reader | +1.2% | -1.5% | From fb878f134c245276629828203a186dd2f26df981 Mon Sep 17 00:00:00 2001 From: Rob Johnson Date: Mon, 12 Oct 2026 15:02:24 -0700 Subject: [PATCH 5/7] Validate disk formats in the owning modules --- .../layering-20261012/.gitattributes | 3 + .../layering-20261012/0-before.log | 29 + .../layering-20261012/1-layered.log | 29 + .../layering-20261012/2-layered.log | 29 + .../layering-20261012/3-before.log | 29 + .../layering-20261012/4-layered.log | 29 + .../layering-20261012/5-before.log | 29 + .../layering-20261012/6-before.log | 29 + .../layering-20261012/7-layered.log | 29 + .../before-preservation.json | 33 + .../layering-20261012/block-summary.csv | 13 + .../layering-20261012/block-summary.txt | 21 + .../layering-20261012/build_kvbench.py | 27 + .../layering-20261012/candidate-build.json | 30 + .../layering-20261012/debug-btree.log | 35 + ...ld-config.e33dfc688a9c2bf2dad4e17ef758ec3e | 14 + .../layering-20261012/debug-filter.log | 65 + .../layering-20261012/debug-log.log | 11 + .../layering-20261012/debug-quick.log | 51 + .../layering-20261012/debug-splinter.log | 76 + .../layering-20261012/debug-superblock.log | 43 + .../layering-20261012/debug-trunk-async.log | 3 + .../format_layering_compare.py | 179 ++ .../layering-20261012/manifest.json | 615 ++++++ .../layering-20261012/measured-source.patch | 1744 +++++++++++++++++ ...ld-config.fb6bc8eeb8390429253883c09491f5bd | 14 + .../layering-20261012/release-corruption.log | 17 + .../layering-20261012/results.csv | 41 + .../layering-20261012/run_job.py | 54 + .../layering-20261012/summary.csv | 5 + .../layering-20261012/summary.txt | 5 + .../layering-20261012/test_format_layering.py | 14 + .../layering-20261012/validation.json | 87 + docs/on-disk-format.md | 73 + src/btree.c | 107 +- src/btree.h | 45 +- src/core.c | 18 +- src/rc_allocator.c | 13 + src/rc_allocator.h | 6 + src/routing_filter.c | 13 + src/routing_filter.h | 5 + src/shard_log.c | 34 +- src/shard_log.h | 4 + src/splinterdb.c | 1 + src/superblock.c | 36 +- src/superblock.h | 4 +- src/trunk.c | 93 +- src/trunk.h | 18 + tests/functional/btree_test.c | 31 +- tests/unit/btree_stress_test.c | 29 +- tests/unit/btree_test.c | 175 ++ tests/unit/splinter_test.c | 133 +- tests/unit/superblock_test.c | 8 +- tests/unit/trunk_split_test.c | 1 + 54 files changed, 4149 insertions(+), 130 deletions(-) create mode 100644 benchmarks/results-20261011/layering-20261012/.gitattributes create mode 100644 benchmarks/results-20261011/layering-20261012/0-before.log create mode 100644 benchmarks/results-20261011/layering-20261012/1-layered.log create mode 100644 benchmarks/results-20261011/layering-20261012/2-layered.log create mode 100644 benchmarks/results-20261011/layering-20261012/3-before.log create mode 100644 benchmarks/results-20261011/layering-20261012/4-layered.log create mode 100644 benchmarks/results-20261011/layering-20261012/5-before.log create mode 100644 benchmarks/results-20261011/layering-20261012/6-before.log create mode 100644 benchmarks/results-20261011/layering-20261012/7-layered.log create mode 100644 benchmarks/results-20261011/layering-20261012/before-preservation.json create mode 100644 benchmarks/results-20261011/layering-20261012/block-summary.csv create mode 100644 benchmarks/results-20261011/layering-20261012/block-summary.txt create mode 100644 benchmarks/results-20261011/layering-20261012/build_kvbench.py create mode 100644 benchmarks/results-20261011/layering-20261012/candidate-build.json create mode 100644 benchmarks/results-20261011/layering-20261012/debug-btree.log create mode 100644 benchmarks/results-20261011/layering-20261012/debug-build-config.e33dfc688a9c2bf2dad4e17ef758ec3e create mode 100644 benchmarks/results-20261011/layering-20261012/debug-filter.log create mode 100644 benchmarks/results-20261011/layering-20261012/debug-log.log create mode 100644 benchmarks/results-20261011/layering-20261012/debug-quick.log create mode 100644 benchmarks/results-20261011/layering-20261012/debug-splinter.log create mode 100644 benchmarks/results-20261011/layering-20261012/debug-superblock.log create mode 100644 benchmarks/results-20261011/layering-20261012/debug-trunk-async.log create mode 100644 benchmarks/results-20261011/layering-20261012/format_layering_compare.py create mode 100644 benchmarks/results-20261011/layering-20261012/manifest.json create mode 100644 benchmarks/results-20261011/layering-20261012/measured-source.patch create mode 100644 benchmarks/results-20261011/layering-20261012/release-build-config.fb6bc8eeb8390429253883c09491f5bd create mode 100644 benchmarks/results-20261011/layering-20261012/release-corruption.log create mode 100644 benchmarks/results-20261011/layering-20261012/results.csv create mode 100644 benchmarks/results-20261011/layering-20261012/run_job.py create mode 100644 benchmarks/results-20261011/layering-20261012/summary.csv create mode 100644 benchmarks/results-20261011/layering-20261012/summary.txt create mode 100644 benchmarks/results-20261011/layering-20261012/test_format_layering.py create mode 100644 benchmarks/results-20261011/layering-20261012/validation.json diff --git a/benchmarks/results-20261011/layering-20261012/.gitattributes b/benchmarks/results-20261011/layering-20261012/.gitattributes new file mode 100644 index 000000000..0350b5330 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/.gitattributes @@ -0,0 +1,3 @@ +# Preserve raw measurement files and the byte-identical source patch. +*.csv whitespace=cr-at-eol +*.patch -whitespace diff --git a/benchmarks/results-20261011/layering-20261012/0-before.log b/benchmarks/results-20261011/layering-20261012/0-before.log new file mode 100644 index 000000000..68285630b --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/0-before.log @@ -0,0 +1,29 @@ +kvbench version=before splinter=splinterdb_build_version db3047a8-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/0-before.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=0 sabotage=0 +open took 0.004 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9863487 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.05s ops/cpu-s=3757774 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=5 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 14272991 ops (+1427128 ops/s) + found=19400472 notfound=0 +== point:uniform@1 T=1 ops=19400472 secs= 15.00 ops/s= 1293343 lat us p50=0.71 p99=1.07 p99.9=5.18 max=128.1 mean=0.75 + cpu=15.00s ops/cpu-s=1293419 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 33697844 ops (+1942303 ops/s) + [point:uniform@8] progress 97769864 ops (+6406744 ops/s) + found=97294988 notfound=0 +== point:uniform@8 T=8 ops=97294988 secs= 15.00 ops/s= 6486242 lat us p50=1.17 p99=1.65 p99.9=2.21 max=98.1 mean=1.19 + cpu=119.98s ops/cpu-s=810906 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 1044899 ops (+1844514294851272192 ops/s) +== range:uniform:1@1 T=1 ops=1751557 secs= 15.00 ops/s= 116768 lat us p50=8.32 p99=12.16 p99.9=35.33 max=164.1 mean=8.54 + cpu=15.00s ops/cpu-s=116773 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.13 p99=11.65 mean=8.26 per-tuple=nan ns deinit mean=0.25 us tuples/s=116768 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1763139 ops (+71816 ops/s) + [range:uniform:1@8] progress 7435455 ops (+567192 ops/s) +close took 0.061 s +== range:uniform:1@8 T=8 ops=8517651 secs= 15.00 ops/s= 567835 lat us p50=12.93 p99=19.20 p99.9=29.44 max=454.0 mean=14.04 + cpu=119.98s ops/cpu-s=70990 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.42 p99=18.18 mean=13.40 per-tuple=nan ns deinit mean=0.61 us tuples/s=567835 + insert/run=15.00s drain(close)=0.06s total=15.06s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/1-layered.log b/benchmarks/results-20261011/layering-20261012/1-layered.log new file mode 100644 index 000000000..e87f00e97 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/1-layered.log @@ -0,0 +1,29 @@ +kvbench version=layered splinter=splinterdb_build_version 07dfa176-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/1-layered.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=0 sabotage=0 +open took 0.005 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9546301 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3430002 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=2 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 13408500 ops (+1340677 ops/s) + found=18294339 notfound=0 +== point:uniform@1 T=1 ops=18294339 secs= 15.00 ops/s= 1219599 lat us p50=0.78 p99=1.10 p99.9=4.29 max=138.0 mean=0.80 + cpu=15.00s ops/cpu-s=1219338 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 31405485 ops (+1799538 ops/s) + [point:uniform@8] progress 91937290 ops (+6052753 ops/s) + found=91477273 notfound=0 +== point:uniform@8 T=8 ops=91477273 secs= 15.00 ops/s= 6098403 lat us p50=1.23 p99=1.81 p99.9=2.34 max=460.4 mean=1.27 + cpu=119.97s ops/cpu-s=762488 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 1011488 ops (+1844478220911477760 ops/s) +== range:uniform:1@1 T=1 ops=1700424 secs= 15.00 ops/s= 113360 lat us p50=8.83 p99=10.62 p99.9=29.95 max=128.6 mean=8.79 + cpu=15.00s ops/cpu-s=113364 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.58 p99=10.37 mean=8.53 per-tuple=nan ns deinit mean=0.23 us tuples/s=113360 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1866184 ops (+85460 ops/s) + [range:uniform:1@8] progress 7593104 ops (+572651 ops/s) +close took 0.078 s +== range:uniform:1@8 T=8 ops=8636692 secs= 15.00 ops/s= 575771 lat us p50=12.93 p99=19.20 p99.9=27.90 max=804.7 mean=13.85 + cpu=119.97s ops/cpu-s=71988 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.42 p99=18.18 mean=13.25 per-tuple=nan ns deinit mean=0.56 us tuples/s=575771 + insert/run=15.00s drain(close)=0.08s total=15.08s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/2-layered.log b/benchmarks/results-20261011/layering-20261012/2-layered.log new file mode 100644 index 000000000..37af89ad7 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/2-layered.log @@ -0,0 +1,29 @@ +kvbench version=layered splinter=splinterdb_build_version 07dfa176-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/2-layered.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=0 sabotage=0 +open took 0.004 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9320557 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3346552 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 13680093 ops (+1367814 ops/s) + found=18684579 notfound=0 +== point:uniform@1 T=1 ops=18684579 secs= 15.00 ops/s= 1245618 lat us p50=0.74 p99=1.07 p99.9=4.29 max=53.5 mean=0.78 + cpu=15.00s ops/cpu-s=1245643 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 30613877 ops (+1693174 ops/s) + [point:uniform@8] progress 89317314 ops (+5869954 ops/s) + found=88865085 notfound=0 +== point:uniform@8 T=8 ops=88865085 secs= 15.00 ops/s= 5924258 lat us p50=1.26 p99=1.87 p99.9=2.66 max=124.6 mean=1.31 + cpu=119.98s ops/cpu-s=740641 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=1 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 1040554 ops (+1844398305915607040 ops/s) +== range:uniform:1@1 T=1 ops=1748496 secs= 15.00 ops/s= 116565 lat us p50=8.32 p99=12.67 p99.9=36.35 max=59.4 mean=8.55 + cpu=15.00s ops/cpu-s=116567 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.13 p99=12.16 mean=8.30 per-tuple=nan ns deinit mean=0.23 us tuples/s=116565 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1723657 ops (+68303 ops/s) + [range:uniform:1@8] progress 7407505 ops (+568346 ops/s) +close took 0.095 s +== range:uniform:1@8 T=8 ops=8487884 secs= 15.00 ops/s= 565851 lat us p50=12.93 p99=19.20 p99.9=28.42 max=484.7 mean=14.09 + cpu=119.98s ops/cpu-s=70743 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.42 p99=18.18 mean=13.48 per-tuple=nan ns deinit mean=0.57 us tuples/s=565851 + insert/run=15.00s drain(close)=0.09s total=15.10s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/3-before.log b/benchmarks/results-20261011/layering-20261012/3-before.log new file mode 100644 index 000000000..c9fa8a790 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/3-before.log @@ -0,0 +1,29 @@ +kvbench version=before splinter=splinterdb_build_version db3047a8-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/3-before.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=0 sabotage=0 +open took 0.005 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9802225 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3619582 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=4 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 13502389 ops (+1350030 ops/s) + found=18557383 notfound=0 +== point:uniform@1 T=1 ops=18557383 secs= 15.00 ops/s= 1237144 lat us p50=0.78 p99=1.10 p99.9=1.33 max=47.7 mean=0.78 + cpu=15.00s ops/cpu-s=1237172 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 30795643 ops (+1729168 ops/s) + [point:uniform@8] progress 90151142 ops (+5935128 ops/s) + found=89708915 notfound=0 +== point:uniform@8 T=8 ops=89708915 secs= 15.00 ops/s= 5980510 lat us p50=1.26 p99=1.84 p99.9=2.27 max=102.6 mean=1.30 + cpu=119.98s ops/cpu-s=747677 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 1032990 ops (+1844405605717911808 ops/s) +== range:uniform:1@1 T=1 ops=1718133 secs= 15.00 ops/s= 114541 lat us p50=8.58 p99=12.16 p99.9=34.30 max=166.9 mean=8.70 + cpu=15.00s ops/cpu-s=114544 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.32 p99=11.90 mean=8.43 per-tuple=nan ns deinit mean=0.24 us tuples/s=114541 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1799568 ops (+76648 ops/s) + [range:uniform:1@8] progress 7622617 ops (+582264 ops/s) +close took 0.074 s +== range:uniform:1@8 T=8 ops=8717825 secs= 15.00 ops/s= 581179 lat us p50=12.67 p99=18.69 p99.9=26.88 max=111.1 mean=13.72 + cpu=119.98s ops/cpu-s=72660 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.16 p99=17.66 mean=13.11 per-tuple=nan ns deinit mean=0.57 us tuples/s=581179 + insert/run=15.00s drain(close)=0.07s total=15.07s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/4-layered.log b/benchmarks/results-20261011/layering-20261012/4-layered.log new file mode 100644 index 000000000..4bac58186 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/4-layered.log @@ -0,0 +1,29 @@ +kvbench version=layered splinter=splinterdb_build_version 07dfa176-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/4-layered.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=1 sabotage=0 +open took 0.005 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9835214 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.05s ops/cpu-s=3687180 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=3 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 13122720 ops (+1312075 ops/s) + found=18082748 notfound=0 +== point:uniform@1 T=1 ops=18082748 secs= 15.00 ops/s= 1205501 lat us p50=0.79 p99=1.14 p99.9=1.46 max=49.8 mean=0.80 + cpu=15.00s ops/cpu-s=1205486 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 30601249 ops (+1747664 ops/s) + [point:uniform@8] progress 89539946 ops (+5893457 ops/s) + found=89027547 notfound=0 +== point:uniform@8 T=8 ops=89027547 secs= 15.00 ops/s= 5935084 lat us p50=1.26 p99=1.87 p99.9=2.27 max=104.3 mean=1.31 + cpu=119.98s ops/cpu-s=741999 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 1017343 ops (+1844480324502229504 ops/s) +== range:uniform:1@1 T=1 ops=1711833 secs= 15.00 ops/s= 114120 lat us p50=8.58 p99=11.90 p99.9=34.30 max=63.2 mean=8.74 + cpu=15.00s ops/cpu-s=114121 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.32 p99=11.39 mean=8.47 per-tuple=nan ns deinit mean=0.24 us tuples/s=114120 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1690929 ops (+67349 ops/s) + [range:uniform:1@8] progress 7183381 ops (+549206 ops/s) +close took 0.085 s +== range:uniform:1@8 T=8 ops=8249067 secs= 15.00 ops/s= 549930 lat us p50=13.18 p99=19.20 p99.9=27.90 max=97.8 mean=14.50 + cpu=119.99s ops/cpu-s=68750 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.67 p99=18.69 mean=13.87 per-tuple=nan ns deinit mean=0.59 us tuples/s=549930 + insert/run=15.00s drain(close)=0.08s total=15.08s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/5-before.log b/benchmarks/results-20261011/layering-20261012/5-before.log new file mode 100644 index 000000000..858ccdab1 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/5-before.log @@ -0,0 +1,29 @@ +kvbench version=before splinter=splinterdb_build_version db3047a8-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/5-before.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=1 sabotage=0 +open took 0.005 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9613963 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.05s ops/cpu-s=3732666 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=1 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 13070839 ops (+1306959 ops/s) + found=18009054 notfound=0 +== point:uniform@1 T=1 ops=18009054 secs= 15.00 ops/s= 1200582 lat us p50=0.79 p99=1.14 p99.9=1.42 max=146.3 mean=0.81 + cpu=15.00s ops/cpu-s=1200610 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=1 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 29591627 ops (+1651903 ops/s) + [point:uniform@8] progress 85538851 ops (+5594329 ops/s) + found=85184542 notfound=0 +== point:uniform@8 T=8 ops=85184542 secs= 15.00 ops/s= 5678884 lat us p50=1.33 p99=1.97 p99.9=3.62 max=447.2 mean=1.37 + cpu=119.93s ops/cpu-s=710295 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 1010139 ops (+1844416341239076864 ops/s) +== range:uniform:1@1 T=1 ops=1686147 secs= 15.00 ops/s= 112408 lat us p50=8.83 p99=11.39 p99.9=16.26 max=176.4 mean=8.87 + cpu=15.00s ops/cpu-s=112415 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.58 p99=11.14 mean=8.59 per-tuple=nan ns deinit mean=0.25 us tuples/s=112408 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1638629 ops (+62842 ops/s) + [range:uniform:1@8] progress 6982272 ops (+534326 ops/s) +close took 0.060 s +== range:uniform:1@8 T=8 ops=8008622 secs= 15.00 ops/s= 533900 lat us p50=13.70 p99=20.22 p99.9=29.95 max=143.3 mean=14.94 + cpu=119.99s ops/cpu-s=66745 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=13.18 p99=19.20 mean=14.27 per-tuple=nan ns deinit mean=0.63 us tuples/s=533900 + insert/run=15.00s drain(close)=0.06s total=15.06s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/6-before.log b/benchmarks/results-20261011/layering-20261012/6-before.log new file mode 100644 index 000000000..e3ce56678 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/6-before.log @@ -0,0 +1,29 @@ +kvbench version=before splinter=splinterdb_build_version db3047a8-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/6-before.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=1 sabotage=0 +open took 0.005 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9071147 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3270379 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 13434711 ops (+1343308 ops/s) + found=18476511 notfound=0 +== point:uniform@1 T=1 ops=18476511 secs= 15.00 ops/s= 1231746 lat us p50=0.78 p99=1.10 p99.9=1.39 max=46.5 mean=0.79 + cpu=15.00s ops/cpu-s=1231772 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 29235085 ops (+1579875 ops/s) + [point:uniform@8] progress 86061084 ops (+5682198 ops/s) + found=85698129 notfound=0 +== point:uniform@8 T=8 ops=85698129 secs= 15.00 ops/s= 5713124 lat us p50=1.33 p99=1.90 p99.9=2.40 max=8025.1 mean=1.36 + cpu=119.98s ops/cpu-s=714293 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=4 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 999559 ops (+1844391981889194752 ops/s) +== range:uniform:1@1 T=1 ops=1686461 secs= 15.00 ops/s= 112430 lat us p50=8.83 p99=10.37 p99.9=14.98 max=42.0 mean=8.87 + cpu=15.00s ops/cpu-s=112428 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.58 p99=10.11 mean=8.59 per-tuple=nan ns deinit mean=0.25 us tuples/s=112430 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1698937 ops (+69929 ops/s) + [range:uniform:1@8] progress 7232120 ops (+553279 ops/s) +close took 0.080 s +== range:uniform:1@8 T=8 ops=8282728 secs= 15.00 ops/s= 552174 lat us p50=13.44 p99=19.20 p99.9=27.90 max=456.7 mean=14.44 + cpu=119.98s ops/cpu-s=69035 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.67 p99=18.18 mean=13.80 per-tuple=nan ns deinit mean=0.60 us tuples/s=552174 + insert/run=15.00s drain(close)=0.08s total=15.08s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/7-layered.log b/benchmarks/results-20261011/layering-20261012/7-layered.log new file mode 100644 index 000000000..a7411e288 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/7-layered.log @@ -0,0 +1,29 @@ +kvbench version=layered splinter=splinterdb_build_version 07dfa176-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/7-layered.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=1 normal_bg=3 memtable_mib=4 secs=15.0 warmup=1.0 rep=1 sabotage=0 +open took 0.005 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 9187921 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3538070 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=6 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + [point:uniform@1] progress 13298972 ops (+1329757 ops/s) + found=18331364 notfound=0 +== point:uniform@1 T=1 ops=18331364 secs= 15.00 ops/s= 1222070 lat us p50=0.78 p99=1.14 p99.9=1.49 max=49.1 mean=0.79 + cpu=15.00s ops/cpu-s=1222075 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=11 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 29309682 ops (+1600901 ops/s) + [point:uniform@8] progress 85671830 ops (+5635833 ops/s) + found=85169199 notfound=0 +== point:uniform@8 T=8 ops=85169199 secs= 15.00 ops/s= 5677861 lat us p50=1.33 p99=1.94 p99.9=2.98 max=1576.7 mean=1.37 + cpu=119.98s ops/cpu-s=709873 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 998554 ops (+1844450794817042688 ops/s) +== range:uniform:1@1 T=1 ops=1684540 secs= 15.00 ops/s= 112301 lat us p50=8.83 p99=10.37 p99.9=15.49 max=66.9 mean=8.87 + cpu=15.00s ops/cpu-s=112300 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.58 p99=10.11 mean=8.61 per-tuple=nan ns deinit mean=0.24 us tuples/s=112301 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1785661 ops (+78701 ops/s) + [range:uniform:1@8] progress 7515057 ops (+572900 ops/s) +close took 0.072 s +== range:uniform:1@8 T=8 ops=8591609 secs= 15.00 ops/s= 572766 lat us p50=12.67 p99=19.20 p99.9=27.39 max=463.6 mean=13.92 + cpu=119.98s ops/cpu-s=71610 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.16 p99=18.18 mean=13.32 per-tuple=nan ns deinit mean=0.56 us tuples/s=572766 + insert/run=15.00s drain(close)=0.07s total=15.07s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261011/layering-20261012/before-preservation.json b/benchmarks/results-20261011/layering-20261012/before-preservation.json new file mode 100644 index 000000000..e5b0e086d --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/before-preservation.json @@ -0,0 +1,33 @@ +{ + "preserved_at": 1791841463.654878, + "source_head": "07dfa1761182214f18b8269c7b24cc67bd35c0f4", + "files": { + "kvbench": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/bin/kvbench", + "sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "bytes": 4016432 + }, + "kvbench.json": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/bin/kvbench.json", + "sha256": "274e705c37837c0bebd61d49ce0df7f3b568f9ed2ee73574dc34e991a4b8ba38", + "bytes": 978 + }, + "libsplinterdb.a": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/lib/libsplinterdb.a", + "sha256": "77bc8b4ddb7f93ae4308de9c9e3506019f371bd9c3badd2ba5d2772e1ba637f5", + "bytes": 5611354 + }, + "libsplinterdb.so": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/lib/libsplinterdb.so", + "sha256": "dcf51bf2c4071e088e8a144b9a3f2f36f69c7049b24cea573a98738e31ce0ebe", + "bytes": 4099960 + }, + "kvbench.c": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/benchmarks/kvbench.c", + "sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "bytes": 49348 + } + }, + "note": "Existing final scalar reader, before layering fix. kvbench embeds libsplinterdb.a; preserved .so is not loaded by kvbench.", + "dynamic_section": "\nDynamic section at offset 0xb3da0 contains 29 entries:\n Tag Type Name/Value\n 0x0000000000000001 (NEEDED) Shared library: [libaio.so.1t64]\n 0x0000000000000001 (NEEDED) Shared library: [libxxhash.so.0]\n 0x0000000000000001 (NEEDED) Shared library: [libm.so.6]\n 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]\n 0x000000000000000c (INIT) 0x3000\n 0x000000000000000d (FINI) 0x996fc\n 0x0000000000000019 (INIT_ARRAY) 0xb4b60\n 0x000000000000001b (INIT_ARRAYSZ) 24 (bytes)\n 0x000000000000001a (FINI_ARRAY) 0xb4b78\n 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)\n 0x000000006ffffef5 (GNU_HASH) 0x3e8\n 0x0000000000000005 (STRTAB) 0xed8\n 0x0000000000000006 (SYMTAB) 0x410\n 0x000000000000000a (STRSZ) 1570 (bytes)\n 0x000000000000000b (SYMENT) 24 (bytes)\n 0x0000000000000015 (DEBUG) 0x0\n 0x0000000000000003 (PLTGOT) 0xb4fe8\n 0x0000000000000002 (PLTRELSZ) 2568 (bytes)\n 0x0000000000000014 (PLTREL) RELA\n 0x0000000000000017 (JMPREL) 0x2408\n 0x0000000000000007 (RELA) 0x1700\n 0x0000000000000008 (RELASZ) 3336 (bytes)\n 0x0000000000000009 (RELAENT) 24 (bytes)\n 0x000000006ffffffb (FLAGS_1) Flags: PIE\n 0x000000006ffffffe (VERNEED) 0x15e0\n 0x000000006fffffff (VERNEEDNUM) 3\n 0x000000006ffffff0 (VERSYM) 0x14fa\n 0x000000006ffffff9 (RELACOUNT) 132\n 0x0000000000000000 (NULL) 0x0\n" +} diff --git a/benchmarks/results-20261011/layering-20261012/block-summary.csv b/benchmarks/results-20261011/layering-20261012/block-summary.csv new file mode 100644 index 000000000..b128a871c --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/block-summary.csv @@ -0,0 +1,13 @@ +phase,block,before_ops_s,layered_ops_s,delta_pct,before_min,before_max,layered_min,layered_max,before_cpu_us_op,layered_cpu_us_op,cpu_change_pct,ranges_overlap +point:uniform@1,all,1240703.65,1223196.8,-1.411042032478893,1200582.3,1293343.0,1205501.3,1245617.6,0.8065462778214102,0.8176837849638405,1.3808887907231115,True +point:uniform@1,0,1265243.3,1232608.1,-2.579361613691211,1237143.6,1293343.0,1219598.6,1245617.6,0.7907145737793351,0.8114727433095376,2.625241802612255,True +point:uniform@1,1,1216164.0,1213785.5,-0.1955739521972366,1200582.3,1231745.7,1205501.3,1222069.7,0.8223779818634853,0.8238948266181434,0.18444617780510875,True +point:uniform@8,all,5964689.9,5908901.425,-0.9353122448159557,5678883.7,6486241.6,5677861.3,6098402.6,1.3446285441466928,1.3545212332464787,0.7357191056853507,True +point:uniform@8,0,6233375.85,6011330.15,-3.562206183989358,5980510.1,6486241.6,5924257.7,6098402.6,1.2853346109388355,1.3308383607776584,3.540225980967393,True +point:uniform@8,1,5696003.95,5806472.699999999,1.9394078896310951,5678883.7,5713124.2,5677861.3,5935084.1,1.4039224773545498,1.3782041057152994,-1.8318940008505535,True +range:uniform:1@1,all,114036.7,114086.35,0.04353861520021152,112408.4,116768.2,112300.7,116564.9,8.771002412054917,8.766793940595583,-0.04798164749731937,True +range:uniform:1@1,0,115654.4,114962.25,-0.598464044601843,114540.6,116768.2,113359.6,116564.9,8.647108006295996,8.700065353479115,0.6124284228271604,True +range:uniform:1@1,1,112419.0,113210.45,0.7040180040740474,112408.4,112429.6,112300.7,114120.2,8.89489681781384,8.833522527712052,-0.6899944019460036,True +range:uniform:1@8,all,558772.1,566079.5750000001,1.3077737775383014,533900.3,581179.0,549929.8,575771.3,14.329259098129281,14.134208517478168,-1.3612049256376269,True +range:uniform:1@8,0,574507.05,570811.3500000001,-0.643281923172212,567835.1,581179.0,565851.4,575771.3,13.924617299443824,14.01349625859562,0.6382865484952704,True +range:uniform:1@8,1,543037.15,561347.8,3.3718963794650225,533900.3,552174.0,549929.8,572765.8,14.733900896814738,14.254920776360718,-3.250871061292182,True diff --git a/benchmarks/results-20261011/layering-20261012/block-summary.txt b/benchmarks/results-20261011/layering-20261012/block-summary.txt new file mode 100644 index 000000000..376599d20 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/block-summary.txt @@ -0,0 +1,21 @@ +phase | block | before mean [min,max] | layered mean [min,max] | delta | CPU before/layered us/op | ranges overlap +point:uniform@1 | all | 1240704 [1200582,1293343] | 1223197 [1205501,1245618] | -1.41% | 0.8065/0.8177 | True +point:uniform@1 | 0 | 1265243 [1237144,1293343] | 1232608 [1219599,1245618] | -2.58% | 0.7907/0.8115 | True +point:uniform@1 | 1 | 1216164 [1200582,1231746] | 1213786 [1205501,1222070] | -0.20% | 0.8224/0.8239 | True + before rates in order: 0=1293343, 3=1237144, 5=1200582, 6=1231746 + before first→last drift -4.76%; before block0→block1 mean drift -3.88% +point:uniform@8 | all | 5964690 [5678884,6486242] | 5908901 [5677861,6098403] | -0.94% | 1.3446/1.3545 | True +point:uniform@8 | 0 | 6233376 [5980510,6486242] | 6011330 [5924258,6098403] | -3.56% | 1.2853/1.3308 | True +point:uniform@8 | 1 | 5696004 [5678884,5713124] | 5806473 [5677861,5935084] | +1.94% | 1.4039/1.3782 | True + before rates in order: 0=6486242, 3=5980510, 5=5678884, 6=5713124 + before first→last drift -11.92%; before block0→block1 mean drift -8.62% +range:uniform:1@1 | all | 114037 [112408,116768] | 114086 [112301,116565] | +0.04% | 8.7710/8.7668 | True +range:uniform:1@1 | 0 | 115654 [114541,116768] | 114962 [113360,116565] | -0.60% | 8.6471/8.7001 | True +range:uniform:1@1 | 1 | 112419 [112408,112430] | 113210 [112301,114120] | +0.70% | 8.8949/8.8335 | True + before rates in order: 0=116768, 3=114541, 5=112408, 6=112430 + before first→last drift -3.72%; before block0→block1 mean drift -2.80% +range:uniform:1@8 | all | 558772 [533900,581179] | 566080 [549930,575771] | +1.31% | 14.3293/14.1342 | True +range:uniform:1@8 | 0 | 574507 [567835,581179] | 570811 [565851,575771] | -0.64% | 13.9246/14.0135 | True +range:uniform:1@8 | 1 | 543037 [533900,552174] | 561348 [549930,572766] | +3.37% | 14.7339/14.2549 | True + before rates in order: 0=567835, 3=581179, 5=533900, 6=552174 + before first→last drift -2.76%; before block0→block1 mean drift -5.48% diff --git a/benchmarks/results-20261011/layering-20261012/build_kvbench.py b/benchmarks/results-20261011/layering-20261012/build_kvbench.py new file mode 100644 index 000000000..aa5cc0e15 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/build_kvbench.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python3 +import argparse +import hashlib +import json +from pathlib import Path +import subprocess + +p = argparse.ArgumentParser() +p.add_argument('tree', type=Path) +p.add_argument('--build-root', default='build') +a = p.parse_args() +tree = a.tree.resolve() +root = Path(__file__).resolve().parent.parent +source = root / 'baseline/benchmarks/kvbench.c' +binary = tree / a.build_root / 'release/bin/kvbench' +binary.parent.mkdir(parents=True, exist_ok=True) +command = ['cc', '-D_GNU_SOURCE', '-ggdb', '-Wall', '-pthread', '-Werror', + '-DXXH_STATIC_LINKING_ONLY', '-fPIC', '-DSPLINTERDB_PLATFORM_DIR=platform_linux', + '-O3', '-ffast-math', '-flto', '-march=native', '-DKVBENCH_CLOSE_HAS_FORCE', + '-I', str(tree / 'include'), str(source), + str(tree / a.build_root / 'release/lib/libsplinterdb.a'), '-laio', '-lxxhash', '-lm', + '-lpthread', '-o', str(binary)] +subprocess.run(command, check=True) +provenance = dict(command=command, source_sha256=hashlib.sha256(source.read_bytes()).hexdigest(), + binary_sha256=hashlib.sha256(binary.read_bytes()).hexdigest()) +binary.with_suffix('.json').write_text(json.dumps(provenance, indent=2) + '\n') +print(binary) diff --git a/benchmarks/results-20261011/layering-20261012/candidate-build.json b/benchmarks/results-20261011/layering-20261012/candidate-build.json new file mode 100644 index 000000000..aa25dcc5e --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/candidate-build.json @@ -0,0 +1,30 @@ +{ + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/include", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/benchmarks/kvbench.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/bin/kvbench" + ], + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "binary_sha256": "74e89bee01a9d0bece7bff6045090bdd71195de95346a4da6d20dff1b7effc45" +} diff --git a/benchmarks/results-20261011/layering-20261012/debug-btree.log b/benchmarks/results-20261011/layering-20261012/debug-btree.log new file mode 100644 index 000000000..0c676168a --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-btree.log @@ -0,0 +1,35 @@ +Running 1 CTests, suite name 'btree', test case 'all'. +TEST 1/11 btree:test_leaf_hdr [OK] +TEST 2/11 btree:test_leaf_hdr_search [OK] +TEST 3/11 btree:test_index_hdr [OK] +TEST 4/11 btree:test_index_hdr_search [OK] +TEST 5/11 btree:test_leaf_split [OK] +TEST 6/11 btree:test_defragmented_leaf_bytes [OK] +TEST 7/11 btree:test_defragmented_index_bytes [OK] +TEST 8/11 btree:test_invalid_branch_point_lookup Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 9/11 btree:test_invalid_branch_async_point_lookup Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 10/11 btree:test_invalid_branch_iterator Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 11/11 btree:test_invalid_branch_metadata_queries Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +RESULTS: 11 tests (11 ok, 0 failed, 0 skipped) ran in 12 ms diff --git a/benchmarks/results-20261011/layering-20261012/debug-build-config.e33dfc688a9c2bf2dad4e17ef758ec3e b/benchmarks/results-20261011/layering-20261012/debug-build-config.e33dfc688a9c2bf2dad4e17ef758ec3e new file mode 100644 index 000000000..d3474c03c --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-build-config.e33dfc688a9c2bf2dad4e17ef758ec3e @@ -0,0 +1,14 @@ +BUILD_ROOT=build-layering +MAKEFLAGS= -j8 --jobserver-auth=fifo:/tmp/splinter-disk-format-layering-debug-build-9wyktd8f/GMfifo7 -- BUILD_MODE=debug BUILD_ROOT=build-layering +BUILD_MODE=debug +QT_ACCESSIBILITY=1 +CC = cc +DEPFLAGS = -MMD -MP +CFLAGS = -D_GNU_SOURCE -ggdb -Wall -pthread -Wfatal-errors -Werror -Wvla -DXXH_STATIC_LINKING_ONLY -fPIC -DSPLINTERDB_PLATFORM_DIR=platform_linux -DSPLINTER_DEBUG -DPLATFORM_MEMORY_FAULT_INJECTION=0 +TEST_MEMORY_FAULT_DISABLED_CFLAGS = -DPLATFORM_MEMORY_FAULT_INJECTION_DISABLED_IN_THIS_FILE +INCLUDE = -I include -I src -I src/platform_linux -I tests +TARGET_ARCH = +LD = cc +LDFLAGS = -ggdb -pthread +LIBS = -lm -lpthread -laio -lxxhash +AR = ar diff --git a/benchmarks/results-20261011/layering-20261012/debug-filter.log b/benchmarks/results-20261011/layering-20261012/debug-filter.log new file mode 100644 index 000000000..e9e322630 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-filter.log @@ -0,0 +1,65 @@ +build-layering/debug/bin/driver_test: splinterdb_build_version 07dfa176-dirty +Dispatch test filter_test +filter_test: routing filter index expansion test started +filter_test: mixed routing filter fingerprint sizes test started +filter_test: routing filter arithmetic limits test started +filter_test: malicious routing/mini metadata test started +Malformed mini allocator metadata: unsupported format version (addr=4980736). +Malformed mini allocator metadata: metadata page number disagrees with predecessor link (addr=4980736). +Malformed mini allocator metadata: metadata page number is not contiguous (addr=4984832). +Malformed mini allocator metadata: metadata page number is not contiguous (addr=4980736). +Malformed mini allocator metadata: metadata entry position is invalid (addr=4980736). +Malformed mini allocator metadata: metadata entry position is invalid (addr=4980736). +Malformed mini allocator metadata: metadata extent entry is invalid (addr=4980736). +filter_test: routing filter basic test started +num input keys 1048575 estimate 1048653 num_unique 1040503 +num input keys 1572862 estimate 1573255 num_unique 1554958 +num input keys 2097150 estimate 2097684 num_unique 2065239 +num input keys 2534056 estimate 2534923 num_unique 2487644 +num input keys 3075820 estimate 3077087 num_unique 3007608 +num input keys 3460297 estimate 3461800 num_unique 3374028 +num input keys 4014546 estimate 4016523 num_unique 3898690 +num input keys 4392783 estimate 4395331 num_unique 4254486 +across filters: num input keys 4392783 estimate 4408343 +routing filter basic test: false positive rate 0.0625 +filter_test: routing filter basic test started +num input keys 100 estimate 100 num_unique 100 +num input keys 150 estimate 150 num_unique 150 +num input keys 199 estimate 199 num_unique 199 +num input keys 241 estimate 241 num_unique 241 +num input keys 292 estimate 292 num_unique 292 +num input keys 328 estimate 328 num_unique 328 +num input keys 379 estimate 379 num_unique 379 +num input keys 415 estimate 415 num_unique 415 +across filters: num input keys 415 estimate 0 +routing filter basic test: false positive rate 0.0000 +filter_test: routing filter basic test started +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +across filters: num input keys 1 estimate 0 +routing filter basic test: false positive rate 0.0000 +filter_test: routing filter basic test started +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +across filters: num input keys 1 estimate 0 +routing filter basic test: false positive rate 0.0000 diff --git a/benchmarks/results-20261011/layering-20261012/debug-log.log b/benchmarks/results-20261011/layering-20261012/debug-log.log new file mode 100644 index 000000000..8dfa377c1 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-log.log @@ -0,0 +1,11 @@ +build-layering/debug/bin/driver_test: splinterdb_build_version 07dfa176-dirty +Dispatch test log_test + +Started log_test!! +Malformed log page sequence at page 1843200: group=3 page=0 follows group=1 page=0 (terminated). +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +shard_log_iterator_init: record validation failed in group 1 at log page 71172096: Invalid argument; discarding this group and the rest +Malformed log page sequence at page 71958528: group=1 page=0 follows group=1 page=0. +shard_log_iterator_init: record validation failed in group 2 at log page 72749056: Input/output error; discarding this group and the rest +log returned 500000 of 500000 entries diff --git a/benchmarks/results-20261011/layering-20261012/debug-quick.log b/benchmarks/results-20261011/layering-20261012/debug-quick.log new file mode 100644 index 000000000..dc181c248 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-quick.log @@ -0,0 +1,51 @@ +Running 1 CTests, suite name 'splinterdb_quick', test case 'all'. +TEST 1/47 splinterdb_quick:test_get_config_returns_effective_config [OK] +TEST 2/47 splinterdb_quick:test_get_cost_stats [OK] +TEST 3/47 splinterdb_quick:test_checkpoint_log_grace_default [OK] +TEST 4/47 splinterdb_quick:test_durable_barrier [OK] +TEST 5/47 splinterdb_quick:test_logged_oversized_value_failure_preserves_old_value [OK] +TEST 6/47 splinterdb_quick:test_basic_flow [OK] +TEST 7/47 splinterdb_quick:test_pthread_auto_registration [OK] +TEST 8/47 splinterdb_quick:test_result_auto_registration_with_stats [OK] +TEST 9/47 splinterdb_quick:test_stats_reset_clears_cache_io [OK] +TEST 10/47 splinterdb_quick:test_final_merge_error_propagation [OK] +TEST 11/47 splinterdb_quick:test_fetch_and_update_catches_up_after_rotation [OK] +TEST 12/47 splinterdb_quick:test_fetch_and_update_rejects_incorporated_boundary [OK] +TEST 13/47 splinterdb_quick:test_range_init_error_cleanup_preserves_branches setup_ordered_iterators: exception: Cannot allocate memory +core_apply_to_range: range iterator init failed: Cannot allocate memory +[OK] +TEST 14/47 splinterdb_quick:stats_reset_excludes_active_operations [OK] +TEST 15/47 splinterdb_quick:test_apis_for_large_key [OK] +TEST 16/47 splinterdb_quick:test_key_size_gt_max_inline_key_size [OK] +TEST 17/47 splinterdb_quick:test_value_size_gt_max_value_size [OK] +TEST 18/47 splinterdb_quick:test_large_lookup_survives_memtable_recycle [OK] +TEST 19/47 splinterdb_quick:test_iterator_survives_memtable_recycle [OK] +TEST 20/47 splinterdb_quick:test_variable_length_values [OK] +TEST 21/47 splinterdb_quick:test_basic_iterator [OK] +TEST 22/47 splinterdb_quick:test_iterator_delete_current [OK] +TEST 23/47 splinterdb_quick:test_empty_iterator [OK] +TEST 24/47 splinterdb_quick:test_bounded_iterator [OK] +TEST 25/47 splinterdb_quick:test_bounded_iterator_start_normalization [OK] +TEST 26/47 splinterdb_quick:test_splinterdb_iterator_with_startkey [OK] +TEST 27/47 splinterdb_quick:test_splinterdb_iterator_with_non_existent_startkey [OK] +TEST 28/47 splinterdb_quick:test_splinterdb_iterator_with_missing_startkey_in_sequence [OK] +TEST 29/47 splinterdb_quick:test_iterator_prev_and_next [OK] +TEST 30/47 splinterdb_quick:test_iterator_reversal_on_key_in_two_branches [OK] +TEST 31/47 splinterdb_quick:test_iterator_reversal_with_three_branches [OK] +TEST 32/47 splinterdb_quick:test_rewriting_keys_does_not_grow_trunk [OK] +TEST 33/47 splinterdb_quick:test_close_and_reopen [OK] +TEST 34/47 splinterdb_quick:test_logged_close_and_reopen [OK] +TEST 35/47 splinterdb_quick:test_repeated_insert_close_reopen [OK] +TEST 36/47 splinterdb_quick:test_custom_data_config [OK] +TEST 37/47 splinterdb_quick:test_existence_only_memtable_lookup [OK] +TEST 38/47 splinterdb_quick:test_existence_only_trunk_lookup_skips_branches [OK] +TEST 39/47 splinterdb_quick:test_write_api_returns_old_value [OK] +TEST 40/47 splinterdb_quick:test_write_api_old_result_existence_only [OK] +TEST 41/47 splinterdb_quick:test_write_api_old_result_custom_merge_semantics [OK] +TEST 42/47 splinterdb_quick:test_write_api_old_result_merges_memtable_and_trunk [OK] +TEST 43/47 splinterdb_quick:test_write_api_old_result_respects_trunk_delete_shadow [OK] +TEST 44/47 splinterdb_quick:test_iterator_custom_comparator [OK] +TEST 45/47 splinterdb_quick:test_iterator_init_bug [OK] +TEST 46/47 splinterdb_quick:test_splinterdb_create_w_background_threads [OK] +TEST 47/47 splinterdb_quick:test_splinterdb_create_w_all_background_threads [OK] +RESULTS: 47 tests (47 ok, 0 failed, 0 skipped) ran in 9234 ms diff --git a/benchmarks/results-20261011/layering-20261012/debug-splinter.log b/benchmarks/results-20261011/layering-20261012/debug-splinter.log new file mode 100644 index 000000000..c187a3795 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-splinter.log @@ -0,0 +1,76 @@ +Running 1 CTests, suite name 'splinter', test case 'all'. +TEST 1/33 splinter:test_blob_checksums [OK] +TEST 2/33 splinter:test_inserts [OK] +TEST 3/33 splinter:test_btree_pack_abort_releases_reserved_extents btree_pack error status: 5 +[OK] +TEST 4/33 splinter:test_btree_pack_rejects_counter_overflow btree_pack: output statistics exceed the on-disk limit +btree_pack error status: 28 +[OK] +TEST 5/33 splinter:test_malformed_trunk_pivot_bundle [OK] +TEST 6/33 splinter:test_invalid_branch_reference_rejected Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +[OK] +TEST 7/33 splinter:test_malformed_trunk_pivot_stats trunk_merge_lookup: ondisk_node_find_pivot failed: 22 +trunk_merge_lookup_async: ondisk_node_find_pivot_async failed: 22 +[OK] +TEST 8/33 splinter:test_malformed_trunk_child_height trunk_merge_lookup_async: ondisk_node_handle_init_async failed: 22 +[OK] +TEST 9/33 splinter:test_two_log_checkpoint [OK] +TEST 10/33 splinter:test_reopen_with_changed_filter_config [OK] +TEST 11/33 splinter:test_logged_large_insert_shares_blob_extents [OK] +TEST 12/33 splinter:test_durable_barrier_waits_for_visible_insert_log_write [OK] +TEST 13/33 splinter:test_durable_barrier_packs_concurrent_small_tails [OK] +TEST 14/33 splinter:test_shard_log_first_fit_decreasing_packing [OK] +TEST 15/33 splinter:test_shard_log_page_alloc_failure_retries_open_buffer shard_log_graduate_buffer: out of log space +shard_log_graduate_group: failed to terminate group 1: No space left on device +[OK] +TEST 16/33 splinter:test_durable_barrier_without_log_publishes_root [OK] +TEST 17/33 splinter:test_checkpoint_reports_terminal_incorporation_failure trunk_node_serialize():2757: cache_alloc() failedserialize_nodes():2930: node_serialize() failedserialize_nodes_and_save_contingent_compactions: serialize_nodes failed: 12 +restore_balance_leaf():6088: serialize_nodes() failed: Cannot allocate memorytrunk_flush_prepare: flush_then_compact failed: 12 +trunk_incorporate_prepare failed: Cannot allocate memory +Memtable incorporation failed: generation=0 state=INCORPORATION_ASSIGNED status=Cannot allocate memory memtable_root=262144 +memtable flush failed: Cannot allocate memory +Shutdown found memtable from failed incorporation: generation=0 state=INCORPORATION_FAILED status=Cannot allocate memory memtable_root=262144 compacted_root=0 +[OK] +TEST 18/33 splinter:test_durable_barrier_reopens_writers_before_device_barrier [OK] +TEST 19/33 splinter:test_concurrent_durable_barriers_coalesce [OK] +TEST 20/33 splinter:test_durable_barrier_waits_for_checkpoint_publication [OK] +TEST 21/33 splinter:test_log_prepare_failure_leaves_update_unapplied core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_unmount: failed to publish the unmount root: Input/output error +core_unmount: shutdown retained recovery state; the next mount must recover +[OK] +TEST 22/33 splinter:test_checkpoint_advance_retries_cut_publish_failure core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +[OK] +TEST 23/33 splinter:test_checkpoint_advance_retries_completion_failure core_checkpoint_advance: completion publish failed: Input/output error +core_checkpoint_advance: completion publish failed: Input/output error +[OK] +TEST 24/33 splinter:test_self_managed_checkpoints_reclaim_log_space [OK] +TEST 25/33 splinter:test_auto_checkpoint [OK] +TEST 26/33 splinter:test_auto_checkpoint_grace_allows_natural_rotation [OK] +TEST 27/33 splinter:test_auto_checkpoint_grace_forces_overwrite_rotation [OK] +TEST 28/33 splinter:test_auto_checkpoint_on_overwrites [OK] +TEST 29/33 splinter:test_recover_allocations_reproduces_persisted_map [OK] +TEST 30/33 splinter:test_unmount_skips_allocator_map_that_needs_rebuild core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 31/33 splinter:test_mount_active_checkpoint_stays_mountable [OK] +TEST 32/33 splinter:test_lookups [OK] +TEST 33/33 splinter:test_splinter_print_diags [OK] +RESULTS: 33 tests (33 ok, 0 failed, 0 skipped) ran in 72025 ms diff --git a/benchmarks/results-20261011/layering-20261012/debug-superblock.log b/benchmarks/results-20261011/layering-20261012/debug-superblock.log new file mode 100644 index 000000000..d505ee541 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-superblock.log @@ -0,0 +1,43 @@ +Running 1 CTests, suite name 'superblock', test case 'all'. +TEST 1/16 superblock:test_format_sets_fresh_state [OK] +TEST 2/16 superblock:test_snapshot_persists_state [OK] +TEST 3/16 superblock:test_two_log_checkpoint_transitions [OK] +TEST 4/16 superblock:test_snapshot_preserves_unincorporated_sealed_log [OK] +TEST 5/16 superblock:test_two_log_checkpoint_torn_begin [OK] +TEST 6/16 superblock:test_torn_write_falls_back_to_older_generation [OK] +TEST 7/16 superblock:test_generation_wrap_selects_newest_slot [OK] +TEST 8/16 superblock:test_both_slots_corrupt_is_not_found superblock_mount: no valid superblock found +[OK] +TEST 9/16 superblock:test_mount_rejects_invalid_metadata superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +[OK] +TEST 10/16 superblock:test_slot_read_error_falls_back_to_peer [OK] +TEST 11/16 superblock:test_both_slot_read_errors_are_reported superblock_mount: no readable valid superblock found: Input/output error +[OK] +TEST 12/16 superblock:test_read_geometry superblock_read_bootstrap: no valid superblock found +[OK] +TEST 13/16 superblock:test_read_bootstrap_allocation_state superblock_read_bootstrap: no valid superblock found +[OK] +TEST 14/16 superblock:test_bootstrap_geometry_metadata_budget Device geometry needs 134217728 bytes of cache lookup metadata, which exceeds the configured cache budget of 67108864 bytes. +Failed to open existing SplinterDB device 'db' with specified configuration: No space left on device +[OK] +TEST 15/16 superblock:test_unknown_structure_version_rejected Unsupported trunk format version 2; this build requires version 1. +Unsupported trunk format version 2; this build requires version 1. +Unsupported allocator format version 2; this build requires version 1. +Unsupported allocator format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +[OK] +TEST 16/16 superblock:test_unsupported_superblock_version_rejected Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +[OK] +RESULTS: 16 tests (16 ok, 0 failed, 0 skipped) ran in 36 ms diff --git a/benchmarks/results-20261011/layering-20261012/debug-trunk-async.log b/benchmarks/results-20261011/layering-20261012/debug-trunk-async.log new file mode 100644 index 000000000..07065870f --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/debug-trunk-async.log @@ -0,0 +1,3 @@ +Running 1 CTests, suite name 'trunk_async', test case 'all'. +TEST 1/1 trunk_async:pivot_header_survives_cross_thread_yield [OK] +RESULTS: 1 tests (1 ok, 0 failed, 0 skipped) ran in 0 ms diff --git a/benchmarks/results-20261011/layering-20261012/format_layering_compare.py b/benchmarks/results-20261011/layering-20261012/format_layering_compare.py new file mode 100644 index 000000000..955e8c6e3 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/format_layering_compare.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +"""Measure disk-format layering changes on the same format-8 fixture. + +Invoke through run_job.py benchmark mode; it supplies the campaign-wide +benchmark/machine locks and verifies the database directory is on real disk. +The runner never changes the source fixture. Each private copy is fsynced +before O_DIRECT opening and is retained with its successful or failed run. +""" +import argparse +import csv +import hashlib +import json +import os +from pathlib import Path +import shutil +import statistics +import subprocess +import time + + +ROOT = Path(__file__).resolve().parent.parent +TREE = ROOT / 'worktrees/disk-format' +BEFORE = ROOT / 'coordination/disk-format-layering-before' +PHASES = ['warm', 'point:uniform@1', 'point:uniform@8', + 'range:uniform:1@1', 'range:uniform:1@8'] + + +def digest(path): + value = hashlib.sha256() + with Path(path).open('rb') as source: + for block in iter(lambda: source.read(1024 * 1024), b''): + value.update(block) + return value.hexdigest() + + +def save_json(path, value): + path.write_text(json.dumps(value, indent=2) + '\n') + + +def csv_rows(path): + with path.open() as source: + return list(csv.DictReader(source)) + + +def save_csv(path, rows): + with path.open('w') as target: + writer = csv.DictWriter(target, fieldnames=list(rows[0])) + writer.writeheader() + writer.writerows(rows) + + +def provenance(candidate_build): + prior = json.loads((BEFORE / 'preservation.json').read_text()) + for name, record in prior['files'].items(): + assert digest(BEFORE / name) == record['sha256'], name + binary = TREE / candidate_build / 'release/bin/kvbench' + library = TREE / candidate_build / 'release/lib/libsplinterdb.a' + build = json.loads(binary.with_suffix('.json').read_text()) + assert build['binary_sha256'] == digest(binary), 'candidate binary identity' + assert build['source_sha256'] == digest(BEFORE / 'kvbench.c'), 'benchmark source changed' + diff = subprocess.check_output(['git', 'diff', 'HEAD', '--'], cwd=TREE) + candidate = dict(build_record=build, binary=str(binary), binary_sha256=digest(binary), + library=str(library), library_sha256=digest(library), + source_head=subprocess.check_output(['git', 'rev-parse', 'HEAD'], + cwd=TREE, text=True).strip(), + tracked_diff_sha256=hashlib.sha256(diff).hexdigest(), + tracked_diff_bytes=len(diff)) + return {'before': BEFORE / 'kvbench', 'layered': binary}, { + 'before': prior, 'layered': candidate}, diff + + +def summarize(out, rows, rounds): + result = [] + lines = ['phase | before ops/s | layered ops/s | throughput change | CPU us/op change'] + for phase in PHASES[1:]: + groups = {label: [row for row in rows if row['version'] == label + and row['phase'] == phase] for label in ['before', 'layered']} + assert all(len(group) == 2 * rounds for group in groups.values()), (phase, groups) + rates = {label: [float(row['ops_s']) for row in group] + for label, group in groups.items()} + mean = lambda label, key: statistics.fmean(float(row[key]) for row in groups[label]) + cpu = {label: statistics.fmean(1e6 * float(row['cpu_s']) / int(row['ops']) + for row in group) for label, group in groups.items()} + before = statistics.fmean(rates['before']) + layered = statistics.fmean(rates['layered']) + record = dict(phase=phase, threads=int(phase.rsplit('@', 1)[1]), samples_per_variant=2 * rounds, + before_ops_s=before, layered_ops_s=layered, + throughput_change_pct=100 * (layered / before - 1), + before_min_ops_s=min(rates['before']), before_max_ops_s=max(rates['before']), + layered_min_ops_s=min(rates['layered']), layered_max_ops_s=max(rates['layered']), + before_cpu_us_op=cpu['before'], layered_cpu_us_op=cpu['layered'], + cpu_us_op_change_pct=100 * (cpu['layered'] / cpu['before'] - 1), + before_read_B_op=mean('before', 'rd_B_op'), + layered_read_B_op=mean('layered', 'rd_B_op'), + before_write_B_op=mean('before', 'wr_B_op'), + layered_write_B_op=mean('layered', 'wr_B_op'), + before_p99_us=mean('before', 'p99_us'), layered_p99_us=mean('layered', 'p99_us')) + result.append(record) + lines.append(f'{phase} | {before:.0f} | {layered:.0f} | ' + f'{record["throughput_change_pct"]:+.2f}% | ' + f'{record["cpu_us_op_change_pct"]:+.2f}%') + save_csv(out / 'summary.csv', result) + report = '\n'.join(lines) + '\n' + (out / 'summary.txt').write_text(report) + print(report, flush=True) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--seconds', type=int, default=15) + parser.add_argument('--rounds', type=int, default=1) + parser.add_argument('--candidate-build', default='build-layering') + parser.add_argument('--output-name', default='resident-layering-20261012') + args = parser.parse_args() + assert args.seconds >= 10 and args.rounds >= 1 + assert Path(args.output_name).name == args.output_name + paths, builds, diff = provenance(args.candidate_build) + fixture = ROOT / 'benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db' + out = ROOT / 'benchmarks/disk-format' / args.output_name + out.mkdir(parents=True, exist_ok=False) + storage = subprocess.check_output(['findmnt', '-n', '-o', 'FSTYPE,SOURCE', '-T', str(out)], + text=True).strip() + fstype, device = storage.split()[:2] + assert fstype not in {'tmpfs', 'ramfs'} and device.startswith('/dev/'), storage + order = [] + for number in range(args.rounds): + order.extend(['before', 'layered', 'layered', 'before'] if number % 2 == 0 + else ['layered', 'before', 'before', 'layered']) + manifest = dict(project='disk-format-layering', output_directory=str(out), regime='resident', + io='O_DIRECT', cpu_affinity='0-11', filesystem=storage, order=order, + expected_runs=len(order), n=200000, cache_mib=128, log=0, phases=PHASES, + seconds=args.seconds, fixture=str(fixture), fixture_sha256=digest(fixture), + copies_fsynced_before_open=True, build_provenance=builds, + runner_sha256=digest(__file__), runs=[], complete=False) + save_json(out / 'manifest.json', manifest) + shutil.copy2(__file__, out / Path(__file__).name) + (out / 'measured-source.patch').write_bytes(diff) + rows = [] + for index, label in enumerate(order): + db = out / f'{index}-{label}.db' + copy_started = time.time() + subprocess.run(['cp', '--reflink=auto', '--sparse=always', str(fixture), str(db)], check=True) + with db.open('r+b') as image: + os.fsync(image.fileno()) + copied_at = time.time() + argv = ['taskset', '-c', '0-11', str(paths[label]), '--db', str(db), '--open', + '--n', '200000', '--cache-mib', '128', '--disk-gib', '8', '--log', '0', + '--threads', '8', '--mt-bg', '1', '--normal-bg', '3', '--memtable-mib', '4', + '--secs', str(args.seconds), '--warmup', '1', '--csv', str(out / 'results.csv'), + '--vtag', label, '--stag', 'resident', '--otag', str(index), '--rep', str(index // 4), + '--seed', '42', '--hang-secs', '60', '--hang-dir', str(out), *PHASES] + record = dict(index=index, version=label, command=argv, binary_sha256=digest(paths[label]), + copy_started=copy_started, copy_fsync_finished=copied_at, started=time.time(), + correctness_passed=False) + manifest['runs'].append(record) + save_json(out / 'manifest.json', manifest) + print(f'format layering {index + 1}/{len(order)} {label}', flush=True) + with (out / f'{index}-{label}.log').open('w') as target: + process = subprocess.run(argv, stdout=target, stderr=subprocess.STDOUT, + timeout=max(600, args.seconds * len(PHASES) + 120)) + record.update(returncode=process.returncode, finished=time.time(), + db_allocated_bytes=db.stat().st_blocks * 512, db_apparent_bytes=db.stat().st_size) + save_json(out / 'manifest.json', manifest) + assert process.returncode == 0, record + rows = csv_rows(out / 'results.csv') + current = [row for row in rows if row['order'] == str(index)] + assert len(current) == len(PHASES), (index, current) + assert all(int(row['errors']) == 0 and int(row['violations']) == 0 for row in current), current + assert all(int(row['ops']) > 0 for row in current), current + record['correctness_passed'] = True + save_json(out / 'manifest.json', manifest) + summarize(out, rows, args.rounds) + manifest['complete'] = True + save_json(out / 'manifest.json', manifest) + print('Completed', out, flush=True) + + +if __name__ == '__main__': + main() diff --git a/benchmarks/results-20261011/layering-20261012/manifest.json b/benchmarks/results-20261011/layering-20261012/manifest.json new file mode 100644 index 000000000..ec434e061 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/manifest.json @@ -0,0 +1,615 @@ +{ + "project": "disk-format-layering", + "output_directory": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "regime": "resident", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "filesystem": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "order": [ + "before", + "layered", + "layered", + "before", + "layered", + "before", + "before", + "layered" + ], + "expected_runs": 8, + "n": 200000, + "cache_mib": 128, + "log": 0, + "phases": [ + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "seconds": 15, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-log0-final-singleload/1-candidate.db", + "fixture_sha256": "1304b9adc006f4e9776ec7bbdd5a6fa39e2cd5e2db54b588c22a8164154bbe73", + "copies_fsynced_before_open": true, + "build_provenance": { + "before": { + "preserved_at": 1791841463.654878, + "source_head": "07dfa1761182214f18b8269c7b24cc67bd35c0f4", + "files": { + "kvbench": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/bin/kvbench", + "sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "bytes": 4016432 + }, + "kvbench.json": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/bin/kvbench.json", + "sha256": "274e705c37837c0bebd61d49ce0df7f3b568f9ed2ee73574dc34e991a4b8ba38", + "bytes": 978 + }, + "libsplinterdb.a": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/lib/libsplinterdb.a", + "sha256": "77bc8b4ddb7f93ae4308de9c9e3506019f371bd9c3badd2ba5d2772e1ba637f5", + "bytes": 5611354 + }, + "libsplinterdb.so": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-scalar/release/lib/libsplinterdb.so", + "sha256": "dcf51bf2c4071e088e8a144b9a3f2f36f69c7049b24cea573a98738e31ce0ebe", + "bytes": 4099960 + }, + "kvbench.c": { + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/benchmarks/kvbench.c", + "sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "bytes": 49348 + } + }, + "note": "Existing final scalar reader, before layering fix. kvbench embeds libsplinterdb.a; preserved .so is not loaded by kvbench.", + "dynamic_section": "\nDynamic section at offset 0xb3da0 contains 29 entries:\n Tag Type Name/Value\n 0x0000000000000001 (NEEDED) Shared library: [libaio.so.1t64]\n 0x0000000000000001 (NEEDED) Shared library: [libxxhash.so.0]\n 0x0000000000000001 (NEEDED) Shared library: [libm.so.6]\n 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]\n 0x000000000000000c (INIT) 0x3000\n 0x000000000000000d (FINI) 0x996fc\n 0x0000000000000019 (INIT_ARRAY) 0xb4b60\n 0x000000000000001b (INIT_ARRAYSZ) 24 (bytes)\n 0x000000000000001a (FINI_ARRAY) 0xb4b78\n 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)\n 0x000000006ffffef5 (GNU_HASH) 0x3e8\n 0x0000000000000005 (STRTAB) 0xed8\n 0x0000000000000006 (SYMTAB) 0x410\n 0x000000000000000a (STRSZ) 1570 (bytes)\n 0x000000000000000b (SYMENT) 24 (bytes)\n 0x0000000000000015 (DEBUG) 0x0\n 0x0000000000000003 (PLTGOT) 0xb4fe8\n 0x0000000000000002 (PLTRELSZ) 2568 (bytes)\n 0x0000000000000014 (PLTREL) RELA\n 0x0000000000000017 (JMPREL) 0x2408\n 0x0000000000000007 (RELA) 0x1700\n 0x0000000000000008 (RELASZ) 3336 (bytes)\n 0x0000000000000009 (RELAENT) 24 (bytes)\n 0x000000006ffffffb (FLAGS_1) Flags: PIE\n 0x000000006ffffffe (VERNEED) 0x15e0\n 0x000000006fffffff (VERNEEDNUM) 3\n 0x000000006ffffff0 (VERSYM) 0x14fa\n 0x000000006ffffff9 (RELACOUNT) 132\n 0x0000000000000000 (NULL) 0x0\n" + }, + "layered": { + "build_record": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/include", + "/home/rob/projects/splinterdb/.cache/projects-20261011/baseline/benchmarks/kvbench.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/bin/kvbench" + ], + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "binary_sha256": "74e89bee01a9d0bece7bff6045090bdd71195de95346a4da6d20dff1b7effc45" + }, + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/bin/kvbench", + "binary_sha256": "74e89bee01a9d0bece7bff6045090bdd71195de95346a4da6d20dff1b7effc45", + "library": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/lib/libsplinterdb.a", + "library_sha256": "2714fd520b4fe7ebae0e2a84b4bf1eaaad4f95391ad930030f8420347896aae7", + "source_head": "07dfa1761182214f18b8269c7b24cc67bd35c0f4", + "tracked_diff_sha256": "868dbec45883a5efdb8943ca640535cd3c8780a1e8a61a7e26e7ee8d15a0cfcc", + "tracked_diff_bytes": 80502 + } + }, + "runner_sha256": "690f73463dbcaf6135d401a70c885da3630027cd1acde504d548fc529522904c", + "runs": [ + { + "index": 0, + "version": "before", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/disk-format-layering-before/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/0-before.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "before", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "copy_started": 1791841933.3586056, + "copy_fsync_finished": 1791841933.379465, + "started": 1791841933.3824844, + "correctness_passed": true, + "returncode": 0, + "finished": 1791841997.4802585, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + }, + { + "index": 1, + "version": "layered", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/1-layered.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "layered", + "--stag", + "resident", + "--otag", + "1", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "74e89bee01a9d0bece7bff6045090bdd71195de95346a4da6d20dff1b7effc45", + "copy_started": 1791841997.4807737, + "copy_fsync_finished": 1791841997.500524, + "started": 1791841997.5035355, + "correctness_passed": true, + "returncode": 0, + "finished": 1791842061.619791, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + }, + { + "index": 2, + "version": "layered", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/2-layered.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "layered", + "--stag", + "resident", + "--otag", + "2", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "74e89bee01a9d0bece7bff6045090bdd71195de95346a4da6d20dff1b7effc45", + "copy_started": 1791842061.621346, + "copy_fsync_finished": 1791842061.6482897, + "started": 1791842061.6512628, + "correctness_passed": true, + "returncode": 0, + "finished": 1791842125.8023474, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + }, + { + "index": 3, + "version": "before", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/disk-format-layering-before/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/3-before.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "before", + "--stag", + "resident", + "--otag", + "3", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "copy_started": 1791842125.810495, + "copy_fsync_finished": 1791842125.8329332, + "started": 1791842125.8358076, + "correctness_passed": true, + "returncode": 0, + "finished": 1791842189.9438078, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + }, + { + "index": 4, + "version": "layered", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/4-layered.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "layered", + "--stag", + "resident", + "--otag", + "4", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "74e89bee01a9d0bece7bff6045090bdd71195de95346a4da6d20dff1b7effc45", + "copy_started": 1791842189.9446716, + "copy_fsync_finished": 1791842189.9628584, + "started": 1791842189.9659302, + "correctness_passed": true, + "returncode": 0, + "finished": 1791842254.1239367, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + }, + { + "index": 5, + "version": "before", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/disk-format-layering-before/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/5-before.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "before", + "--stag", + "resident", + "--otag", + "5", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "copy_started": 1791842254.132127, + "copy_fsync_finished": 1791842254.154125, + "started": 1791842254.157362, + "correctness_passed": true, + "returncode": 0, + "finished": 1791842318.2676814, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + }, + { + "index": 6, + "version": "before", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/disk-format-layering-before/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/6-before.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "before", + "--stag", + "resident", + "--otag", + "6", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "ed7abc73e806e89b1f953ade2e1b0f9c7e57b27807a0e78a6523e2356f25558d", + "copy_started": 1791842318.2692873, + "copy_fsync_finished": 1791842318.2864814, + "started": 1791842318.2894585, + "correctness_passed": true, + "returncode": 0, + "finished": 1791842382.4417546, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + }, + { + "index": 7, + "version": "layered", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-layering/release/bin/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/7-layered.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012/results.csv", + "--vtag", + "layered", + "--stag", + "resident", + "--otag", + "7", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/disk-format/resident-layering-20261012", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8" + ], + "binary_sha256": "74e89bee01a9d0bece7bff6045090bdd71195de95346a4da6d20dff1b7effc45", + "copy_started": 1791842382.449725, + "copy_fsync_finished": 1791842382.4688013, + "started": 1791842382.4717627, + "correctness_passed": true, + "returncode": 0, + "finished": 1791842446.6275752, + "db_allocated_bytes": 28655616, + "db_apparent_bytes": 162664448 + } + ], + "complete": true +} diff --git a/benchmarks/results-20261011/layering-20261012/measured-source.patch b/benchmarks/results-20261011/layering-20261012/measured-source.patch new file mode 100644 index 000000000..c67f3e83e --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/measured-source.patch @@ -0,0 +1,1744 @@ +diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md +index 1d08e05..bf15d21 100644 +--- a/docs/on-disk-format.md ++++ b/docs/on-disk-format.md +@@ -106,3 +106,34 @@ committed under `benchmarks/results-20261011/`. Full campaign artifacts remain + at `.cache/projects-20261011/benchmarks/disk-format/` in the main checkout; + benchmark database files were on a verified disk filesystem, with the shared + benchmark lock excluding other benchmarks, builds and tests. ++ ++## Format ownership follow-up (2026-10-12) ++ ++Branch compatibility now belongs to the B-tree. Point queries pass the selected ++branch's persisted version to synchronous/asynchronous B-tree entry points; ++`trunk_ondisk_bundle_readable()` no longer scans every branch. The B-tree rejects ++unsupported versions or null roots before reading pages. Range initialization, ++compaction, root-height/count queries, and allocation recovery preserve the ++version alongside the address. Memtables do not acquire a format requirement. ++ ++Reference acquisition needs validation before iterator initialization: a null ++root would otherwise target extent zero through the mini allocator. Existing ++bundle-deserialization and range-collection loops therefore call the B-tree's ++root validator before admitting references. Failed collection discards the ++unreferenced partial bundle, preserving correct cleanup. These checks add no ++branch scan to the ordinary point-query bundle decoder. ++ ++The ownership audit also moved trunk/allocator version constants and support ++checks to their modules. Superblock reading still selects the newest valid ++record before invoking owner validators for trunk, allocator, and both logs; ++it never silently falls back to an older record with supported components. ++Routing-filter instance-capacity calculation now lives in the routing-filter ++module instead of reconstructing its encoding configuration in the trunk. ++Routing-filter descriptor validation and mini-allocator private-header ++validation were already correctly owned. No on-disk layout or version changed ++in this follow-up. ++ ++Direct B-tree regressions cover old/future versions and null roots before any ++cache access, including safe iterator cleanup. The core corruption regression ++covers synchronous/asynchronous point hits and ranges, and verifies that ++routing-filter misses succeed without inspecting the irrelevant branch. +diff --git a/src/btree.c b/src/btree.c +index a9d6009..b95c3a8 100644 +--- a/src/btree.c ++++ b/src/btree.c +@@ -1679,6 +1679,26 @@ btree_create(cache *cc, + return STATUS_OK; + } + ++/* The B-tree owns branch format compatibility, independent of its container. */ ++platform_status ++btree_validate_root(uint64 root_addr, uint16 format_version, page_type type) ++{ ++ if (type == PAGE_TYPE_BRANCH ++ && format_version != BTREE_BRANCH_FORMAT_VERSION) ++ { ++ platform_error_log("Unsupported branch format version %u at %lu; " ++ "this build requires version %u.\n", ++ format_version, ++ root_addr, ++ BTREE_BRANCH_FORMAT_VERSION); ++ return STATUS_INVALID_STATE; ++ } ++ if (root_addr == 0) { ++ return STATUS_INVALID_STATE; ++ } ++ return STATUS_OK; ++} ++ + void + btree_inc_ref(cache *cc, const btree_config *cfg, uint64 root_addr) + { +@@ -1696,8 +1716,15 @@ platform_status + btree_recover_allocations(cache *cc, + const btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + page_type type) + { ++ platform_status version_rc = ++ btree_validate_root(root_addr, format_version, type); ++ if (!SUCCESS(version_rc)) { ++ return version_rc; ++ } ++ + return mini_recover_references( + cc, btree_root_to_meta_addr(cfg, root_addr, 0), type); + } +@@ -1717,9 +1744,16 @@ platform_status + btree_get_root_height(cache *cc, + const btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + page_type type, + uint32 *height) + { ++ platform_status version_rc = ++ btree_validate_root(root_addr, format_version, type); ++ if (!SUCCESS(version_rc)) { ++ return version_rc; ++ } ++ + if (height == NULL + || (type != PAGE_TYPE_BRANCH && type != PAGE_TYPE_MEMTABLE)) + { +@@ -3040,27 +3074,39 @@ btree_lookup_with_ref(cache *cc, // IN + } + + platform_status +-btree_lookup(cache *cc, // IN +- btree_config *cfg, // IN +- uint64 root_addr, // IN +- page_type type, // IN +- key target, // IN +- lookup_result *result) // OUT ++btree_lookup(cache *cc, // IN ++ btree_config *cfg, // IN ++ uint64 root_addr, // IN ++ uint16 format_version, // IN ++ page_type type, // IN ++ key target, // IN ++ lookup_result *result) // OUT + { + lookup_result_reset(result); + return btree_lookup_and_merge( +- cc, cfg, root_addr, type, target, result, NULL); ++ cc, cfg, root_addr, format_version, type, target, result, NULL); + } + + platform_status +-btree_lookup_and_merge(cache *cc, // IN +- const btree_config *cfg, // IN +- uint64 root_addr, // IN +- page_type type, // IN +- key target, // IN +- lookup_result *result, // IN/OUT +- bool32 *local_found) // OUT ++btree_lookup_and_merge(cache *cc, // IN ++ const btree_config *cfg, // IN ++ uint64 root_addr, // IN ++ uint16 format_version, // IN ++ page_type type, // IN ++ key target, // IN ++ lookup_result *result, // IN/OUT ++ bool32 *local_found) // OUT + { ++ if (local_found != NULL) { ++ *local_found = FALSE; ++ } ++ ++ platform_status version_rc = ++ btree_validate_root(root_addr, format_version, type); ++ if (!SUCCESS(version_rc)) { ++ return version_rc; ++ } ++ + btree_node node; + key found_key; + message local_data; +@@ -3068,10 +3114,6 @@ btree_lookup_and_merge(cache *cc, // IN + + log_trace_key(target, "btree_lookup"); + +- if (local_found != NULL) { +- *local_found = FALSE; +- } +- + rc = btree_lookup_with_ref( + cc, cfg, root_addr, type, target, &node, &found_key, &local_data); + if (!SUCCESS(rc)) { +@@ -3116,6 +3158,12 @@ btree_lookup_and_merge_async(btree_lookup_async_state *state) + { + async_begin(state, 0); + ++ platform_status version_rc = ++ btree_validate_root(state->root_addr, state->format_version, state->type); ++ if (!SUCCESS(version_rc)) { ++ async_return(state, version_rc); ++ } ++ + async_await_subroutine(state, btree_lookup_with_ref_async); + + if (!SUCCESS(state->__async_result)) { +@@ -3660,6 +3708,7 @@ btree_iterator_find_end_addr_async(btree_iterator_async_state *state, + state->itor->cc, + state->itor->cfg, + state->itor->root_addr, ++ state->format_version, + state->itor->page_type, + state->itor->max_key, + NULL, +@@ -4670,6 +4719,7 @@ find_btree_node_and_get_idx_bounds_async(btree_iterator_async_state *state, + state->itor->cc, + state->itor->cfg, + state->itor->root_addr, ++ state->format_version, + state->itor->page_type, + state->target, + NULL, +@@ -4874,6 +4924,7 @@ btree_iterator_init_common(cache *cc, + const btree_config *cfg, + btree_iterator *itor, + uint64 root_addr, ++ uint16 format_version, + page_type page_type, + comparison min_key_comparison, + key min_key, +@@ -4885,7 +4936,12 @@ btree_iterator_init_common(cache *cc, + uint32 prefetch_lookahead, + key *normalized_start_key) + { +- platform_assert(root_addr != 0); ++ ZERO_CONTENTS(itor); ++ platform_status version_rc = ++ btree_validate_root(root_addr, format_version, page_type); ++ if (!SUCCESS(version_rc)) { ++ return version_rc; ++ } + debug_assert(page_type == PAGE_TYPE_MEMTABLE + || page_type == PAGE_TYPE_BRANCH); + +@@ -4908,7 +4964,6 @@ btree_iterator_init_common(cache *cc, + start_key = max_key; + } + +- ZERO_CONTENTS(itor); + itor->cc = cc; + itor->cfg = cfg; + itor->root_addr = root_addr; +@@ -4940,6 +4995,7 @@ btree_iterator_init(cache *cc, + const btree_config *cfg, + btree_iterator *itor, + uint64 root_addr, ++ uint16 format_version, + page_type page_type, + comparison min_key_comparison, + key min_key, +@@ -4955,6 +5011,7 @@ btree_iterator_init(cache *cc, + cfg, + itor, + root_addr, ++ format_version, + page_type, + min_key_comparison, + min_key, +@@ -4993,6 +5050,7 @@ btree_iterator_init_async(btree_iterator_async_state *state) + state->cfg, + state->itor, + state->root_addr, ++ state->format_version, + state->type, + state->min_key_comparison, + state->min_key, +@@ -5523,6 +5581,7 @@ platform_status + btree_count_in_range(cache *cc, + const btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + key min_key, + key max_key, + btree_pivot_stats *stats) +@@ -5532,6 +5591,12 @@ btree_count_in_range(cache *cc, + debug_assert(!key_is_null(min_key) && !key_is_null(max_key)); + memset(stats, 0, sizeof(*stats)); + ++ platform_status version_rc = ++ btree_validate_root(root_addr, format_version, PAGE_TYPE_BRANCH); ++ if (!SUCCESS(version_rc)) { ++ return version_rc; ++ } ++ + platform_status rc = btree_get_rank(cc, cfg, root_addr, min_key, &min_stats); + if (!SUCCESS(rc)) { + return rc; +@@ -5560,6 +5625,7 @@ void + btree_count_in_range_by_iterator(cache *cc, + btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + key min_key, + key max_key, + btree_pivot_stats *stats) +@@ -5570,6 +5636,7 @@ btree_count_in_range_by_iterator(cache *cc, + cfg, + &btree_itor, + root_addr, ++ format_version, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + min_key, +diff --git a/src/btree.h b/src/btree.h +index 96b495e..adc6493 100644 +--- a/src/btree.h ++++ b/src/btree.h +@@ -33,7 +33,11 @@ + */ + #define BTREE_MAX_HEIGHT (8) + +-/* Immutable branch version is stored in its trunk branch_ref. */ ++/* ++ * Immutable branch versions travel with the root address. Query and recovery ++ * entry points validate the version before interpreting branch pages. The ++ * version argument is ignored for PAGE_TYPE_MEMTABLE (an in-memory structure). ++ */ + #define BTREE_BRANCH_FORMAT_VERSION (1) + + /* +@@ -298,6 +302,12 @@ btree_create(cache *cc, + page_type type, + uint64 *root_addr); + ++/* Validate persisted roots before acquiring references. Query entry points ++ * also validate before reading pages. Refcount operations below require a ++ * validated root or one freshly created by this module. */ ++platform_status ++btree_validate_root(uint64 root_addr, uint16 format_version, page_type type); ++ + void + btree_inc_ref(cache *cc, const btree_config *cfg, uint64 root_addr); + +@@ -311,6 +321,7 @@ platform_status + btree_get_root_height(cache *cc, + const btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + page_type type, + uint32 *height); + +@@ -322,6 +333,7 @@ platform_status + btree_recover_allocations(cache *cc, + const btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + page_type type); + + void +@@ -331,6 +343,7 @@ platform_status + btree_lookup(cache *cc, + btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + page_type type, + key target, + lookup_result *result); +@@ -345,6 +358,7 @@ platform_status + btree_lookup_and_merge(cache *cc, + const btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + page_type type, + key target, + lookup_result *result, +@@ -355,6 +369,7 @@ DEFINE_ASYNC_STATE(btree_lookup_async_state, 3, + param, cache *, cc, + param, const btree_config *, cfg, + param, uint64, root_addr, ++ param, uint16, format_version, + param, page_type, type, + param, key, target, + param, lookup_result *, result, +@@ -377,14 +392,23 @@ btree_lookup_and_merge_async_state_init(btree_lookup_async_state *state, + cache *cc, + const btree_config *cfg, + uint64 root_addr, +- page_type type, +- key target, +- lookup_result *result, +- async_callback_fn callback, +- void *callback_arg) ++ uint16 format_version, ++ page_type type, ++ key target, ++ lookup_result *result, ++ async_callback_fn callback, ++ void *callback_arg) + { +- btree_lookup_async_state_init( +- state, cc, cfg, root_addr, type, target, result, callback, callback_arg); ++ btree_lookup_async_state_init(state, ++ cc, ++ cfg, ++ root_addr, ++ format_version, ++ type, ++ target, ++ result, ++ callback, ++ callback_arg); + } + + async_status +@@ -403,6 +427,7 @@ btree_iterator_init(cache *cc, + const btree_config *cfg, + btree_iterator *itor, + uint64 root_addr, ++ uint16 format_version, + page_type page_type, + comparison min_key_comparison, + key min_key, +@@ -420,6 +445,7 @@ DEFINE_ASYNC_STATE(btree_iterator_async_state, 6, + param, const btree_config *, cfg, + param, btree_iterator *, itor, + param, uint64, root_addr, ++ param, uint16, format_version, + param, page_type, type, + param, comparison, min_key_comparison, + param, key, min_key, +@@ -446,7 +472,6 @@ DEFINE_ASYNC_STATE(btree_iterator_async_state, 6, + local, uint64, next_addr, + local, uint64, prev_addr, + local, uint64, expected_page_no, +- local, uint64, num_entries, + local, key, first_key, + local, key, last_key) + // clang-format on +@@ -508,6 +533,7 @@ platform_status + btree_count_in_range(cache *cc, + const btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + key min_key, + key max_key, + btree_pivot_stats *stats); +@@ -516,6 +542,7 @@ void + btree_count_in_range_by_iterator(cache *cc, + btree_config *cfg, + uint64 root_addr, ++ uint16 format_version, + key min_key, + key max_key, + btree_pivot_stats *stats); +diff --git a/src/core.c b/src/core.c +index d20844f..8951084 100644 +--- a/src/core.c ++++ b/src/core.c +@@ -1247,6 +1247,7 @@ core_memtable_iterator_init(core_handle *spl, + spl->cfg.btree_cfg, + itor, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + min_key_comparison, + min_key, +@@ -1832,8 +1833,14 @@ core_memtable_lookup(core_handle *spl, + page_type type = + memtable_is_compacted ? PAGE_TYPE_BRANCH : PAGE_TYPE_MEMTABLE; + +- return btree_lookup_and_merge( +- cc, cfg, root_addr, type, target, result, NULL); ++ return btree_lookup_and_merge(cc, ++ cfg, ++ root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, ++ type, ++ target, ++ result, ++ NULL); + } + + static platform_status +@@ -1920,6 +1927,7 @@ core_start_btree_iterator_init_async( + core_btree_iterator_init_async_context *ctxt, + btree_iterator *itor, + uint64 root_addr, ++ uint16 format_version, + page_type page_type, + comparison min_key_comparison, + key min_key, +@@ -1935,6 +1943,7 @@ core_start_btree_iterator_init_async( + spl->cfg.btree_cfg, + itor, + root_addr, ++ format_version, + page_type, + min_key_comparison, + min_key, +@@ -2171,6 +2180,8 @@ core_range_iterator_init(core_handle *spl, + } + + range_itor->branch[range_itor->num_branches].addr = root_addr; ++ range_itor->branch[range_itor->num_branches].format_version = ++ compacted ? BTREE_BRANCH_FORMAT_VERSION : 0; + range_itor->branch[range_itor->num_branches].type = + compacted ? PAGE_TYPE_BRANCH : PAGE_TYPE_MEMTABLE; + range_itor->num_branches++; +@@ -2278,6 +2289,7 @@ core_range_iterator_init(core_handle *spl, + &init_ctxt[i], + btree_itor, + branch_addr, ++ range_itor->branch[branch_no].format_version, + page_type, + range_itor->local_min_key_comparison, + key_buffer_key(&range_itor->local_min_key), +@@ -2693,6 +2705,7 @@ core_fetch_and_update_catch_up(core_handle *spl, + rc = btree_lookup_and_merge(spl->cc, + spl->cfg.btree_cfg, + refs[i].root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + refs[i].type, + tuple_key, + catch_up_result, +@@ -4758,6 +4771,7 @@ core_print_lookup(core_handle *spl, key target, platform_log_handle *log_handle) + rc = btree_lookup(spl->cc, + spl->cfg.btree_cfg, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + target, + &lookup); +diff --git a/src/rc_allocator.c b/src/rc_allocator.c +index aba2f3b..d9d1088 100644 +--- a/src/rc_allocator.c ++++ b/src/rc_allocator.c +@@ -17,6 +17,19 @@ + #include "platform_typed_alloc.h" + #include "poison.h" + ++platform_status ++rc_allocator_validate_format_version(uint16 format_version) ++{ ++ if (format_version != RC_ALLOCATOR_FORMAT_VERSION) { ++ platform_error_log("Unsupported allocator format version %u; " ++ "this build requires version %u.\n", ++ format_version, ++ RC_ALLOCATOR_FORMAT_VERSION); ++ return STATUS_INVALID_STATE; ++ } ++ return STATUS_OK; ++} ++ + /* + * The refcount map is the allocator's only durable structure. It lives in a + * fixed run of extents starting at extent 1 (extent 0 is reserved for the +diff --git a/src/rc_allocator.h b/src/rc_allocator.h +index 91eccb7..954bb77 100644 +--- a/src/rc_allocator.h ++++ b/src/rc_allocator.h +@@ -15,6 +15,12 @@ + #include "allocator.h" + #include "util.h" + ++#define RC_ALLOCATOR_FORMAT_VERSION (1) ++ ++/* Validate the allocator version stored in the database superblock. */ ++platform_status ++rc_allocator_validate_format_version(uint16 format_version); ++ + /* + *---------------------------------------------------------------------- + * rc_allocator_stats -- +diff --git a/src/routing_filter.c b/src/routing_filter.c +index 4eb8a4b..318e71f 100644 +--- a/src/routing_filter.c ++++ b/src/routing_filter.c +@@ -252,6 +252,19 @@ routing_filter_get_params(const routing_config *cfg, + return STATUS_OK; + } + ++uint64 ++routing_filter_max_fingerprints_for_filter(cache_config *cache_cfg, ++ const routing_filter *filter) ++{ ++ debug_assert(routing_filter_descriptor_is_valid(filter)); ++ debug_assert(filter->addr != 0); ++ routing_config cfg = { ++ .fingerprint_size = filter->fingerprint_size, ++ .log_index_size = filter->log_index_size, ++ }; ++ return routing_filter_max_fingerprints(cache_cfg, &cfg); ++} ++ + static platform_status + routing_get_header(cache *cc, + const routing_config *cfg, +diff --git a/src/routing_filter.h b/src/routing_filter.h +index cac429f..3188318 100644 +--- a/src/routing_filter.h ++++ b/src/routing_filter.h +@@ -184,6 +184,11 @@ routing_filter_max_fingerprints(cache_config *cache_cfg, + return MIN(UINT32_MAX, MIN(index_limit, fingerprint_limit)); + } + ++/* Capacity of an existing instance, independent of new-filter settings. */ ++uint64 ++routing_filter_max_fingerprints_for_filter(cache_config *cache_cfg, ++ const routing_filter *filter); ++ + // clang-format off + DEFINE_ASYNC_STATE(routing_filter_lookup_async_state, 2, + param, cache *, cc, +diff --git a/src/shard_log.c b/src/shard_log.c +index 1983b2a..e3610aa 100644 +--- a/src/shard_log.c ++++ b/src/shard_log.c +@@ -21,6 +21,19 @@ + #include "platform_sort.h" + #include "poison.h" + ++platform_status ++shard_log_validate_format_version(uint16 format_version) ++{ ++ if (format_version != SHARD_LOG_FORMAT_VERSION) { ++ platform_error_log("Unsupported log format version %u; " ++ "this build requires version %u.\n", ++ format_version, ++ SHARD_LOG_FORMAT_VERSION); ++ return STATUS_INVALID_STATE; ++ } ++ return STATUS_OK; ++} ++ + static const page_type shard_log_page_type_table[NUM_BLOB_BATCHES + 1] = { + PAGE_TYPE_LOG, + [1 ... NUM_BLOB_BATCHES] = PAGE_TYPE_BLOB, +@@ -2189,8 +2202,9 @@ shard_log_recover_allocations(cache *cc, shard_log_config *cfg, log_head head) + if (head.addr == 0) { + return STATUS_OK; // no such log + } +- if (head.format_version != SHARD_LOG_FORMAT_VERSION) { +- return STATUS_INVALID_STATE; ++ platform_status rc = shard_log_validate_format_version(head.format_version); ++ if (!SUCCESS(rc)) { ++ return rc; + } + shard_log_config read_cfg = *cfg; + read_cfg.seed = head.checksum_seed; +@@ -2202,10 +2216,9 @@ shard_log_recover_allocations(cache *cc, shard_log_config *cfg, log_head head) + * page-header chain never reaches it. It has to be recorded on its own or + * the extent is left looking free while the durable record still names it. + */ +- allocator *al = cache_get_allocator(cc); +- uint64 meta_base = shard_log_extent_base(cc, head.meta_addr); +- platform_status rc = +- allocator_recovery_record_reference(al, meta_base, PAGE_TYPE_LOG); ++ allocator *al = cache_get_allocator(cc); ++ uint64 meta_base = shard_log_extent_base(cc, head.meta_addr); ++ rc = allocator_recovery_record_reference(al, meta_base, PAGE_TYPE_LOG); + if (!SUCCESS(rc)) { + return rc; + } +@@ -3015,9 +3028,10 @@ shard_log_iterator_create(cache *cc, + if (itor_out == NULL) { + return STATUS_BAD_PARAM; + } +- *itor_out = NULL; +- if (head.format_version != SHARD_LOG_FORMAT_VERSION) { +- return STATUS_INVALID_STATE; ++ *itor_out = NULL; ++ platform_status rc = shard_log_validate_format_version(head.format_version); ++ if (!SUCCESS(rc)) { ++ return rc; + } + + shard_log_iterator *itor = TYPED_MALLOC(hid, itor); +@@ -3026,7 +3040,7 @@ shard_log_iterator_create(cache *cc, + "shard_log_iterator\n"); + return STATUS_NO_MEMORY; + } +- platform_status rc = shard_log_iterator_init( ++ rc = shard_log_iterator_init( + cc, cfg, hid, head.addr, head, first_needed_generation, itor); + if (!SUCCESS(rc)) { + platform_error_log("shard_log_iterator_create: shard_log_iterator_init " +diff --git a/src/shard_log.h b/src/shard_log.h +index b0a61e4..c3a2f35 100644 +--- a/src/shard_log.h ++++ b/src/shard_log.h +@@ -20,6 +20,10 @@ + #include "platform_mutex.h" + #include "writeback_set.h" + ++/* Validate the version persisted in a log head before accessing its stream. */ ++platform_status ++shard_log_validate_format_version(uint16 format_version); ++ + /* + * Configuration structure to set up the sharded log sub-system. + */ +diff --git a/src/splinterdb.c b/src/splinterdb.c +index 144fc96..e830dd1 100644 +--- a/src/splinterdb.c ++++ b/src/splinterdb.c +@@ -753,6 +753,7 @@ splinterdb_get_cost_stats(splinterdb *kvs, splinterdb_cost_stats *stats) + status = btree_get_root_height(mt_ctxt->cc, + mt_ctxt->cfg.btree_cfg, + memtable_root_addr(&mt_ctxt->mt[mt_no]), ++ 0, + PAGE_TYPE_MEMTABLE, + &height); + if (!SUCCESS(status)) { +diff --git a/src/superblock.c b/src/superblock.c +index da36f73..acb3fea 100644 +--- a/src/superblock.c ++++ b/src/superblock.c +@@ -8,6 +8,9 @@ + */ + + #include "superblock.h" ++#include "trunk.h" ++#include "rc_allocator.h" ++#include "shard_log.h" + #include "poison.h" + + #define SUPERBLOCK_CSUM_SEED (0x53555045524231ULL) // "SUPERB1" +@@ -130,21 +133,24 @@ superblock_check_versions(const superblock *sb) + if (sb->format_version != SUPERBLOCK_FORMAT_VERSION) { + return superblock_unsupported_version(sb->format_version); + } +- if (sb->trunk_format_version != TRUNK_FORMAT_VERSION +- || sb->allocator_format_version != RC_ALLOCATOR_FORMAT_VERSION +- || (!SUPERBLOCK_NO_LOG(sb->tree.live_log) +- && sb->tree.live_log.head.format_version != SHARD_LOG_FORMAT_VERSION) +- || (!SUPERBLOCK_NO_LOG(sb->tree.sealed_log) +- && sb->tree.sealed_log.head.format_version +- != SHARD_LOG_FORMAT_VERSION)) +- { +- platform_error_log("Unsupported on-disk structure version: " +- "trunk=%u allocator=%u live-log=%u sealed-log=%u\n", +- sb->trunk_format_version, +- sb->allocator_format_version, +- sb->tree.live_log.head.format_version, +- sb->tree.sealed_log.head.format_version); +- return STATUS_INVALID_STATE; ++ platform_status rc = trunk_validate_format_version(sb->trunk_format_version); ++ if (!SUCCESS(rc)) { ++ return rc; ++ } ++ rc = rc_allocator_validate_format_version(sb->allocator_format_version); ++ if (!SUCCESS(rc)) { ++ return rc; ++ } ++ if (!SUPERBLOCK_NO_LOG(sb->tree.live_log)) { ++ rc = shard_log_validate_format_version( ++ sb->tree.live_log.head.format_version); ++ if (!SUCCESS(rc)) { ++ return rc; ++ } ++ } ++ if (!SUPERBLOCK_NO_LOG(sb->tree.sealed_log)) { ++ return shard_log_validate_format_version( ++ sb->tree.sealed_log.head.format_version); + } + return STATUS_OK; + } +diff --git a/src/superblock.h b/src/superblock.h +index 3343e91..fff86ab 100644 +--- a/src/superblock.h ++++ b/src/superblock.h +@@ -48,9 +48,7 @@ + * descriptors carry their format version and fingerprint size; v8 adds + * independent structure versions and complete routing-filter/log read config. + */ +-#define SUPERBLOCK_FORMAT_VERSION (8) +-#define TRUNK_FORMAT_VERSION (1) +-#define RC_ALLOCATOR_FORMAT_VERSION (1) ++#define SUPERBLOCK_FORMAT_VERSION (8) + + /* The two physical superblock copies live at pages 0 and 1. */ + #define SUPERBLOCK_NUM_SLOTS (2) +diff --git a/src/trunk.c b/src/trunk.c +index e4c8b56..a89fa71 100644 +--- a/src/trunk.c ++++ b/src/trunk.c +@@ -1528,13 +1528,6 @@ trunk_ondisk_bundle_readable(trunk_ondisk_bundle *bundle, + { + return FALSE; + } +- for (uint64 i = 0; i < bundle->num_branches; i++) { +- if (bundle->branches[i].addr == 0 +- || bundle->branches[i].format_version != BTREE_BRANCH_FORMAT_VERSION) +- { +- return FALSE; +- } +- } + if (!routing_filter_descriptor_is_valid(&bundle->maplet)) { + return FALSE; + } +@@ -1904,6 +1897,13 @@ bundle_deserialize(bundle *bndl, platform_heap_id hid, trunk_ondisk_bundle *odb) + bndl->maplet = odb->maplet; + + for (uint64 i = 0; i < odb->num_branches; i++) { ++ rc = btree_validate_root(odb->branches[i].addr, ++ odb->branches[i].format_version, ++ trunk_ondisk_bundle_branch_type(odb)); ++ if (!SUCCESS(rc)) { ++ bundle_deinit(bndl); ++ return rc; ++ } + rc = vector_append(&bndl->branches, odb->branches[i]); + platform_assert_status_ok(rc); + } +@@ -2337,8 +2337,11 @@ trunk_recover_bundle_refs(const trunk_context *context, bundle *bndl) + page_type type = bundle_branch_type(bndl); + for (uint64 i = 0; i < vector_length(&bndl->branches); i++) { + branch_ref bref = vector_get(&bndl->branches, i); +- platform_status rc = btree_recover_allocations( +- context->cc, context->cfg->btree_cfg, branch_ref_addr(bref), type); ++ platform_status rc = btree_recover_allocations(context->cc, ++ context->cfg->btree_cfg, ++ branch_ref_addr(bref), ++ bref.format_version, ++ type); + if (!SUCCESS(rc)) { + return rc; + } +@@ -2973,13 +2976,14 @@ trunk_branch_merger_init(trunk_branch_merger *merger, + static platform_status + trunk_branch_merger_add_branch(trunk_branch_merger *merger, + uint64 addr, ++ uint16 format_version, + page_type type) + { + platform_assert(merger != NULL); + platform_assert(addr != 0); + platform_assert(PAGE_TYPE_FIRST <= type && type < NUM_PAGE_TYPES); + +- trunk_branch_info branch = {addr, type}; ++ trunk_branch_info branch = {addr, type, format_version}; + platform_status rc = vector_append(&merger->branches, branch); + if (!SUCCESS(rc)) { + platform_error_log("%s():%d: vector_append() failed: %s", +@@ -3010,8 +3014,10 @@ trunk_branch_merger_add_branches(trunk_branch_merger *merger, + } + + for (uint64 i = 0; i < num_branches; i++) { +- rc = trunk_branch_merger_add_branch( +- merger, branches[i].addr, branches[i].type); ++ rc = trunk_branch_merger_add_branch(merger, ++ branches[i].addr, ++ branches[i].format_version, ++ branches[i].type); + if (!SUCCESS(rc)) { + platform_error_log("%s():%d: btree_merger_add_branch() failed: %s", + __func__, +@@ -3043,8 +3049,10 @@ trunk_branch_merger_add_bundle(trunk_branch_merger *merger, + + for (uint64 i = 0; i < bundle_num_branches(routed); i++) { + branch_ref bref = vector_get(&routed->branches, i); +- rc = trunk_branch_merger_add_branch( +- merger, branch_ref_addr(bref), bundle_branch_type(routed)); ++ rc = trunk_branch_merger_add_branch(merger, ++ branch_ref_addr(bref), ++ bref.format_version, ++ bundle_branch_type(routed)); + if (!SUCCESS(rc)) { + platform_error_log("%s():%d: btree_merger_add_branch() failed: %s", + __func__, +@@ -3095,8 +3103,12 @@ trunk_branch_merger_build_merge_itor(trunk_branch_merger *merger, + for (uint64 i = 0; i < num_branches; i++) { + trunk_branch_info branch = vector_get(&merger->branches, i); + uint32 root_height; +- rc = btree_get_root_height( +- cc, merger->cfg->btree_cfg, branch.addr, branch.type, &root_height); ++ rc = btree_get_root_height(cc, ++ merger->cfg->btree_cfg, ++ branch.addr, ++ branch.format_version, ++ branch.type, ++ &root_height); + if (!SUCCESS(rc)) { + platform_error_log("%s():%d: btree_get_root_height() failed: %s", + __func__, +@@ -3122,6 +3134,7 @@ trunk_branch_merger_build_merge_itor(trunk_branch_merger *merger, + merger->cfg->btree_cfg, + itor, + branch.addr, ++ branch.format_version, + branch.type, + greater_than_or_equal, + merger->min_key, +@@ -3970,7 +3983,7 @@ bundle_compaction_create(trunk_context *context, + btree_inc_ref( + context->cc, context->cfg->btree_cfg, branch_ref_addr(bref)); + page_type type = bundle_branch_type(bndl); +- trunk_branch_info bi = {bref.addr, type}; ++ trunk_branch_info bi = {bref.addr, type, bref.format_version}; + rc = vector_append(&result->input.branches, bi); + platform_assert_status_ok(rc); + __sync_fetch_and_add(&bc_incs, 1); +@@ -4728,8 +4741,13 @@ compute_input_stats(trunk_context *context, + for (uint64 i = 0; i < vector_length(branches); i++) { + trunk_branch_info bi = vector_get(branches, i); + btree_pivot_stats btree_stats; +- platform_status rc = btree_count_in_range( +- context->cc, context->cfg->btree_cfg, bi.addr, lb, ub, &btree_stats); ++ platform_status rc = btree_count_in_range(context->cc, ++ context->cfg->btree_cfg, ++ bi.addr, ++ bi.format_version, ++ lb, ++ ub, ++ &btree_stats); + if (!SUCCESS(rc)) { + return rc; + } +@@ -5063,6 +5081,7 @@ accumulate_branch_tuple_counts_in_range(branch_ref bref, + platform_status rc = btree_count_in_range(context->cc, + context->cfg->btree_cfg, + branch_ref_addr(bref), ++ bref.format_version, + minkey, + maxkey, + &stats); +@@ -5267,6 +5286,7 @@ leaf_estimate_unique_keys(trunk_context *context, + rc = btree_count_in_range(context->cc, + context->cfg->btree_cfg, + bundle_branch(bndl, 0).addr, ++ bundle_branch(bndl, 0).format_version, + trunk_node_pivot_min_key(leaf), + trunk_node_pivot_max_key(leaf), + &stats); +@@ -5288,12 +5308,14 @@ leaf_estimate_unique_keys(trunk_context *context, + platform_assert(bundle_num_branches(&pivot_bundle) <= 1); + if (bundle_num_branches(&pivot_bundle) == 1) { + btree_pivot_stats stats; +- rc = btree_count_in_range(context->cc, +- context->cfg->btree_cfg, +- bundle_branch(&pivot_bundle, 0).addr, +- trunk_node_pivot_min_key(leaf), +- trunk_node_pivot_max_key(leaf), +- &stats); ++ rc = ++ btree_count_in_range(context->cc, ++ context->cfg->btree_cfg, ++ bundle_branch(&pivot_bundle, 0).addr, ++ bundle_branch(&pivot_bundle, 0).format_version, ++ trunk_node_pivot_min_key(leaf), ++ trunk_node_pivot_max_key(leaf), ++ &stats); + if (!SUCCESS(rc)) { + goto cleanup; + } +@@ -5382,10 +5404,9 @@ trunk_node_filter_limit(const trunk_context *context, const trunk_node *node) + const routing_filter *filter = + &vector_get_ptr(&node->pivot_bundles, i)->maplet; + if (filter->addr != 0) { +- routing_config cfg = *context->cfg->filter_cfg; +- cfg.fingerprint_size = filter->fingerprint_size; +- cfg.log_index_size = filter->log_index_size; +- limit = MIN(limit, routing_filter_max_fingerprints(cache_cfg, &cfg)); ++ limit = ++ MIN(limit, ++ routing_filter_max_fingerprints_for_filter(cache_cfg, filter)); + } + } + return limit; +@@ -7048,6 +7069,7 @@ trunk_ondisk_bundle_merge_lookup(trunk_context *context, + rc = btree_lookup_and_merge(context->cc, + context->cfg->btree_cfg, + branch_ref_addr(bndl->branches[idx]), ++ bndl->branches[idx].format_version, + trunk_ondisk_bundle_branch_type(bndl), + tgt, + result, +@@ -7080,6 +7102,7 @@ trunk_ondisk_bundle_merge_lookup(trunk_context *context, + rc = btree_lookup_and_merge(context->cc, + context->cfg->btree_cfg, + branch_ref_addr(bndl->branches[idx]), ++ bndl->branches[idx].format_version, + trunk_ondisk_bundle_branch_type(bndl), + tgt, + &temp_result, +@@ -7185,6 +7208,7 @@ trunk_ondisk_bundle_merge_lookup_async(trunk_merge_lookup_async_state *state, + state->context->cc, + state->context->cfg->btree_cfg, + branch_ref_addr(state->bndl->branches[state->idx]), ++ state->bndl->branches[state->idx].format_version, + trunk_ondisk_bundle_branch_type(state->bndl), + state->tgt, + state->result, +@@ -7221,6 +7245,7 @@ trunk_ondisk_bundle_merge_lookup_async(trunk_merge_lookup_async_state *state, + state->context->cc, + state->context->cfg->btree_cfg, + branch_ref_addr(state->bndl->branches[state->idx]), ++ state->bndl->branches[state->idx].format_version, + trunk_ondisk_bundle_branch_type(state->bndl), + state->tgt, + &temp_result, +@@ -7537,7 +7562,17 @@ trunk_collect_bundle_branches(trunk_ondisk_bundle *bndl, + *num_branches = bundle_start; + return STATUS_LIMIT_EXCEEDED; + } ++ platform_status rc = ++ btree_validate_root(bndl->branches[i].addr, ++ bndl->branches[i].format_version, ++ trunk_ondisk_bundle_branch_type(bndl)); ++ if (!SUCCESS(rc)) { ++ /* References are acquired only after the whole bundle is copied. */ ++ *num_branches = bundle_start; ++ return rc; ++ } + branches[*num_branches].addr = branch_ref_addr(bndl->branches[i]); ++ branches[*num_branches].format_version = bndl->branches[i].format_version; + branches[*num_branches].type = trunk_ondisk_bundle_branch_type(bndl); + + (*num_branches)++; +diff --git a/src/trunk.h b/src/trunk.h +index 19abf28..de7768a 100644 +--- a/src/trunk.h ++++ b/src/trunk.h +@@ -19,6 +19,23 @@ + #include "data_internal.h" + #include "batch_rwlock.h" + ++#define TRUNK_FORMAT_VERSION (1) ++ ++/* The superblock persists this version; the trunk owns its supported readers. ++ */ ++static inline platform_status ++trunk_validate_format_version(uint16 format_version) ++{ ++ if (format_version != TRUNK_FORMAT_VERSION) { ++ platform_error_log("Unsupported trunk format version %u; " ++ "this build requires version %u.\n", ++ format_version, ++ TRUNK_FORMAT_VERSION); ++ return STATUS_INVALID_STATE; ++ } ++ return STATUS_OK; ++} ++ + typedef struct trunk_config { + const data_config *data_cfg; + const btree_config *btree_cfg; +@@ -335,6 +352,7 @@ trunk_merge_lookup(trunk_context *context, + typedef struct trunk_branch_info { + uint64 addr; + page_type type; ++ uint16 format_version; + } trunk_branch_info; + + platform_status +diff --git a/tests/functional/btree_test.c b/tests/functional/btree_test.c +index 55f3a2c..ab7eb54 100644 +--- a/tests/functional/btree_test.c ++++ b/tests/functional/btree_test.c +@@ -411,7 +411,8 @@ test_btree_lookup(cache *cc, + + lookup_result_init(&result, cfg->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); + +- rc = btree_lookup(cc, cfg, root_addr, type, target, &result); ++ rc = btree_lookup( ++ cc, cfg, root_addr, BTREE_BRANCH_FORMAT_VERSION, type, target, &result); + platform_assert_status_ok(rc); + + message data = +@@ -648,6 +649,7 @@ test_btree_scan_once(cache *cc, + btree_cfg, + &itor, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + type, + greater_than_or_equal, + min_key, +@@ -947,6 +949,7 @@ test_btree_scan_perf(cache *cc, + btree_cfg, + &itor, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1183,6 +1186,7 @@ test_btree_async_lookup(cache *cc, + cc, + cfg, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + type, + target, + &async_ctxt->result, +@@ -1251,7 +1255,8 @@ test_btree_lookup_status(cache *cc, + { + lookup_result result; + lookup_result_init(&result, cfg->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); +- platform_status rc = btree_lookup(cc, cfg, root_addr, type, target, &result); ++ platform_status rc = btree_lookup( ++ cc, cfg, root_addr, BTREE_BRANCH_FORMAT_VERSION, type, target, &result); + lookup_result_deinit(&result); + return rc; + } +@@ -1270,6 +1275,7 @@ test_btree_async_lookup_status(cache *cc, + cc, + cfg, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + type, + target, + &ctxt.result, +@@ -1334,6 +1340,7 @@ test_btree_iterator_init_status(cache *cc, + cfg, + &itor, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1357,6 +1364,7 @@ test_btree_iterator_init_status(cache *cc, + cfg, + &itor, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1429,6 +1437,7 @@ test_btree_query_validation(cache *cc, + cfg, + &itor, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1571,6 +1580,7 @@ test_btree_query_validation(cache *cc, + cfg, + &sequence_itor, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1732,6 +1742,7 @@ test_btree_query_validation(cache *cc, + cfg, + &height_itor, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + key_buffer_key(&touched_key), +@@ -1906,6 +1917,7 @@ test_btree_basic(cache *cc, + btree_cfg, + &itor, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -2098,6 +2110,7 @@ test_btree_create_packed_trees(cache *cc, + btree_cfg, + &itor, + memtable_root_addr(mt), ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -2156,6 +2169,7 @@ test_count_tuples_in_range(cache *cc, + cfg, + &itor, + root_addr[i], ++ BTREE_BRANCH_FORMAT_VERSION, + type, + greater_than_or_equal, + low_key, +@@ -2255,6 +2269,7 @@ test_btree_print_all_keys(cache *cc, + cfg, + &itor, + root_addr[i], ++ BTREE_BRANCH_FORMAT_VERSION, + type, + greater_than_or_equal, + low_key, +@@ -2334,6 +2349,7 @@ test_btree_merge_basic(cache *cc, + btree_cfg, + &btree_itor_arr[tree_no], + root_addr[tree_no], ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + lo, +@@ -2475,8 +2491,13 @@ test_btree_count_in_range(cache *cc, + btree_pivot_stats stats; + key min_key = key_buffer_key(&bound_key[0]); + key max_key = key_buffer_key(&bound_key[1]); +- rc = btree_count_in_range( +- cc, btree_cfg, root_addr, min_key, max_key, &stats); ++ rc = btree_count_in_range(cc, ++ btree_cfg, ++ root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, ++ min_key, ++ max_key, ++ &stats); + if (!SUCCESS(rc)) { + goto destroy_btree; + } +@@ -2561,6 +2582,7 @@ test_btree_rough_iterator(cache *cc, + btree_cfg, + &rough_btree_itor[tree_no], + root_addr[tree_no], ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -2727,6 +2749,7 @@ test_btree_merge_perf(cache *cc, + btree_cfg, + &btree_itor_arr[tree_no], + root_addr[global_tree_no], ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + min_key, +diff --git a/tests/unit/btree_stress_test.c b/tests/unit/btree_stress_test.c +index c343f47..5ca2171 100644 +--- a/tests/unit/btree_stress_test.c ++++ b/tests/unit/btree_stress_test.c +@@ -310,6 +310,7 @@ CTEST2(btree_stress, insert_callback_failure_abandons_insert) + rc = btree_lookup((cache *)&data->cc, + &data->dbtree_cfg, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + tuple_key, + &lookup); +@@ -352,6 +353,7 @@ CTEST2(btree_stress, insert_callback_failure_after_root_growth) + ASSERT_TRUE(SUCCESS(btree_get_root_height((cache *)&data->cc, + &data->dbtree_cfg, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + &height))); + ASSERT_EQUAL_U(0, height); +@@ -384,6 +386,7 @@ CTEST2(btree_stress, insert_callback_failure_after_root_growth) + ASSERT_TRUE(SUCCESS(btree_get_root_height((cache *)&data->cc, + &data->dbtree_cfg, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + &height))); + ASSERT_TRUE(height > 0); +@@ -396,6 +399,7 @@ CTEST2(btree_stress, insert_callback_failure_after_root_growth) + rc = btree_lookup((cache *)&data->cc, + &data->dbtree_cfg, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + rejected_key, + &lookup); +@@ -540,8 +544,8 @@ CTEST2(btree_stress, insert_rebuild_failure_releases_parent_claim) + + lookup_result lookup; + lookup_result_init(&lookup, cfg->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); +- ASSERT_TRUE(SUCCESS( +- btree_lookup(cc, cfg, root_addr, PAGE_TYPE_MEMTABLE, target, &lookup))); ++ ASSERT_TRUE(SUCCESS(btree_lookup( ++ cc, cfg, root_addr, 0, PAGE_TYPE_MEMTABLE, target, &lookup))); + ASSERT_EQUAL(0, + slice_lex_cmp(merge_accumulator_to_value( + lookup_result_accumulator(&lookup)), +@@ -792,6 +796,7 @@ CTEST2(btree_stress, overwrite_returns_old_value_after_tree_growth) + rc = btree_lookup((cache *)&data->cc, + &data->dbtree_cfg, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + gen_key(&data->dbtree_cfg, i, keybuf, bt_page_size), + &verify_result); +@@ -929,6 +934,7 @@ query_tests(cache *cc, + btree_lookup(cc, + cfg, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + type, + gen_key(cfg, i, keybuf, bt_page_size), + &result); +@@ -1027,6 +1033,7 @@ iterator_tests(cache *cc, + cfg, + &dbiter, + root_addr, ++ BTREE_BRANCH_FORMAT_VERSION, + type, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1080,6 +1087,7 @@ iterator_seek_tests(cache *cc, + cfg, + &dbiter, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1128,6 +1136,7 @@ pack_tests(cache *cc, + cfg, + &dbiter, + root_addr, ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1229,6 +1238,7 @@ bounds_init_with_copy(cache *cc, + cfg, + itor, + root, ++ BTREE_BRANCH_FORMAT_VERSION, + type, + lo_cmp, + lo, +@@ -1247,6 +1257,7 @@ bounds_init_with_copy(cache *cc, + cfg, + itor, + root, ++ BTREE_BRANCH_FORMAT_VERSION, + type, + lo_cmp, + lo, +@@ -1368,6 +1379,7 @@ iterator_endpoint_fault_init(cache *cc, + cfg, + itor, + root, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1387,6 +1399,7 @@ iterator_endpoint_fault_init(cache *cc, + cfg, + itor, + root, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1424,6 +1437,7 @@ CTEST2(btree_stress, iterator_endpoint_error_is_returned_by_init) + cfg, + &source, + source_root, ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1450,6 +1464,7 @@ CTEST2(btree_stress, iterator_endpoint_error_is_returned_by_init) + cfg, + &probe, + packed_root, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1530,6 +1545,7 @@ CTEST2(btree_stress, count_in_range_returns_lookup_error) + cfg, + &source, + source_root, ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +@@ -1568,7 +1584,13 @@ CTEST2(btree_stress, count_in_range_returns_lookup_error) + + btree_pivot_stats stats = { + .num_kvs = 1, .key_bytes = 1, .message_bytes = 1}; +- rc = btree_count_in_range(cc, cfg, packed_root, min_key, max_key, &stats); ++ rc = btree_count_in_range(cc, ++ cfg, ++ packed_root, ++ BTREE_BRANCH_FORMAT_VERSION, ++ min_key, ++ max_key, ++ &stats); + + cc->ops = fault.original; + active_iterator_endpoint_fault = NULL; +@@ -1834,6 +1856,7 @@ CTEST2(btree_stress, copied_iterator_split_on_release) + cfg, + &probe, + root, ++ 0, + PAGE_TYPE_MEMTABLE, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, +diff --git a/tests/unit/btree_test.c b/tests/unit/btree_test.c +index 1885165..5a6e74b 100644 +--- a/tests/unit/btree_test.c ++++ b/tests/unit/btree_test.c +@@ -195,6 +195,181 @@ CTEST2(btree, test_defragmented_index_bytes) + ASSERT_EQUAL(0, rc); + } + ++/* ++ * Branch references must be rejected by the btree before any page access. ++ * A NULL cache makes an accidental read fail, including in the async paths. ++ */ ++static const struct { ++ uint64 addr; ++ uint16 format_version; ++} invalid_branch_refs[] = { ++ {4096, BTREE_BRANCH_FORMAT_VERSION - 1}, ++ {4096, BTREE_BRANCH_FORMAT_VERSION + 1}, ++ {0, BTREE_BRANCH_FORMAT_VERSION}, ++}; ++ ++CTEST2(btree, test_invalid_branch_point_lookup) ++{ ++ lookup_result result; ++ lookup_result_init( ++ &result, data->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); ++ uint64 keybuf = 1; ++ key target = key_create(FALSE, sizeof(keybuf), &keybuf); ++ ++ for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { ++ platform_status rc = btree_lookup(NULL, ++ &data->dbtree_cfg, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH, ++ target, ++ &result); ++ ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); ++ ASSERT_FALSE(lookup_result_found(&result)); ++ ++ bool32 local_found = TRUE; ++ rc = btree_lookup_and_merge(NULL, ++ &data->dbtree_cfg, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH, ++ target, ++ &result, ++ &local_found); ++ ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); ++ ASSERT_FALSE(local_found); ++ ASSERT_FALSE(lookup_result_found(&result)); ++ } ++ lookup_result_deinit(&result); ++} ++ ++CTEST2(btree, test_invalid_branch_async_point_lookup) ++{ ++ lookup_result result; ++ lookup_result_init( ++ &result, data->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); ++ uint64 keybuf = 1; ++ key target = key_create(FALSE, sizeof(keybuf), &keybuf); ++ ++ for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { ++ btree_lookup_async_state state; ++ btree_lookup_async_state_init(&state, ++ NULL, ++ &data->dbtree_cfg, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH, ++ target, ++ &result, ++ NULL, ++ NULL); ++ ASSERT_EQUAL(ASYNC_STATUS_DONE, btree_lookup_async(&state)); ++ ASSERT_TRUE(STATUS_IS_EQ(async_result(&state), STATUS_INVALID_STATE)); ++ ASSERT_FALSE(lookup_result_found(&result)); ++ ++ btree_lookup_and_merge_async_state_init( ++ &state, ++ NULL, ++ &data->dbtree_cfg, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH, ++ target, ++ &result, ++ NULL, ++ NULL); ++ ASSERT_EQUAL(ASYNC_STATUS_DONE, btree_lookup_and_merge_async(&state)); ++ ASSERT_TRUE(STATUS_IS_EQ(async_result(&state), STATUS_INVALID_STATE)); ++ ASSERT_FALSE(lookup_result_found(&result)); ++ } ++ lookup_result_deinit(&result); ++} ++ ++CTEST2(btree, test_invalid_branch_iterator) ++{ ++ for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { ++ for (bool32 copy_nodes = FALSE; copy_nodes <= TRUE; copy_nodes++) { ++ btree_iterator itor; ++ memset(&itor, 0xa5, sizeof(itor)); ++ platform_status rc = ++ btree_iterator_init(NULL, ++ &data->dbtree_cfg, ++ &itor, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH, ++ greater_than_or_equal, ++ NEGATIVE_INFINITY_KEY, ++ less_than, ++ POSITIVE_INFINITY_KEY, ++ greater_than_or_equal, ++ NEGATIVE_INFINITY_KEY, ++ copy_nodes, ++ 0, ++ 1); ++ ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); ++ btree_iterator_deinit(&itor); ++ ++ memset(&itor, 0xa5, sizeof(itor)); ++ btree_iterator_async_state state; ++ btree_iterator_async_state_init(&state, ++ NULL, ++ &data->dbtree_cfg, ++ &itor, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH, ++ greater_than_or_equal, ++ NEGATIVE_INFINITY_KEY, ++ less_than, ++ POSITIVE_INFINITY_KEY, ++ greater_than_or_equal, ++ NEGATIVE_INFINITY_KEY, ++ copy_nodes, ++ 0, ++ 1, ++ NULL, ++ NULL); ++ ASSERT_EQUAL(ASYNC_STATUS_DONE, btree_iterator_init_async(&state)); ++ ASSERT_TRUE(STATUS_IS_EQ(btree_iterator_init_async_result(&state), ++ STATUS_INVALID_STATE)); ++ btree_iterator_deinit(&itor); ++ } ++ } ++} ++ ++CTEST2(btree, test_invalid_branch_metadata_queries) ++{ ++ for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { ++ uint32 height; ++ platform_status rc = ++ btree_get_root_height(NULL, ++ &data->dbtree_cfg, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH, ++ &height); ++ ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); ++ ++ btree_pivot_stats stats; ++ rc = btree_count_in_range(NULL, ++ &data->dbtree_cfg, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ NEGATIVE_INFINITY_KEY, ++ POSITIVE_INFINITY_KEY, ++ &stats); ++ ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); ++ ++ rc = btree_recover_allocations(NULL, ++ &data->dbtree_cfg, ++ invalid_branch_refs[i].addr, ++ invalid_branch_refs[i].format_version, ++ PAGE_TYPE_BRANCH); ++ ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); ++ } ++} ++ + /* + * ***************************************************************** + * Helper functions, and actual test-case methods. +diff --git a/tests/unit/splinter_test.c b/tests/unit/splinter_test.c +index 85272c9..d1a84a0 100644 +--- a/tests/unit/splinter_test.c ++++ b/tests/unit/splinter_test.c +@@ -1494,7 +1494,7 @@ CTEST2(splinter, test_malformed_trunk_pivot_bundle) + ASSERT_TRUE(range_rejected); + } + +-CTEST2(splinter, test_unknown_branch_version_rejected) ++CTEST2(splinter, test_invalid_branch_reference_rejected) + { + cache *cc = (cache *)data->clock_cache; + core_handle spl; +@@ -1532,41 +1532,108 @@ CTEST2(splinter, test_unknown_branch_version_rejected) + test_trunk_bundle *bundle = + (test_trunk_bundle *)(page->data + offset % page_size); + ASSERT_TRUE(bundle->num_branches != 0); +- uint16 saved_version = bundle->branches[0].format_version; +- bundle->branches[0].format_version++; ++ branch_ref saved_branch = bundle->branches[0]; ++ routing_filter saved_maplet = bundle->maplet; + test_splinter_unclaim_page(cc, page); + +- lookup_result result; +- lookup_result_init( +- &result, spl.cfg.data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); +- platform_status sync_rc = +- core_lookup(&spl, key_buffer_key(&keybuf), &result); +- platform_status async_rc = async_call_sync_callback(cache_cleanup(cc), +- core_lookup_async, +- &spl, +- key_buffer_key(&keybuf), +- &result); +- lookup_result_deinit(&result); +- core_range_iterator itor; +- platform_status range_rc = core_range_iterator_init(&spl, +- &itor, +- greater_than_or_equal, +- NEGATIVE_INFINITY_KEY, +- less_than, +- POSITIVE_INFINITY_KEY, +- greater_than_or_equal, +- NEGATIVE_INFINITY_KEY); +- if (SUCCESS(range_rc)) { +- core_range_iterator_deinit(&itor); +- } +- page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); +- bundle = (test_trunk_bundle *)(page->data + offset % page_size); +- bundle->branches[0].format_version = saved_version; +- test_splinter_unclaim_page(cc, page); ++ /* Select the branch for its key, but skip it for a known filter miss. The ++ * latter must not inspect a branch that the query does not need to read. */ ++ const routing_config *filter_cfg = spl.cfg.trunk_node_cfg->filter_cfg; ++ routing_filter null_filter = NULL_ROUTING_FILTER; ++ routing_filter filter = NULL_ROUTING_FILTER; ++ uint32 fp = data_key_hash( ++ spl.cfg.data_cfg, key_buffer_key(&keybuf), filter_cfg->seed); ++ ASSERT_TRUE(SUCCESS( ++ routing_filter_add(cc, filter_cfg, &null_filter, &filter, &fp, 1, 0))); ++ uint64 found_values; ++ ASSERT_TRUE(SUCCESS(routing_filter_lookup( ++ cc, filter_cfg, &filter, key_buffer_key(&keybuf), &found_values))); ++ ASSERT_EQUAL_U(1, found_values); ++ ++ DECLARE_AUTO_KEY_BUFFER(miss_keybuf, data->hid); ++ for (uint64 i = 2; i < 1002; i++) { ++ test_key( ++ &miss_keybuf, TEST_RANDOM, i, 0, 0, data->workload_cfg->key_size, 0); ++ ASSERT_TRUE(SUCCESS(routing_filter_lookup(cc, ++ filter_cfg, ++ &filter, ++ key_buffer_key(&miss_keybuf), ++ &found_values))); ++ if (found_values == 0) { ++ break; ++ } ++ } ++ ASSERT_EQUAL_U(0, found_values); ++ ++ platform_status sync_rc[2]; ++ platform_status async_rc[2]; ++ platform_status range_rc[2]; ++ bool32 sync_miss_ok[2]; ++ bool32 async_miss_ok[2]; ++ for (uint64 invalid_field = 0; invalid_field < 2; invalid_field++) { ++ page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); ++ bundle = (test_trunk_bundle *)(page->data + offset % page_size); ++ bundle->maplet = filter; ++ if (invalid_field == 0) { ++ bundle->branches[0].format_version++; ++ } else { ++ bundle->branches[0].addr = 0; ++ } ++ test_splinter_unclaim_page(cc, page); ++ ++ lookup_result result; ++ lookup_result_init( ++ &result, spl.cfg.data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); ++ sync_rc[invalid_field] = ++ core_lookup(&spl, key_buffer_key(&keybuf), &result); ++ async_rc[invalid_field] = ++ async_call_sync_callback(cache_cleanup(cc), ++ core_lookup_async, ++ &spl, ++ key_buffer_key(&keybuf), ++ &result); ++ platform_status rc = ++ core_lookup(&spl, key_buffer_key(&miss_keybuf), &result); ++ sync_miss_ok[invalid_field] = ++ SUCCESS(rc) && !lookup_result_found(&result); ++ rc = async_call_sync_callback(cache_cleanup(cc), ++ core_lookup_async, ++ &spl, ++ key_buffer_key(&miss_keybuf), ++ &result); ++ async_miss_ok[invalid_field] = ++ SUCCESS(rc) && !lookup_result_found(&result); ++ lookup_result_deinit(&result); ++ ++ core_range_iterator itor; ++ range_rc[invalid_field] = core_range_iterator_init(&spl, ++ &itor, ++ greater_than_or_equal, ++ NEGATIVE_INFINITY_KEY, ++ less_than, ++ POSITIVE_INFINITY_KEY, ++ greater_than_or_equal, ++ NEGATIVE_INFINITY_KEY); ++ if (SUCCESS(range_rc[invalid_field])) { ++ core_range_iterator_deinit(&itor); ++ } ++ ++ /* Restore the owning reference before cleanup can release it. */ ++ page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); ++ bundle = (test_trunk_bundle *)(page->data + offset % page_size); ++ bundle->branches[0] = saved_branch; ++ bundle->maplet = saved_maplet; ++ test_splinter_unclaim_page(cc, page); ++ } ++ routing_filter_dec_ref(cc, &filter); + core_destroy(&spl); +- ASSERT_FALSE(SUCCESS(sync_rc)); +- ASSERT_FALSE(SUCCESS(async_rc)); +- ASSERT_FALSE(SUCCESS(range_rc)); ++ for (uint64 invalid_field = 0; invalid_field < 2; invalid_field++) { ++ ASSERT_TRUE(STATUS_IS_EQ(sync_rc[invalid_field], STATUS_INVALID_STATE)); ++ ASSERT_TRUE(STATUS_IS_EQ(async_rc[invalid_field], STATUS_INVALID_STATE)); ++ ASSERT_TRUE(STATUS_IS_EQ(range_rc[invalid_field], STATUS_INVALID_STATE)); ++ ASSERT_TRUE(sync_miss_ok[invalid_field]); ++ ASSERT_TRUE(async_miss_ok[invalid_field]); ++ } + } + + /* Persisted pivot statistics are sizing hints, but they still feed arithmetic +diff --git a/tests/unit/superblock_test.c b/tests/unit/superblock_test.c +index 6649f87..8f55e86 100644 +--- a/tests/unit/superblock_test.c ++++ b/tests/unit/superblock_test.c +@@ -715,7 +715,7 @@ CTEST2(superblock, test_bootstrap_geometry_metadata_budget) + /* A checksummed newer version is not permission to roll back to its peer. */ + CTEST2(superblock, test_unknown_structure_version_rejected) + { +- for (uint64 component = 0; component < 3; component++) { ++ for (uint64 component = 0; component < 4; component++) { + superblock_context ctx; + platform_status rc = superblock_context_init( + &ctx, data->ioh, &data->allocator_cfg, data->hid); +@@ -725,9 +725,13 @@ CTEST2(superblock, test_unknown_structure_version_rejected) + ctx.image->trunk_format_version++; + } else if (component == 1) { + ctx.image->allocator_format_version++; +- } else { ++ } else if (component == 2) { + superblock_log_cut(&ctx, TEST_LOG_L1); + ctx.image->tree.live_log.head.format_version++; ++ } else { ++ superblock_log_cut(&ctx, TEST_LOG_L1); ++ superblock_log_cut(&ctx, TEST_LOG_L2); ++ ctx.image->tree.sealed_log.head.format_version++; + } + ASSERT_TRUE(SUCCESS(superblock_make_durable(&ctx))); + superblock image; +diff --git a/tests/unit/trunk_split_test.c b/tests/unit/trunk_split_test.c +index ecaa338..bf0baeb 100644 +--- a/tests/unit/trunk_split_test.c ++++ b/tests/unit/trunk_split_test.c +@@ -237,6 +237,7 @@ CTEST2(trunk_split, split_scan_propagates_io_error) + data->cfg.btree_cfg, + &itor, + branch, ++ BTREE_BRANCH_FORMAT_VERSION, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, diff --git a/benchmarks/results-20261011/layering-20261012/release-build-config.fb6bc8eeb8390429253883c09491f5bd b/benchmarks/results-20261011/layering-20261012/release-build-config.fb6bc8eeb8390429253883c09491f5bd new file mode 100644 index 000000000..a84ace657 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/release-build-config.fb6bc8eeb8390429253883c09491f5bd @@ -0,0 +1,14 @@ +BUILD_ROOT=build-layering +MAKEFLAGS= -j8 --jobserver-auth=fifo:/tmp/splinter-disk-format-layering-build-1f6y1wss/GMfifo9 -- BUILD_MODE=release BUILD_ROOT=build-layering +BUILD_MODE=release +QT_ACCESSIBILITY=1 +CC = cc +DEPFLAGS = -MMD -MP +CFLAGS = -D_GNU_SOURCE -ggdb -Wall -pthread -Wfatal-errors -Werror -Wvla -DXXH_STATIC_LINKING_ONLY -fPIC -DSPLINTERDB_PLATFORM_DIR=platform_linux -O3 -ffast-math -flto -DPLATFORM_MEMORY_FAULT_INJECTION=0 +TEST_MEMORY_FAULT_DISABLED_CFLAGS = -DPLATFORM_MEMORY_FAULT_INJECTION_DISABLED_IN_THIS_FILE +INCLUDE = -I include -I src -I src/platform_linux -I tests +TARGET_ARCH = +LD = cc +LDFLAGS = -ggdb -pthread -O3 -ffast-math -flto +LIBS = -lm -lpthread -laio -lxxhash +AR = ar diff --git a/benchmarks/results-20261011/layering-20261012/release-corruption.log b/benchmarks/results-20261011/layering-20261012/release-corruption.log new file mode 100644 index 000000000..318fca27b --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/release-corruption.log @@ -0,0 +1,17 @@ +Starting disk-format-layering-final-release-corruption test +Running 1 CTests, suite name 'splinter', test case 'test_invalid_branch_reference_rejected'. +TEST 1/1 splinter:test_invalid_branch_reference_rejected Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +[OK] +RESULTS: 1 tests (1 ok, 0 failed, 0 skipped) ran in 77 ms diff --git a/benchmarks/results-20261011/layering-20261012/results.csv b/benchmarks/results-20261011/layering-20261012/results.csv new file mode 100644 index 000000000..7599ba9f7 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/results.csv @@ -0,0 +1,41 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +before,resident,0,0,warm,8,0,200000,200000,0.020,9863486.9,nan,nan,nan,0.000,nan,0.053,3757773.9,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9863486.9,0.020,0.000,0.020,0.000,0,0,nan,0,0,0.004,0,0,0,0,5 +before,resident,0,0,point:uniform@1,1,0,200000,19400472,15.000,1293343.0,0.712,1.072,5.184,128.124,0.750,14.999,1293419.1,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.004,0,0,0,0,0 +before,resident,0,0,point:uniform@8,8,0,200000,97294988,15.000,6486241.6,1.168,1.648,2.208,98.057,1.194,119.983,810905.5,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.004,0,0,0,0,0 +before,resident,0,0,range:uniform:1@1,1,0,200000,1751557,15.000,116768.2,8.320,12.160,35.328,164.053,8.538,15.000,116773.1,0,4096,0.00,0.00,8.128,11.648,8.258,nan,0.254,1751557,116768.2,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.004,0,0,0,0,0 +before,resident,0,0,range:uniform:1@8,8,0,200000,8517651,15.000,567835.1,12.928,19.200,29.440,454.042,14.045,119.984,70990.1,0,4096,0.00,0.00,12.416,18.176,13.395,nan,0.610,8517651,567835.1,15.000,0.061,15.061,0.012,0,307200,nan,162664448,28655616,0.004,0,0,0,0,0 +layered,resident,1,0,warm,8,0,200000,200000,0.021,9546301.1,nan,nan,nan,0.000,nan,0.058,3430002.2,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9546301.1,0.021,0.000,0.021,0.000,0,0,nan,0,0,0.005,0,0,0,0,2 +layered,resident,1,0,point:uniform@1,1,0,200000,18294339,15.000,1219598.6,0.776,1.104,4.288,137.976,0.796,15.004,1219337.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,1,0,point:uniform@8,8,0,200000,91477273,15.000,6098402.6,1.232,1.808,2.336,460.386,1.273,119.972,762488.4,0,8192,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,1,0,range:uniform:1@1,1,0,200000,1700424,15.000,113359.6,8.832,10.624,29.952,128.631,8.794,15.000,113363.8,0,4096,0.00,0.00,8.576,10.368,8.534,nan,0.233,1700424,113359.6,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,1,0,range:uniform:1@8,8,0,200000,8636692,15.000,575771.3,12.928,19.200,27.904,804.740,13.851,119.975,71987.5,0,4096,0.00,0.00,12.416,18.176,13.249,nan,0.563,8636692,575771.3,15.000,0.078,15.078,0.012,0,307200,nan,162664448,28655616,0.005,0,0,0,0,0 +layered,resident,2,0,warm,8,0,200000,200000,0.021,9320556.6,nan,nan,nan,0.000,nan,0.060,3346552.2,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9320556.6,0.021,0.000,0.021,0.000,0,0,nan,0,0,0.004,0,0,0,0,0 +layered,resident,2,0,point:uniform@1,1,0,200000,18684579,15.000,1245617.6,0.744,1.072,4.288,53.481,0.779,15.000,1245643.3,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.004,0,0,0,0,0 +layered,resident,2,0,point:uniform@8,8,0,200000,88865085,15.000,5924257.7,1.264,1.872,2.656,124.633,1.309,119.984,740640.6,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.004,0,0,1,0,0 +layered,resident,2,0,range:uniform:1@1,1,0,200000,1748496,15.000,116564.9,8.320,12.672,36.352,59.360,8.552,15.000,116566.9,0,4096,0.00,0.00,8.128,12.160,8.297,nan,0.229,1748496,116564.9,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.004,0,0,0,0,0 +layered,resident,2,0,range:uniform:1@8,8,0,200000,8487884,15.000,565851.4,12.928,19.200,28.416,484.738,14.093,119.982,70743.0,0,4096,0.00,0.00,12.416,18.176,13.482,nan,0.572,8487884,565851.4,15.000,0.095,15.095,0.012,0,307200,nan,162664448,28655616,0.004,0,0,0,0,0 +before,resident,3,0,warm,8,0,200000,200000,0.020,9802224.9,nan,nan,nan,0.000,nan,0.055,3619581.9,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9802224.9,0.020,0.000,0.020,0.000,0,0,nan,0,0,0.005,0,0,0,0,4 +before,resident,3,0,point:uniform@1,1,0,200000,18557383,15.000,1237143.6,0.776,1.104,1.328,47.715,0.783,15.000,1237171.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,3,0,point:uniform@8,8,0,200000,89708915,15.000,5980510.1,1.264,1.840,2.272,102.629,1.296,119.984,747677.1,0,4096,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,3,0,range:uniform:1@1,1,0,200000,1718133,15.000,114540.6,8.576,12.160,34.304,166.928,8.704,15.000,114544.2,0,4096,0.00,0.00,8.320,11.904,8.434,nan,0.244,1718133,114540.6,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,3,0,range:uniform:1@8,8,0,200000,8717825,15.000,581179.0,12.672,18.688,26.880,111.103,13.722,119.981,72660.1,0,8192,0.00,0.00,12.160,17.664,13.109,nan,0.574,8717825,581179.0,15.000,0.074,15.075,0.012,0,311296,nan,162664448,28655616,0.005,0,0,0,0,0 +layered,resident,4,0,warm,8,1,200000,200000,0.020,9835214.4,nan,nan,nan,0.000,nan,0.054,3687179.7,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9835214.4,0.020,0.000,0.020,0.000,0,0,nan,0,0,0.005,0,0,0,0,3 +layered,resident,4,0,point:uniform@1,1,1,200000,18082748,15.000,1205501.3,0.792,1.136,1.456,49.757,0.805,15.000,1205485.5,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,4,0,point:uniform@8,8,1,200000,89027547,15.000,5935084.1,1.264,1.872,2.272,104.324,1.308,119.983,741998.5,0,8192,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,4,0,range:uniform:1@1,1,1,200000,1711833,15.000,114120.2,8.576,11.904,34.304,63.177,8.735,15.000,114120.5,0,4096,0.00,0.00,8.320,11.392,8.465,nan,0.244,1711833,114120.2,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,4,0,range:uniform:1@8,8,1,200000,8249067,15.000,549929.8,13.184,19.200,27.904,97.770,14.502,119.986,68750.2,0,8192,0.00,0.00,12.672,18.688,13.872,nan,0.588,8249067,549929.8,15.000,0.085,15.085,0.013,0,307200,nan,162664448,28655616,0.005,0,0,0,0,0 +before,resident,5,0,warm,8,1,200000,200000,0.021,9613963.3,nan,nan,nan,0.000,nan,0.054,3732666.4,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9613963.3,0.021,0.000,0.021,0.000,0,0,nan,0,0,0.005,0,0,0,0,1 +before,resident,5,0,point:uniform@1,1,1,200000,18009054,15.000,1200582.3,0.792,1.136,1.424,146.343,0.808,15.000,1200610.4,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,1,0,0 +before,resident,5,0,point:uniform@8,8,1,200000,85184542,15.000,5678883.7,1.328,1.968,3.616,447.230,1.367,119.928,710295.4,0,8192,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,5,0,range:uniform:1@1,1,1,200000,1686147,15.000,112408.4,8.832,11.392,16.256,176.396,8.869,14.999,112415.4,0,4096,0.00,0.00,8.576,11.136,8.589,nan,0.253,1686147,112408.4,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,5,0,range:uniform:1@8,8,1,200000,8008622,15.000,533900.3,13.696,20.224,29.952,143.274,14.937,119.989,66744.8,0,8192,0.00,0.00,13.184,19.200,14.267,nan,0.628,8008622,533900.3,15.000,0.060,15.061,0.013,0,311296,nan,162664448,28655616,0.005,0,0,0,0,0 +before,resident,6,0,warm,8,1,200000,200000,0.022,9071147.2,nan,nan,nan,0.000,nan,0.061,3270378.5,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9071147.2,0.022,0.000,0.022,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,6,0,point:uniform@1,1,1,200000,18476511,15.000,1231745.7,0.776,1.104,1.392,46.547,0.787,15.000,1231771.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,6,0,point:uniform@8,8,1,200000,85698129,15.000,5713124.2,1.328,1.904,2.400,8025.143,1.358,119.976,714293.5,0,8192,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,4,0,0 +before,resident,6,0,range:uniform:1@1,1,1,200000,1686461,15.000,112429.6,8.832,10.368,14.976,42.028,8.868,15.000,112428.2,0,4096,0.00,0.00,8.576,10.112,8.588,nan,0.253,1686461,112429.6,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +before,resident,6,0,range:uniform:1@8,8,1,200000,8282728,15.000,552174.0,13.440,19.200,27.904,456.716,14.443,119.978,69035.4,0,4096,0.00,0.00,12.672,18.176,13.803,nan,0.600,8282728,552174.0,15.000,0.080,15.080,0.013,0,307200,nan,162664448,28655616,0.005,0,0,0,0,0 +layered,resident,7,0,warm,8,1,200000,200000,0.022,9187920.6,nan,nan,nan,0.000,nan,0.057,3538069.6,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,9187920.6,0.022,0.000,0.022,0.000,0,0,nan,0,0,0.005,0,0,0,0,6 +layered,resident,7,0,point:uniform@1,1,1,200000,18331364,15.000,1222069.7,0.776,1.136,1.488,49.060,0.793,15.000,1222074.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,11,0,0 +layered,resident,7,0,point:uniform@8,8,1,200000,85169199,15.000,5677861.3,1.328,1.936,2.976,1576.707,1.367,119.978,709873.0,0,8192,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,7,0,range:uniform:1@1,1,1,200000,1684540,15.000,112300.7,8.832,10.368,15.488,66.912,8.874,15.000,112299.5,0,4096,0.00,0.00,8.576,10.112,8.607,nan,0.240,1684540,112300.7,15.000,0.000,15.000,0.000,0,0,nan,0,0,0.005,0,0,0,0,0 +layered,resident,7,0,range:uniform:1@8,8,1,200000,8591609,15.000,572765.8,12.672,19.200,27.392,463.564,13.923,119.977,71610.5,0,4096,0.00,0.00,12.160,18.176,13.320,nan,0.564,8591609,572765.8,15.000,0.072,15.073,0.012,0,311296,nan,162664448,28655616,0.005,0,0,0,0,0 diff --git a/benchmarks/results-20261011/layering-20261012/run_job.py b/benchmarks/results-20261011/layering-20261012/run_job.py new file mode 100644 index 000000000..299c91084 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/run_job.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Coordinate this campaign's builds/tests and disk-backed benchmarks.""" +import argparse +import fcntl +import json +import os +from pathlib import Path +import subprocess +import tempfile +import time + +ROOT = Path(__file__).resolve().parent.parent +p = argparse.ArgumentParser() +p.add_argument('mode', choices=['build', 'test', 'benchmark']) +p.add_argument('--name', required=True) +p.add_argument('--db-dir', type=Path) +p.add_argument('command', nargs=argparse.REMAINDER) +a = p.parse_args() +command = a.command[1:] if a.command[:1] == ['--'] else a.command +if not command: + p.error('command required after --') +if a.mode == 'benchmark': + if a.db_dir is None: + p.error('benchmark requires --db-dir') + a.db_dir.mkdir(parents=True, exist_ok=True) + db_dir = a.db_dir.resolve() + if not db_dir.is_relative_to(ROOT / 'benchmarks'): + p.error('benchmark database directory must be under the campaign benchmarks directory') + fs = subprocess.check_output(['findmnt', '-n', '-o', 'FSTYPE,SOURCE', '-T', str(db_dir)], text=True).strip() + if fs.split()[0] in {'tmpfs', 'ramfs'} or not fs.split()[1].startswith('/dev/'): + p.error('benchmark storage must be a verified disk filesystem: ' + fs) + bench_lock = open(ROOT / 'coordination' / 'benchmark.lock', 'a+') + fcntl.flock(bench_lock, fcntl.LOCK_EX) + print('Benchmark database storage:', fs, str(db_dir), flush=True) +machine_lock = open(ROOT / 'coordination' / 'machine.lock', 'a+') +fcntl.flock(machine_lock, fcntl.LOCK_EX if a.mode == 'benchmark' else fcntl.LOCK_SH) +record = dict(name=a.name, mode=a.mode, cwd=os.getcwd(), command=command, + started=time.time(), pid=os.getpid()) +if a.db_dir is not None: + record['db_dir'] = str(a.db_dir.resolve()) +print('Starting', a.name, a.mode, flush=True) +try: + # Tool commands run in distinct PID namespaces; GNU make's FIFO names + # otherwise collide when several builds choose /tmp/GMfifo8. + with tempfile.TemporaryDirectory(prefix='splinter-' + a.name + '-', dir='/tmp') as temporary: + environment = dict(os.environ, TMPDIR=temporary) + result = subprocess.run(command, env=environment) + record['returncode'] = result.returncode +finally: + record['finished'] = time.time() + with open(ROOT / 'coordination' / 'jobs.jsonl', 'a') as journal: + fcntl.flock(journal, fcntl.LOCK_EX) + journal.write(json.dumps(record) + '\n') +raise SystemExit(result.returncode) diff --git a/benchmarks/results-20261011/layering-20261012/summary.csv b/benchmarks/results-20261011/layering-20261012/summary.csv new file mode 100644 index 000000000..365595185 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/summary.csv @@ -0,0 +1,5 @@ +phase,threads,samples_per_variant,before_ops_s,layered_ops_s,throughput_change_pct,before_min_ops_s,before_max_ops_s,layered_min_ops_s,layered_max_ops_s,before_cpu_us_op,layered_cpu_us_op,cpu_us_op_change_pct,before_read_B_op,layered_read_B_op,before_write_B_op,layered_write_B_op,before_p99_us,layered_p99_us +point:uniform@1,1,4,1240703.65,1223196.8,-1.411042032478893,1200582.3,1293343.0,1205501.3,1245617.6,0.8065462778214102,0.8176837849638405,1.3808887907231115,0.0,0.0,0.0,0.0,1.104,1.112 +point:uniform@8,8,4,5964689.9,5908901.425,-0.9353122448159557,5678883.7,6486241.6,5677861.3,6098402.6,1.3446285441466928,1.3545212332464787,0.7357191056853507,0.0,0.0,0.0,0.0,1.8399999999999999,1.872 +range:uniform:1@1,1,4,114036.7,114086.35,0.04353861520021152,112408.4,116768.2,112300.7,116564.9,8.771002412054917,8.766793940595583,-0.04798164749731937,0.0,0.0,0.0,0.0,11.52,11.392 +range:uniform:1@8,8,4,558772.1,566079.5750000001,1.3077737775383014,533900.3,581179.0,549929.8,575771.3,14.329259098129281,14.134208517478168,-1.3612049256376269,0.0,0.0,0.0,0.0,19.328,19.2 diff --git a/benchmarks/results-20261011/layering-20261012/summary.txt b/benchmarks/results-20261011/layering-20261012/summary.txt new file mode 100644 index 000000000..a58575acb --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/summary.txt @@ -0,0 +1,5 @@ +phase | before ops/s | layered ops/s | throughput change | CPU us/op change +point:uniform@1 | 1240704 | 1223197 | -1.41% | +1.38% +point:uniform@8 | 5964690 | 5908901 | -0.94% | +0.74% +range:uniform:1@1 | 114037 | 114086 | +0.04% | -0.05% +range:uniform:1@8 | 558772 | 566080 | +1.31% | -1.36% diff --git a/benchmarks/results-20261011/layering-20261012/test_format_layering.py b/benchmarks/results-20261011/layering-20261012/test_format_layering.py new file mode 100644 index 000000000..1597fc074 --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/test_format_layering.py @@ -0,0 +1,14 @@ +import json,subprocess,time +from pathlib import Path +root=Path(__file__).resolve().parent.parent +out=root/'coordination' +base=Path('build-layering/debug/bin') +commands=[('btree',[str(base/'unit/btree_test')]),('superblock',[str(base/'unit/superblock_test')]),('splinter',[str(base/'unit/splinter_test'),'--db-location','.cache/format-tests/layering-splinter.db','--cache-capacity-mib','128','--db-capacity-gib','4']),('quick',[str(base/'unit/splinterdb_quick_test')]),('trunk-async',[str(base/'unit/trunk_async_test')]),('filter',[str(base/'driver_test'),'filter_test','--db-location','.cache/format-tests/layering-filter.db','--cache-capacity-mib','64','--db-capacity-gib','4']),('log',[str(base/'driver_test'),'log_test','--db-location','.cache/format-tests/layering-log.db','--cache-capacity-mib','64','--db-capacity-gib','4','--num-inserts','10000'])] +records=[] +for name,cmd in commands: + print(name,flush=True) + record=dict(name=name,command=cmd,started=time.time());records.append(record) + with (out/f'disk-format-layering-debug-{name}.log').open('w') as f:r=subprocess.run(cmd,stdout=f,stderr=subprocess.STDOUT) + record.update(returncode=r.returncode,finished=time.time()) + (out/'disk-format-layering-tests.json').write_text(json.dumps(records,indent=2)+'\n') + if r.returncode:raise SystemExit(r.returncode) diff --git a/benchmarks/results-20261011/layering-20261012/validation.json b/benchmarks/results-20261011/layering-20261012/validation.json new file mode 100644 index 000000000..7b907e8ca --- /dev/null +++ b/benchmarks/results-20261011/layering-20261012/validation.json @@ -0,0 +1,87 @@ +[ + { + "name": "btree", + "command": [ + "build-layering/debug/bin/unit/btree_test" + ], + "started": 1791841847.266642, + "returncode": 0, + "finished": 1791841847.2800844 + }, + { + "name": "superblock", + "command": [ + "build-layering/debug/bin/unit/superblock_test" + ], + "started": 1791841847.2801974, + "returncode": 0, + "finished": 1791841847.3184545 + }, + { + "name": "splinter", + "command": [ + "build-layering/debug/bin/unit/splinter_test", + "--db-location", + ".cache/format-tests/layering-splinter.db", + "--cache-capacity-mib", + "128", + "--db-capacity-gib", + "4" + ], + "started": 1791841847.3186405, + "returncode": 0, + "finished": 1791841919.3461256 + }, + { + "name": "quick", + "command": [ + "build-layering/debug/bin/unit/splinterdb_quick_test" + ], + "started": 1791841919.3462799, + "returncode": 0, + "finished": 1791841928.5821726 + }, + { + "name": "trunk-async", + "command": [ + "build-layering/debug/bin/unit/trunk_async_test" + ], + "started": 1791841928.5823367, + "returncode": 0, + "finished": 1791841928.5837073 + }, + { + "name": "filter", + "command": [ + "build-layering/debug/bin/driver_test", + "filter_test", + "--db-location", + ".cache/format-tests/layering-filter.db", + "--cache-capacity-mib", + "64", + "--db-capacity-gib", + "4" + ], + "started": 1791841928.5838318, + "returncode": 0, + "finished": 1791841932.665015 + }, + { + "name": "log", + "command": [ + "build-layering/debug/bin/driver_test", + "log_test", + "--db-location", + ".cache/format-tests/layering-log.db", + "--cache-capacity-mib", + "64", + "--db-capacity-gib", + "4", + "--num-inserts", + "10000" + ], + "started": 1791841932.6694396, + "returncode": 0, + "finished": 1791841933.154115 + } +] diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md index 1d08e0526..7e5199c48 100644 --- a/docs/on-disk-format.md +++ b/docs/on-disk-format.md @@ -106,3 +106,76 @@ committed under `benchmarks/results-20261011/`. Full campaign artifacts remain at `.cache/projects-20261011/benchmarks/disk-format/` in the main checkout; benchmark database files were on a verified disk filesystem, with the shared benchmark lock excluding other benchmarks, builds and tests. + +## Format ownership follow-up (2026-10-12) + +Branch compatibility now belongs to the B-tree. Point queries pass the selected +branch's persisted version to synchronous/asynchronous B-tree entry points; +`trunk_ondisk_bundle_readable()` no longer scans every branch. The B-tree rejects +unsupported versions or null roots before reading pages. Range initialization, +compaction, root-height/count queries, and allocation recovery preserve the +version alongside the address. Memtables do not acquire a format requirement. + +Reference acquisition needs validation before iterator initialization: a null +root would otherwise target extent zero through the mini allocator. Existing +bundle-deserialization and range-collection loops therefore call the B-tree's +root validator before admitting references. Failed collection discards the +unreferenced partial bundle, preserving correct cleanup. These checks add no +branch scan to the ordinary point-query bundle decoder. + +The ownership audit also moved trunk/allocator version constants and support +checks to their modules. Superblock reading still selects the newest valid +record before invoking owner validators for trunk, allocator, and both logs; +it never silently falls back to an older record with supported components. +Routing-filter instance-capacity calculation now lives in the routing-filter +module instead of reconstructing its encoding configuration in the trunk. +Routing-filter descriptor validation and mini-allocator private-header +validation were already correctly owned. No on-disk layout or version changed +in this follow-up. + +Direct B-tree regressions cover old/future versions and null roots before any +cache access, including safe iterator cleanup. The core corruption regression +covers synchronous/asynchronous point hits and ranges, and verifies that +routing-filter misses succeed without inspecting the irrelevant branch. + +Final debug validation passed all 108 cases across B-tree (11), superblock (16), +core (33), quick API (47), and async trunk (1), plus the functional filter and +log suites. Release builds of all tests and the inspection tool also passed; +the earlier release correctness run passed those same 108 cases before the +last reference-admission and corruption-test extensions. + +The follow-up resident comparison uses the prior final-scalar executable and +the new reader on copies of the same format-8 image, avoiding independently +built tree shapes. Every private copy is explicitly fsynced before opening. +The runner uses O_DIRECT on verified ext4 storage, 200,000 keys, 128 MiB cache, +24-byte keys, 100-byte values, WAL disabled, CPU affinity 0-11, and 15-second +phases. ABBA then BAAB gives four samples per reader for point queries and +length-1 ranges at one/eight threads. The shared benchmark and machine locks +exclude simultaneous benchmarks, builds, and tests. This compares the complete +ownership refactor, not a one-line validation-loop ablation, and does not +remeasure the original main branch. + +| Ownership refactor versus prior scalar reader | Throughput | CPU/op | +| --- | ---: | ---: | +| Point, 1 reader | -1.41% | +1.38% | +| Point, 8 readers | -0.94% | +0.74% | +| Length-1 range, 1 reader | +0.04% | -0.05% | +| Length-1 range, 8 readers | +1.31% | -1.36% | + +All eight runs passed with zero errors/violations. Timed read/write bytes were +zero: these are resident CPU measurements, though the database files remain +on physical disk. Every before/after sample interval overlaps, both overall +and within each block. For example, eight-reader point throughput changed +-3.56% in ABBA but +1.94% in BAAB; the prior reader's block means drifted +-8.62%. The measurements establish neither a clear benefit nor a reliably +attributable regression. Removing the eager loop has not demonstrated a cure +for the original project's overhead; neutrality versus the original main +branch remains unestablished. Disk-limited and insertion matrices were not +repeated for this ownership-only follow-up. + +The final release corruption regression also passes with the reference +admission guards and filter-negative cases. Compact raw measurements, block +summaries, test logs, hashes, commands, and the measured source patch are in +[`benchmarks/results-20261011/layering-20261012/`](../benchmarks/results-20261011/layering-20261012/). +Full fixtures and preserved measured executables remain in the campaign's +`benchmarks/disk-format/resident-layering-20261012/` directory. diff --git a/src/btree.c b/src/btree.c index a9d6009b2..b95c3a83c 100644 --- a/src/btree.c +++ b/src/btree.c @@ -1679,6 +1679,26 @@ btree_create(cache *cc, return STATUS_OK; } +/* The B-tree owns branch format compatibility, independent of its container. */ +platform_status +btree_validate_root(uint64 root_addr, uint16 format_version, page_type type) +{ + if (type == PAGE_TYPE_BRANCH + && format_version != BTREE_BRANCH_FORMAT_VERSION) + { + platform_error_log("Unsupported branch format version %u at %lu; " + "this build requires version %u.\n", + format_version, + root_addr, + BTREE_BRANCH_FORMAT_VERSION); + return STATUS_INVALID_STATE; + } + if (root_addr == 0) { + return STATUS_INVALID_STATE; + } + return STATUS_OK; +} + void btree_inc_ref(cache *cc, const btree_config *cfg, uint64 root_addr) { @@ -1696,8 +1716,15 @@ platform_status btree_recover_allocations(cache *cc, const btree_config *cfg, uint64 root_addr, + uint16 format_version, page_type type) { + platform_status version_rc = + btree_validate_root(root_addr, format_version, type); + if (!SUCCESS(version_rc)) { + return version_rc; + } + return mini_recover_references( cc, btree_root_to_meta_addr(cfg, root_addr, 0), type); } @@ -1717,9 +1744,16 @@ platform_status btree_get_root_height(cache *cc, const btree_config *cfg, uint64 root_addr, + uint16 format_version, page_type type, uint32 *height) { + platform_status version_rc = + btree_validate_root(root_addr, format_version, type); + if (!SUCCESS(version_rc)) { + return version_rc; + } + if (height == NULL || (type != PAGE_TYPE_BRANCH && type != PAGE_TYPE_MEMTABLE)) { @@ -3040,27 +3074,39 @@ btree_lookup_with_ref(cache *cc, // IN } platform_status -btree_lookup(cache *cc, // IN - btree_config *cfg, // IN - uint64 root_addr, // IN - page_type type, // IN - key target, // IN - lookup_result *result) // OUT +btree_lookup(cache *cc, // IN + btree_config *cfg, // IN + uint64 root_addr, // IN + uint16 format_version, // IN + page_type type, // IN + key target, // IN + lookup_result *result) // OUT { lookup_result_reset(result); return btree_lookup_and_merge( - cc, cfg, root_addr, type, target, result, NULL); + cc, cfg, root_addr, format_version, type, target, result, NULL); } platform_status -btree_lookup_and_merge(cache *cc, // IN - const btree_config *cfg, // IN - uint64 root_addr, // IN - page_type type, // IN - key target, // IN - lookup_result *result, // IN/OUT - bool32 *local_found) // OUT +btree_lookup_and_merge(cache *cc, // IN + const btree_config *cfg, // IN + uint64 root_addr, // IN + uint16 format_version, // IN + page_type type, // IN + key target, // IN + lookup_result *result, // IN/OUT + bool32 *local_found) // OUT { + if (local_found != NULL) { + *local_found = FALSE; + } + + platform_status version_rc = + btree_validate_root(root_addr, format_version, type); + if (!SUCCESS(version_rc)) { + return version_rc; + } + btree_node node; key found_key; message local_data; @@ -3068,10 +3114,6 @@ btree_lookup_and_merge(cache *cc, // IN log_trace_key(target, "btree_lookup"); - if (local_found != NULL) { - *local_found = FALSE; - } - rc = btree_lookup_with_ref( cc, cfg, root_addr, type, target, &node, &found_key, &local_data); if (!SUCCESS(rc)) { @@ -3116,6 +3158,12 @@ btree_lookup_and_merge_async(btree_lookup_async_state *state) { async_begin(state, 0); + platform_status version_rc = + btree_validate_root(state->root_addr, state->format_version, state->type); + if (!SUCCESS(version_rc)) { + async_return(state, version_rc); + } + async_await_subroutine(state, btree_lookup_with_ref_async); if (!SUCCESS(state->__async_result)) { @@ -3660,6 +3708,7 @@ btree_iterator_find_end_addr_async(btree_iterator_async_state *state, state->itor->cc, state->itor->cfg, state->itor->root_addr, + state->format_version, state->itor->page_type, state->itor->max_key, NULL, @@ -4670,6 +4719,7 @@ find_btree_node_and_get_idx_bounds_async(btree_iterator_async_state *state, state->itor->cc, state->itor->cfg, state->itor->root_addr, + state->format_version, state->itor->page_type, state->target, NULL, @@ -4874,6 +4924,7 @@ btree_iterator_init_common(cache *cc, const btree_config *cfg, btree_iterator *itor, uint64 root_addr, + uint16 format_version, page_type page_type, comparison min_key_comparison, key min_key, @@ -4885,7 +4936,12 @@ btree_iterator_init_common(cache *cc, uint32 prefetch_lookahead, key *normalized_start_key) { - platform_assert(root_addr != 0); + ZERO_CONTENTS(itor); + platform_status version_rc = + btree_validate_root(root_addr, format_version, page_type); + if (!SUCCESS(version_rc)) { + return version_rc; + } debug_assert(page_type == PAGE_TYPE_MEMTABLE || page_type == PAGE_TYPE_BRANCH); @@ -4908,7 +4964,6 @@ btree_iterator_init_common(cache *cc, start_key = max_key; } - ZERO_CONTENTS(itor); itor->cc = cc; itor->cfg = cfg; itor->root_addr = root_addr; @@ -4940,6 +4995,7 @@ btree_iterator_init(cache *cc, const btree_config *cfg, btree_iterator *itor, uint64 root_addr, + uint16 format_version, page_type page_type, comparison min_key_comparison, key min_key, @@ -4955,6 +5011,7 @@ btree_iterator_init(cache *cc, cfg, itor, root_addr, + format_version, page_type, min_key_comparison, min_key, @@ -4993,6 +5050,7 @@ btree_iterator_init_async(btree_iterator_async_state *state) state->cfg, state->itor, state->root_addr, + state->format_version, state->type, state->min_key_comparison, state->min_key, @@ -5523,6 +5581,7 @@ platform_status btree_count_in_range(cache *cc, const btree_config *cfg, uint64 root_addr, + uint16 format_version, key min_key, key max_key, btree_pivot_stats *stats) @@ -5532,6 +5591,12 @@ btree_count_in_range(cache *cc, debug_assert(!key_is_null(min_key) && !key_is_null(max_key)); memset(stats, 0, sizeof(*stats)); + platform_status version_rc = + btree_validate_root(root_addr, format_version, PAGE_TYPE_BRANCH); + if (!SUCCESS(version_rc)) { + return version_rc; + } + platform_status rc = btree_get_rank(cc, cfg, root_addr, min_key, &min_stats); if (!SUCCESS(rc)) { return rc; @@ -5560,6 +5625,7 @@ void btree_count_in_range_by_iterator(cache *cc, btree_config *cfg, uint64 root_addr, + uint16 format_version, key min_key, key max_key, btree_pivot_stats *stats) @@ -5570,6 +5636,7 @@ btree_count_in_range_by_iterator(cache *cc, cfg, &btree_itor, root_addr, + format_version, PAGE_TYPE_BRANCH, greater_than_or_equal, min_key, diff --git a/src/btree.h b/src/btree.h index 96b495ef1..adc649344 100644 --- a/src/btree.h +++ b/src/btree.h @@ -33,7 +33,11 @@ */ #define BTREE_MAX_HEIGHT (8) -/* Immutable branch version is stored in its trunk branch_ref. */ +/* + * Immutable branch versions travel with the root address. Query and recovery + * entry points validate the version before interpreting branch pages. The + * version argument is ignored for PAGE_TYPE_MEMTABLE (an in-memory structure). + */ #define BTREE_BRANCH_FORMAT_VERSION (1) /* @@ -298,6 +302,12 @@ btree_create(cache *cc, page_type type, uint64 *root_addr); +/* Validate persisted roots before acquiring references. Query entry points + * also validate before reading pages. Refcount operations below require a + * validated root or one freshly created by this module. */ +platform_status +btree_validate_root(uint64 root_addr, uint16 format_version, page_type type); + void btree_inc_ref(cache *cc, const btree_config *cfg, uint64 root_addr); @@ -311,6 +321,7 @@ platform_status btree_get_root_height(cache *cc, const btree_config *cfg, uint64 root_addr, + uint16 format_version, page_type type, uint32 *height); @@ -322,6 +333,7 @@ platform_status btree_recover_allocations(cache *cc, const btree_config *cfg, uint64 root_addr, + uint16 format_version, page_type type); void @@ -331,6 +343,7 @@ platform_status btree_lookup(cache *cc, btree_config *cfg, uint64 root_addr, + uint16 format_version, page_type type, key target, lookup_result *result); @@ -345,6 +358,7 @@ platform_status btree_lookup_and_merge(cache *cc, const btree_config *cfg, uint64 root_addr, + uint16 format_version, page_type type, key target, lookup_result *result, @@ -355,6 +369,7 @@ DEFINE_ASYNC_STATE(btree_lookup_async_state, 3, param, cache *, cc, param, const btree_config *, cfg, param, uint64, root_addr, + param, uint16, format_version, param, page_type, type, param, key, target, param, lookup_result *, result, @@ -377,14 +392,23 @@ btree_lookup_and_merge_async_state_init(btree_lookup_async_state *state, cache *cc, const btree_config *cfg, uint64 root_addr, - page_type type, - key target, - lookup_result *result, - async_callback_fn callback, - void *callback_arg) + uint16 format_version, + page_type type, + key target, + lookup_result *result, + async_callback_fn callback, + void *callback_arg) { - btree_lookup_async_state_init( - state, cc, cfg, root_addr, type, target, result, callback, callback_arg); + btree_lookup_async_state_init(state, + cc, + cfg, + root_addr, + format_version, + type, + target, + result, + callback, + callback_arg); } async_status @@ -403,6 +427,7 @@ btree_iterator_init(cache *cc, const btree_config *cfg, btree_iterator *itor, uint64 root_addr, + uint16 format_version, page_type page_type, comparison min_key_comparison, key min_key, @@ -420,6 +445,7 @@ DEFINE_ASYNC_STATE(btree_iterator_async_state, 6, param, const btree_config *, cfg, param, btree_iterator *, itor, param, uint64, root_addr, + param, uint16, format_version, param, page_type, type, param, comparison, min_key_comparison, param, key, min_key, @@ -446,7 +472,6 @@ DEFINE_ASYNC_STATE(btree_iterator_async_state, 6, local, uint64, next_addr, local, uint64, prev_addr, local, uint64, expected_page_no, - local, uint64, num_entries, local, key, first_key, local, key, last_key) // clang-format on @@ -508,6 +533,7 @@ platform_status btree_count_in_range(cache *cc, const btree_config *cfg, uint64 root_addr, + uint16 format_version, key min_key, key max_key, btree_pivot_stats *stats); @@ -516,6 +542,7 @@ void btree_count_in_range_by_iterator(cache *cc, btree_config *cfg, uint64 root_addr, + uint16 format_version, key min_key, key max_key, btree_pivot_stats *stats); diff --git a/src/core.c b/src/core.c index d20844f20..8951084ed 100644 --- a/src/core.c +++ b/src/core.c @@ -1247,6 +1247,7 @@ core_memtable_iterator_init(core_handle *spl, spl->cfg.btree_cfg, itor, root_addr, + 0, PAGE_TYPE_MEMTABLE, min_key_comparison, min_key, @@ -1832,8 +1833,14 @@ core_memtable_lookup(core_handle *spl, page_type type = memtable_is_compacted ? PAGE_TYPE_BRANCH : PAGE_TYPE_MEMTABLE; - return btree_lookup_and_merge( - cc, cfg, root_addr, type, target, result, NULL); + return btree_lookup_and_merge(cc, + cfg, + root_addr, + BTREE_BRANCH_FORMAT_VERSION, + type, + target, + result, + NULL); } static platform_status @@ -1920,6 +1927,7 @@ core_start_btree_iterator_init_async( core_btree_iterator_init_async_context *ctxt, btree_iterator *itor, uint64 root_addr, + uint16 format_version, page_type page_type, comparison min_key_comparison, key min_key, @@ -1935,6 +1943,7 @@ core_start_btree_iterator_init_async( spl->cfg.btree_cfg, itor, root_addr, + format_version, page_type, min_key_comparison, min_key, @@ -2171,6 +2180,8 @@ core_range_iterator_init(core_handle *spl, } range_itor->branch[range_itor->num_branches].addr = root_addr; + range_itor->branch[range_itor->num_branches].format_version = + compacted ? BTREE_BRANCH_FORMAT_VERSION : 0; range_itor->branch[range_itor->num_branches].type = compacted ? PAGE_TYPE_BRANCH : PAGE_TYPE_MEMTABLE; range_itor->num_branches++; @@ -2278,6 +2289,7 @@ core_range_iterator_init(core_handle *spl, &init_ctxt[i], btree_itor, branch_addr, + range_itor->branch[branch_no].format_version, page_type, range_itor->local_min_key_comparison, key_buffer_key(&range_itor->local_min_key), @@ -2693,6 +2705,7 @@ core_fetch_and_update_catch_up(core_handle *spl, rc = btree_lookup_and_merge(spl->cc, spl->cfg.btree_cfg, refs[i].root_addr, + BTREE_BRANCH_FORMAT_VERSION, refs[i].type, tuple_key, catch_up_result, @@ -4758,6 +4771,7 @@ core_print_lookup(core_handle *spl, key target, platform_log_handle *log_handle) rc = btree_lookup(spl->cc, spl->cfg.btree_cfg, root_addr, + 0, PAGE_TYPE_MEMTABLE, target, &lookup); diff --git a/src/rc_allocator.c b/src/rc_allocator.c index aba2f3be4..d9d108800 100644 --- a/src/rc_allocator.c +++ b/src/rc_allocator.c @@ -17,6 +17,19 @@ #include "platform_typed_alloc.h" #include "poison.h" +platform_status +rc_allocator_validate_format_version(uint16 format_version) +{ + if (format_version != RC_ALLOCATOR_FORMAT_VERSION) { + platform_error_log("Unsupported allocator format version %u; " + "this build requires version %u.\n", + format_version, + RC_ALLOCATOR_FORMAT_VERSION); + return STATUS_INVALID_STATE; + } + return STATUS_OK; +} + /* * The refcount map is the allocator's only durable structure. It lives in a * fixed run of extents starting at extent 1 (extent 0 is reserved for the diff --git a/src/rc_allocator.h b/src/rc_allocator.h index 91eccb721..954bb7730 100644 --- a/src/rc_allocator.h +++ b/src/rc_allocator.h @@ -15,6 +15,12 @@ #include "allocator.h" #include "util.h" +#define RC_ALLOCATOR_FORMAT_VERSION (1) + +/* Validate the allocator version stored in the database superblock. */ +platform_status +rc_allocator_validate_format_version(uint16 format_version); + /* *---------------------------------------------------------------------- * rc_allocator_stats -- diff --git a/src/routing_filter.c b/src/routing_filter.c index 4eb8a4bce..318e71f0c 100644 --- a/src/routing_filter.c +++ b/src/routing_filter.c @@ -252,6 +252,19 @@ routing_filter_get_params(const routing_config *cfg, return STATUS_OK; } +uint64 +routing_filter_max_fingerprints_for_filter(cache_config *cache_cfg, + const routing_filter *filter) +{ + debug_assert(routing_filter_descriptor_is_valid(filter)); + debug_assert(filter->addr != 0); + routing_config cfg = { + .fingerprint_size = filter->fingerprint_size, + .log_index_size = filter->log_index_size, + }; + return routing_filter_max_fingerprints(cache_cfg, &cfg); +} + static platform_status routing_get_header(cache *cc, const routing_config *cfg, diff --git a/src/routing_filter.h b/src/routing_filter.h index cac429fa7..3188318ec 100644 --- a/src/routing_filter.h +++ b/src/routing_filter.h @@ -184,6 +184,11 @@ routing_filter_max_fingerprints(cache_config *cache_cfg, return MIN(UINT32_MAX, MIN(index_limit, fingerprint_limit)); } +/* Capacity of an existing instance, independent of new-filter settings. */ +uint64 +routing_filter_max_fingerprints_for_filter(cache_config *cache_cfg, + const routing_filter *filter); + // clang-format off DEFINE_ASYNC_STATE(routing_filter_lookup_async_state, 2, param, cache *, cc, diff --git a/src/shard_log.c b/src/shard_log.c index 1983b2a66..e3610aa94 100644 --- a/src/shard_log.c +++ b/src/shard_log.c @@ -21,6 +21,19 @@ #include "platform_sort.h" #include "poison.h" +platform_status +shard_log_validate_format_version(uint16 format_version) +{ + if (format_version != SHARD_LOG_FORMAT_VERSION) { + platform_error_log("Unsupported log format version %u; " + "this build requires version %u.\n", + format_version, + SHARD_LOG_FORMAT_VERSION); + return STATUS_INVALID_STATE; + } + return STATUS_OK; +} + static const page_type shard_log_page_type_table[NUM_BLOB_BATCHES + 1] = { PAGE_TYPE_LOG, [1 ... NUM_BLOB_BATCHES] = PAGE_TYPE_BLOB, @@ -2189,8 +2202,9 @@ shard_log_recover_allocations(cache *cc, shard_log_config *cfg, log_head head) if (head.addr == 0) { return STATUS_OK; // no such log } - if (head.format_version != SHARD_LOG_FORMAT_VERSION) { - return STATUS_INVALID_STATE; + platform_status rc = shard_log_validate_format_version(head.format_version); + if (!SUCCESS(rc)) { + return rc; } shard_log_config read_cfg = *cfg; read_cfg.seed = head.checksum_seed; @@ -2202,10 +2216,9 @@ shard_log_recover_allocations(cache *cc, shard_log_config *cfg, log_head head) * page-header chain never reaches it. It has to be recorded on its own or * the extent is left looking free while the durable record still names it. */ - allocator *al = cache_get_allocator(cc); - uint64 meta_base = shard_log_extent_base(cc, head.meta_addr); - platform_status rc = - allocator_recovery_record_reference(al, meta_base, PAGE_TYPE_LOG); + allocator *al = cache_get_allocator(cc); + uint64 meta_base = shard_log_extent_base(cc, head.meta_addr); + rc = allocator_recovery_record_reference(al, meta_base, PAGE_TYPE_LOG); if (!SUCCESS(rc)) { return rc; } @@ -3015,9 +3028,10 @@ shard_log_iterator_create(cache *cc, if (itor_out == NULL) { return STATUS_BAD_PARAM; } - *itor_out = NULL; - if (head.format_version != SHARD_LOG_FORMAT_VERSION) { - return STATUS_INVALID_STATE; + *itor_out = NULL; + platform_status rc = shard_log_validate_format_version(head.format_version); + if (!SUCCESS(rc)) { + return rc; } shard_log_iterator *itor = TYPED_MALLOC(hid, itor); @@ -3026,7 +3040,7 @@ shard_log_iterator_create(cache *cc, "shard_log_iterator\n"); return STATUS_NO_MEMORY; } - platform_status rc = shard_log_iterator_init( + rc = shard_log_iterator_init( cc, cfg, hid, head.addr, head, first_needed_generation, itor); if (!SUCCESS(rc)) { platform_error_log("shard_log_iterator_create: shard_log_iterator_init " diff --git a/src/shard_log.h b/src/shard_log.h index b0a61e46b..c3a2f3552 100644 --- a/src/shard_log.h +++ b/src/shard_log.h @@ -20,6 +20,10 @@ #include "platform_mutex.h" #include "writeback_set.h" +/* Validate the version persisted in a log head before accessing its stream. */ +platform_status +shard_log_validate_format_version(uint16 format_version); + /* * Configuration structure to set up the sharded log sub-system. */ diff --git a/src/splinterdb.c b/src/splinterdb.c index 144fc9619..e830dd1b9 100644 --- a/src/splinterdb.c +++ b/src/splinterdb.c @@ -753,6 +753,7 @@ splinterdb_get_cost_stats(splinterdb *kvs, splinterdb_cost_stats *stats) status = btree_get_root_height(mt_ctxt->cc, mt_ctxt->cfg.btree_cfg, memtable_root_addr(&mt_ctxt->mt[mt_no]), + 0, PAGE_TYPE_MEMTABLE, &height); if (!SUCCESS(status)) { diff --git a/src/superblock.c b/src/superblock.c index da36f73a8..acb3fead1 100644 --- a/src/superblock.c +++ b/src/superblock.c @@ -8,6 +8,9 @@ */ #include "superblock.h" +#include "trunk.h" +#include "rc_allocator.h" +#include "shard_log.h" #include "poison.h" #define SUPERBLOCK_CSUM_SEED (0x53555045524231ULL) // "SUPERB1" @@ -130,21 +133,24 @@ superblock_check_versions(const superblock *sb) if (sb->format_version != SUPERBLOCK_FORMAT_VERSION) { return superblock_unsupported_version(sb->format_version); } - if (sb->trunk_format_version != TRUNK_FORMAT_VERSION - || sb->allocator_format_version != RC_ALLOCATOR_FORMAT_VERSION - || (!SUPERBLOCK_NO_LOG(sb->tree.live_log) - && sb->tree.live_log.head.format_version != SHARD_LOG_FORMAT_VERSION) - || (!SUPERBLOCK_NO_LOG(sb->tree.sealed_log) - && sb->tree.sealed_log.head.format_version - != SHARD_LOG_FORMAT_VERSION)) - { - platform_error_log("Unsupported on-disk structure version: " - "trunk=%u allocator=%u live-log=%u sealed-log=%u\n", - sb->trunk_format_version, - sb->allocator_format_version, - sb->tree.live_log.head.format_version, - sb->tree.sealed_log.head.format_version); - return STATUS_INVALID_STATE; + platform_status rc = trunk_validate_format_version(sb->trunk_format_version); + if (!SUCCESS(rc)) { + return rc; + } + rc = rc_allocator_validate_format_version(sb->allocator_format_version); + if (!SUCCESS(rc)) { + return rc; + } + if (!SUPERBLOCK_NO_LOG(sb->tree.live_log)) { + rc = shard_log_validate_format_version( + sb->tree.live_log.head.format_version); + if (!SUCCESS(rc)) { + return rc; + } + } + if (!SUPERBLOCK_NO_LOG(sb->tree.sealed_log)) { + return shard_log_validate_format_version( + sb->tree.sealed_log.head.format_version); } return STATUS_OK; } diff --git a/src/superblock.h b/src/superblock.h index 3343e91b9..fff86abab 100644 --- a/src/superblock.h +++ b/src/superblock.h @@ -48,9 +48,7 @@ * descriptors carry their format version and fingerprint size; v8 adds * independent structure versions and complete routing-filter/log read config. */ -#define SUPERBLOCK_FORMAT_VERSION (8) -#define TRUNK_FORMAT_VERSION (1) -#define RC_ALLOCATOR_FORMAT_VERSION (1) +#define SUPERBLOCK_FORMAT_VERSION (8) /* The two physical superblock copies live at pages 0 and 1. */ #define SUPERBLOCK_NUM_SLOTS (2) diff --git a/src/trunk.c b/src/trunk.c index e4c8b56e3..a89fa712f 100644 --- a/src/trunk.c +++ b/src/trunk.c @@ -1528,13 +1528,6 @@ trunk_ondisk_bundle_readable(trunk_ondisk_bundle *bundle, { return FALSE; } - for (uint64 i = 0; i < bundle->num_branches; i++) { - if (bundle->branches[i].addr == 0 - || bundle->branches[i].format_version != BTREE_BRANCH_FORMAT_VERSION) - { - return FALSE; - } - } if (!routing_filter_descriptor_is_valid(&bundle->maplet)) { return FALSE; } @@ -1904,6 +1897,13 @@ bundle_deserialize(bundle *bndl, platform_heap_id hid, trunk_ondisk_bundle *odb) bndl->maplet = odb->maplet; for (uint64 i = 0; i < odb->num_branches; i++) { + rc = btree_validate_root(odb->branches[i].addr, + odb->branches[i].format_version, + trunk_ondisk_bundle_branch_type(odb)); + if (!SUCCESS(rc)) { + bundle_deinit(bndl); + return rc; + } rc = vector_append(&bndl->branches, odb->branches[i]); platform_assert_status_ok(rc); } @@ -2337,8 +2337,11 @@ trunk_recover_bundle_refs(const trunk_context *context, bundle *bndl) page_type type = bundle_branch_type(bndl); for (uint64 i = 0; i < vector_length(&bndl->branches); i++) { branch_ref bref = vector_get(&bndl->branches, i); - platform_status rc = btree_recover_allocations( - context->cc, context->cfg->btree_cfg, branch_ref_addr(bref), type); + platform_status rc = btree_recover_allocations(context->cc, + context->cfg->btree_cfg, + branch_ref_addr(bref), + bref.format_version, + type); if (!SUCCESS(rc)) { return rc; } @@ -2973,13 +2976,14 @@ trunk_branch_merger_init(trunk_branch_merger *merger, static platform_status trunk_branch_merger_add_branch(trunk_branch_merger *merger, uint64 addr, + uint16 format_version, page_type type) { platform_assert(merger != NULL); platform_assert(addr != 0); platform_assert(PAGE_TYPE_FIRST <= type && type < NUM_PAGE_TYPES); - trunk_branch_info branch = {addr, type}; + trunk_branch_info branch = {addr, type, format_version}; platform_status rc = vector_append(&merger->branches, branch); if (!SUCCESS(rc)) { platform_error_log("%s():%d: vector_append() failed: %s", @@ -3010,8 +3014,10 @@ trunk_branch_merger_add_branches(trunk_branch_merger *merger, } for (uint64 i = 0; i < num_branches; i++) { - rc = trunk_branch_merger_add_branch( - merger, branches[i].addr, branches[i].type); + rc = trunk_branch_merger_add_branch(merger, + branches[i].addr, + branches[i].format_version, + branches[i].type); if (!SUCCESS(rc)) { platform_error_log("%s():%d: btree_merger_add_branch() failed: %s", __func__, @@ -3043,8 +3049,10 @@ trunk_branch_merger_add_bundle(trunk_branch_merger *merger, for (uint64 i = 0; i < bundle_num_branches(routed); i++) { branch_ref bref = vector_get(&routed->branches, i); - rc = trunk_branch_merger_add_branch( - merger, branch_ref_addr(bref), bundle_branch_type(routed)); + rc = trunk_branch_merger_add_branch(merger, + branch_ref_addr(bref), + bref.format_version, + bundle_branch_type(routed)); if (!SUCCESS(rc)) { platform_error_log("%s():%d: btree_merger_add_branch() failed: %s", __func__, @@ -3095,8 +3103,12 @@ trunk_branch_merger_build_merge_itor(trunk_branch_merger *merger, for (uint64 i = 0; i < num_branches; i++) { trunk_branch_info branch = vector_get(&merger->branches, i); uint32 root_height; - rc = btree_get_root_height( - cc, merger->cfg->btree_cfg, branch.addr, branch.type, &root_height); + rc = btree_get_root_height(cc, + merger->cfg->btree_cfg, + branch.addr, + branch.format_version, + branch.type, + &root_height); if (!SUCCESS(rc)) { platform_error_log("%s():%d: btree_get_root_height() failed: %s", __func__, @@ -3122,6 +3134,7 @@ trunk_branch_merger_build_merge_itor(trunk_branch_merger *merger, merger->cfg->btree_cfg, itor, branch.addr, + branch.format_version, branch.type, greater_than_or_equal, merger->min_key, @@ -3970,7 +3983,7 @@ bundle_compaction_create(trunk_context *context, btree_inc_ref( context->cc, context->cfg->btree_cfg, branch_ref_addr(bref)); page_type type = bundle_branch_type(bndl); - trunk_branch_info bi = {bref.addr, type}; + trunk_branch_info bi = {bref.addr, type, bref.format_version}; rc = vector_append(&result->input.branches, bi); platform_assert_status_ok(rc); __sync_fetch_and_add(&bc_incs, 1); @@ -4728,8 +4741,13 @@ compute_input_stats(trunk_context *context, for (uint64 i = 0; i < vector_length(branches); i++) { trunk_branch_info bi = vector_get(branches, i); btree_pivot_stats btree_stats; - platform_status rc = btree_count_in_range( - context->cc, context->cfg->btree_cfg, bi.addr, lb, ub, &btree_stats); + platform_status rc = btree_count_in_range(context->cc, + context->cfg->btree_cfg, + bi.addr, + bi.format_version, + lb, + ub, + &btree_stats); if (!SUCCESS(rc)) { return rc; } @@ -5063,6 +5081,7 @@ accumulate_branch_tuple_counts_in_range(branch_ref bref, platform_status rc = btree_count_in_range(context->cc, context->cfg->btree_cfg, branch_ref_addr(bref), + bref.format_version, minkey, maxkey, &stats); @@ -5267,6 +5286,7 @@ leaf_estimate_unique_keys(trunk_context *context, rc = btree_count_in_range(context->cc, context->cfg->btree_cfg, bundle_branch(bndl, 0).addr, + bundle_branch(bndl, 0).format_version, trunk_node_pivot_min_key(leaf), trunk_node_pivot_max_key(leaf), &stats); @@ -5288,12 +5308,14 @@ leaf_estimate_unique_keys(trunk_context *context, platform_assert(bundle_num_branches(&pivot_bundle) <= 1); if (bundle_num_branches(&pivot_bundle) == 1) { btree_pivot_stats stats; - rc = btree_count_in_range(context->cc, - context->cfg->btree_cfg, - bundle_branch(&pivot_bundle, 0).addr, - trunk_node_pivot_min_key(leaf), - trunk_node_pivot_max_key(leaf), - &stats); + rc = + btree_count_in_range(context->cc, + context->cfg->btree_cfg, + bundle_branch(&pivot_bundle, 0).addr, + bundle_branch(&pivot_bundle, 0).format_version, + trunk_node_pivot_min_key(leaf), + trunk_node_pivot_max_key(leaf), + &stats); if (!SUCCESS(rc)) { goto cleanup; } @@ -5382,10 +5404,9 @@ trunk_node_filter_limit(const trunk_context *context, const trunk_node *node) const routing_filter *filter = &vector_get_ptr(&node->pivot_bundles, i)->maplet; if (filter->addr != 0) { - routing_config cfg = *context->cfg->filter_cfg; - cfg.fingerprint_size = filter->fingerprint_size; - cfg.log_index_size = filter->log_index_size; - limit = MIN(limit, routing_filter_max_fingerprints(cache_cfg, &cfg)); + limit = + MIN(limit, + routing_filter_max_fingerprints_for_filter(cache_cfg, filter)); } } return limit; @@ -7048,6 +7069,7 @@ trunk_ondisk_bundle_merge_lookup(trunk_context *context, rc = btree_lookup_and_merge(context->cc, context->cfg->btree_cfg, branch_ref_addr(bndl->branches[idx]), + bndl->branches[idx].format_version, trunk_ondisk_bundle_branch_type(bndl), tgt, result, @@ -7080,6 +7102,7 @@ trunk_ondisk_bundle_merge_lookup(trunk_context *context, rc = btree_lookup_and_merge(context->cc, context->cfg->btree_cfg, branch_ref_addr(bndl->branches[idx]), + bndl->branches[idx].format_version, trunk_ondisk_bundle_branch_type(bndl), tgt, &temp_result, @@ -7185,6 +7208,7 @@ trunk_ondisk_bundle_merge_lookup_async(trunk_merge_lookup_async_state *state, state->context->cc, state->context->cfg->btree_cfg, branch_ref_addr(state->bndl->branches[state->idx]), + state->bndl->branches[state->idx].format_version, trunk_ondisk_bundle_branch_type(state->bndl), state->tgt, state->result, @@ -7221,6 +7245,7 @@ trunk_ondisk_bundle_merge_lookup_async(trunk_merge_lookup_async_state *state, state->context->cc, state->context->cfg->btree_cfg, branch_ref_addr(state->bndl->branches[state->idx]), + state->bndl->branches[state->idx].format_version, trunk_ondisk_bundle_branch_type(state->bndl), state->tgt, &temp_result, @@ -7537,7 +7562,17 @@ trunk_collect_bundle_branches(trunk_ondisk_bundle *bndl, *num_branches = bundle_start; return STATUS_LIMIT_EXCEEDED; } + platform_status rc = + btree_validate_root(bndl->branches[i].addr, + bndl->branches[i].format_version, + trunk_ondisk_bundle_branch_type(bndl)); + if (!SUCCESS(rc)) { + /* References are acquired only after the whole bundle is copied. */ + *num_branches = bundle_start; + return rc; + } branches[*num_branches].addr = branch_ref_addr(bndl->branches[i]); + branches[*num_branches].format_version = bndl->branches[i].format_version; branches[*num_branches].type = trunk_ondisk_bundle_branch_type(bndl); (*num_branches)++; diff --git a/src/trunk.h b/src/trunk.h index 19abf286f..de7768aa9 100644 --- a/src/trunk.h +++ b/src/trunk.h @@ -19,6 +19,23 @@ #include "data_internal.h" #include "batch_rwlock.h" +#define TRUNK_FORMAT_VERSION (1) + +/* The superblock persists this version; the trunk owns its supported readers. + */ +static inline platform_status +trunk_validate_format_version(uint16 format_version) +{ + if (format_version != TRUNK_FORMAT_VERSION) { + platform_error_log("Unsupported trunk format version %u; " + "this build requires version %u.\n", + format_version, + TRUNK_FORMAT_VERSION); + return STATUS_INVALID_STATE; + } + return STATUS_OK; +} + typedef struct trunk_config { const data_config *data_cfg; const btree_config *btree_cfg; @@ -335,6 +352,7 @@ trunk_merge_lookup(trunk_context *context, typedef struct trunk_branch_info { uint64 addr; page_type type; + uint16 format_version; } trunk_branch_info; platform_status diff --git a/tests/functional/btree_test.c b/tests/functional/btree_test.c index 55f3a2c6e..ab7eb5444 100644 --- a/tests/functional/btree_test.c +++ b/tests/functional/btree_test.c @@ -411,7 +411,8 @@ test_btree_lookup(cache *cc, lookup_result_init(&result, cfg->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); - rc = btree_lookup(cc, cfg, root_addr, type, target, &result); + rc = btree_lookup( + cc, cfg, root_addr, BTREE_BRANCH_FORMAT_VERSION, type, target, &result); platform_assert_status_ok(rc); message data = @@ -648,6 +649,7 @@ test_btree_scan_once(cache *cc, btree_cfg, &itor, root_addr, + BTREE_BRANCH_FORMAT_VERSION, type, greater_than_or_equal, min_key, @@ -947,6 +949,7 @@ test_btree_scan_perf(cache *cc, btree_cfg, &itor, root_addr, + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1183,6 +1186,7 @@ test_btree_async_lookup(cache *cc, cc, cfg, root_addr, + BTREE_BRANCH_FORMAT_VERSION, type, target, &async_ctxt->result, @@ -1251,7 +1255,8 @@ test_btree_lookup_status(cache *cc, { lookup_result result; lookup_result_init(&result, cfg->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); - platform_status rc = btree_lookup(cc, cfg, root_addr, type, target, &result); + platform_status rc = btree_lookup( + cc, cfg, root_addr, BTREE_BRANCH_FORMAT_VERSION, type, target, &result); lookup_result_deinit(&result); return rc; } @@ -1270,6 +1275,7 @@ test_btree_async_lookup_status(cache *cc, cc, cfg, root_addr, + BTREE_BRANCH_FORMAT_VERSION, type, target, &ctxt.result, @@ -1334,6 +1340,7 @@ test_btree_iterator_init_status(cache *cc, cfg, &itor, root_addr, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1357,6 +1364,7 @@ test_btree_iterator_init_status(cache *cc, cfg, &itor, root_addr, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1429,6 +1437,7 @@ test_btree_query_validation(cache *cc, cfg, &itor, root_addr, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1571,6 +1580,7 @@ test_btree_query_validation(cache *cc, cfg, &sequence_itor, root_addr, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1732,6 +1742,7 @@ test_btree_query_validation(cache *cc, cfg, &height_itor, root_addr, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, key_buffer_key(&touched_key), @@ -1906,6 +1917,7 @@ test_btree_basic(cache *cc, btree_cfg, &itor, root_addr, + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -2098,6 +2110,7 @@ test_btree_create_packed_trees(cache *cc, btree_cfg, &itor, memtable_root_addr(mt), + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -2156,6 +2169,7 @@ test_count_tuples_in_range(cache *cc, cfg, &itor, root_addr[i], + BTREE_BRANCH_FORMAT_VERSION, type, greater_than_or_equal, low_key, @@ -2255,6 +2269,7 @@ test_btree_print_all_keys(cache *cc, cfg, &itor, root_addr[i], + BTREE_BRANCH_FORMAT_VERSION, type, greater_than_or_equal, low_key, @@ -2334,6 +2349,7 @@ test_btree_merge_basic(cache *cc, btree_cfg, &btree_itor_arr[tree_no], root_addr[tree_no], + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, lo, @@ -2475,8 +2491,13 @@ test_btree_count_in_range(cache *cc, btree_pivot_stats stats; key min_key = key_buffer_key(&bound_key[0]); key max_key = key_buffer_key(&bound_key[1]); - rc = btree_count_in_range( - cc, btree_cfg, root_addr, min_key, max_key, &stats); + rc = btree_count_in_range(cc, + btree_cfg, + root_addr, + BTREE_BRANCH_FORMAT_VERSION, + min_key, + max_key, + &stats); if (!SUCCESS(rc)) { goto destroy_btree; } @@ -2561,6 +2582,7 @@ test_btree_rough_iterator(cache *cc, btree_cfg, &rough_btree_itor[tree_no], root_addr[tree_no], + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -2727,6 +2749,7 @@ test_btree_merge_perf(cache *cc, btree_cfg, &btree_itor_arr[tree_no], root_addr[global_tree_no], + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, min_key, diff --git a/tests/unit/btree_stress_test.c b/tests/unit/btree_stress_test.c index c343f470a..5ca217169 100644 --- a/tests/unit/btree_stress_test.c +++ b/tests/unit/btree_stress_test.c @@ -310,6 +310,7 @@ CTEST2(btree_stress, insert_callback_failure_abandons_insert) rc = btree_lookup((cache *)&data->cc, &data->dbtree_cfg, root_addr, + 0, PAGE_TYPE_MEMTABLE, tuple_key, &lookup); @@ -352,6 +353,7 @@ CTEST2(btree_stress, insert_callback_failure_after_root_growth) ASSERT_TRUE(SUCCESS(btree_get_root_height((cache *)&data->cc, &data->dbtree_cfg, root_addr, + 0, PAGE_TYPE_MEMTABLE, &height))); ASSERT_EQUAL_U(0, height); @@ -384,6 +386,7 @@ CTEST2(btree_stress, insert_callback_failure_after_root_growth) ASSERT_TRUE(SUCCESS(btree_get_root_height((cache *)&data->cc, &data->dbtree_cfg, root_addr, + 0, PAGE_TYPE_MEMTABLE, &height))); ASSERT_TRUE(height > 0); @@ -396,6 +399,7 @@ CTEST2(btree_stress, insert_callback_failure_after_root_growth) rc = btree_lookup((cache *)&data->cc, &data->dbtree_cfg, root_addr, + 0, PAGE_TYPE_MEMTABLE, rejected_key, &lookup); @@ -540,8 +544,8 @@ CTEST2(btree_stress, insert_rebuild_failure_releases_parent_claim) lookup_result lookup; lookup_result_init(&lookup, cfg->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); - ASSERT_TRUE(SUCCESS( - btree_lookup(cc, cfg, root_addr, PAGE_TYPE_MEMTABLE, target, &lookup))); + ASSERT_TRUE(SUCCESS(btree_lookup( + cc, cfg, root_addr, 0, PAGE_TYPE_MEMTABLE, target, &lookup))); ASSERT_EQUAL(0, slice_lex_cmp(merge_accumulator_to_value( lookup_result_accumulator(&lookup)), @@ -792,6 +796,7 @@ CTEST2(btree_stress, overwrite_returns_old_value_after_tree_growth) rc = btree_lookup((cache *)&data->cc, &data->dbtree_cfg, root_addr, + 0, PAGE_TYPE_MEMTABLE, gen_key(&data->dbtree_cfg, i, keybuf, bt_page_size), &verify_result); @@ -929,6 +934,7 @@ query_tests(cache *cc, btree_lookup(cc, cfg, root_addr, + BTREE_BRANCH_FORMAT_VERSION, type, gen_key(cfg, i, keybuf, bt_page_size), &result); @@ -1027,6 +1033,7 @@ iterator_tests(cache *cc, cfg, &dbiter, root_addr, + BTREE_BRANCH_FORMAT_VERSION, type, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1080,6 +1087,7 @@ iterator_seek_tests(cache *cc, cfg, &dbiter, root_addr, + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1128,6 +1136,7 @@ pack_tests(cache *cc, cfg, &dbiter, root_addr, + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1229,6 +1238,7 @@ bounds_init_with_copy(cache *cc, cfg, itor, root, + BTREE_BRANCH_FORMAT_VERSION, type, lo_cmp, lo, @@ -1247,6 +1257,7 @@ bounds_init_with_copy(cache *cc, cfg, itor, root, + BTREE_BRANCH_FORMAT_VERSION, type, lo_cmp, lo, @@ -1368,6 +1379,7 @@ iterator_endpoint_fault_init(cache *cc, cfg, itor, root, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1387,6 +1399,7 @@ iterator_endpoint_fault_init(cache *cc, cfg, itor, root, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1424,6 +1437,7 @@ CTEST2(btree_stress, iterator_endpoint_error_is_returned_by_init) cfg, &source, source_root, + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1450,6 +1464,7 @@ CTEST2(btree_stress, iterator_endpoint_error_is_returned_by_init) cfg, &probe, packed_root, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1530,6 +1545,7 @@ CTEST2(btree_stress, count_in_range_returns_lookup_error) cfg, &source, source_root, + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, @@ -1568,7 +1584,13 @@ CTEST2(btree_stress, count_in_range_returns_lookup_error) btree_pivot_stats stats = { .num_kvs = 1, .key_bytes = 1, .message_bytes = 1}; - rc = btree_count_in_range(cc, cfg, packed_root, min_key, max_key, &stats); + rc = btree_count_in_range(cc, + cfg, + packed_root, + BTREE_BRANCH_FORMAT_VERSION, + min_key, + max_key, + &stats); cc->ops = fault.original; active_iterator_endpoint_fault = NULL; @@ -1834,6 +1856,7 @@ CTEST2(btree_stress, copied_iterator_split_on_release) cfg, &probe, root, + 0, PAGE_TYPE_MEMTABLE, greater_than_or_equal, NEGATIVE_INFINITY_KEY, diff --git a/tests/unit/btree_test.c b/tests/unit/btree_test.c index 1885165a0..5a6e74b73 100644 --- a/tests/unit/btree_test.c +++ b/tests/unit/btree_test.c @@ -195,6 +195,181 @@ CTEST2(btree, test_defragmented_index_bytes) ASSERT_EQUAL(0, rc); } +/* + * Branch references must be rejected by the btree before any page access. + * A NULL cache makes an accidental read fail, including in the async paths. + */ +static const struct { + uint64 addr; + uint16 format_version; +} invalid_branch_refs[] = { + {4096, BTREE_BRANCH_FORMAT_VERSION - 1}, + {4096, BTREE_BRANCH_FORMAT_VERSION + 1}, + {0, BTREE_BRANCH_FORMAT_VERSION}, +}; + +CTEST2(btree, test_invalid_branch_point_lookup) +{ + lookup_result result; + lookup_result_init( + &result, data->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); + uint64 keybuf = 1; + key target = key_create(FALSE, sizeof(keybuf), &keybuf); + + for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { + platform_status rc = btree_lookup(NULL, + &data->dbtree_cfg, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH, + target, + &result); + ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); + ASSERT_FALSE(lookup_result_found(&result)); + + bool32 local_found = TRUE; + rc = btree_lookup_and_merge(NULL, + &data->dbtree_cfg, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH, + target, + &result, + &local_found); + ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); + ASSERT_FALSE(local_found); + ASSERT_FALSE(lookup_result_found(&result)); + } + lookup_result_deinit(&result); +} + +CTEST2(btree, test_invalid_branch_async_point_lookup) +{ + lookup_result result; + lookup_result_init( + &result, data->data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); + uint64 keybuf = 1; + key target = key_create(FALSE, sizeof(keybuf), &keybuf); + + for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { + btree_lookup_async_state state; + btree_lookup_async_state_init(&state, + NULL, + &data->dbtree_cfg, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH, + target, + &result, + NULL, + NULL); + ASSERT_EQUAL(ASYNC_STATUS_DONE, btree_lookup_async(&state)); + ASSERT_TRUE(STATUS_IS_EQ(async_result(&state), STATUS_INVALID_STATE)); + ASSERT_FALSE(lookup_result_found(&result)); + + btree_lookup_and_merge_async_state_init( + &state, + NULL, + &data->dbtree_cfg, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH, + target, + &result, + NULL, + NULL); + ASSERT_EQUAL(ASYNC_STATUS_DONE, btree_lookup_and_merge_async(&state)); + ASSERT_TRUE(STATUS_IS_EQ(async_result(&state), STATUS_INVALID_STATE)); + ASSERT_FALSE(lookup_result_found(&result)); + } + lookup_result_deinit(&result); +} + +CTEST2(btree, test_invalid_branch_iterator) +{ + for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { + for (bool32 copy_nodes = FALSE; copy_nodes <= TRUE; copy_nodes++) { + btree_iterator itor; + memset(&itor, 0xa5, sizeof(itor)); + platform_status rc = + btree_iterator_init(NULL, + &data->dbtree_cfg, + &itor, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, + less_than, + POSITIVE_INFINITY_KEY, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, + copy_nodes, + 0, + 1); + ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); + btree_iterator_deinit(&itor); + + memset(&itor, 0xa5, sizeof(itor)); + btree_iterator_async_state state; + btree_iterator_async_state_init(&state, + NULL, + &data->dbtree_cfg, + &itor, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, + less_than, + POSITIVE_INFINITY_KEY, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, + copy_nodes, + 0, + 1, + NULL, + NULL); + ASSERT_EQUAL(ASYNC_STATUS_DONE, btree_iterator_init_async(&state)); + ASSERT_TRUE(STATUS_IS_EQ(btree_iterator_init_async_result(&state), + STATUS_INVALID_STATE)); + btree_iterator_deinit(&itor); + } + } +} + +CTEST2(btree, test_invalid_branch_metadata_queries) +{ + for (uint64 i = 0; i < ARRAY_SIZE(invalid_branch_refs); i++) { + uint32 height; + platform_status rc = + btree_get_root_height(NULL, + &data->dbtree_cfg, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH, + &height); + ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); + + btree_pivot_stats stats; + rc = btree_count_in_range(NULL, + &data->dbtree_cfg, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + NEGATIVE_INFINITY_KEY, + POSITIVE_INFINITY_KEY, + &stats); + ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); + + rc = btree_recover_allocations(NULL, + &data->dbtree_cfg, + invalid_branch_refs[i].addr, + invalid_branch_refs[i].format_version, + PAGE_TYPE_BRANCH); + ASSERT_TRUE(STATUS_IS_EQ(rc, STATUS_INVALID_STATE)); + } +} + /* * ***************************************************************** * Helper functions, and actual test-case methods. diff --git a/tests/unit/splinter_test.c b/tests/unit/splinter_test.c index 85272c994..d1a84a076 100644 --- a/tests/unit/splinter_test.c +++ b/tests/unit/splinter_test.c @@ -1494,7 +1494,7 @@ CTEST2(splinter, test_malformed_trunk_pivot_bundle) ASSERT_TRUE(range_rejected); } -CTEST2(splinter, test_unknown_branch_version_rejected) +CTEST2(splinter, test_invalid_branch_reference_rejected) { cache *cc = (cache *)data->clock_cache; core_handle spl; @@ -1532,41 +1532,108 @@ CTEST2(splinter, test_unknown_branch_version_rejected) test_trunk_bundle *bundle = (test_trunk_bundle *)(page->data + offset % page_size); ASSERT_TRUE(bundle->num_branches != 0); - uint16 saved_version = bundle->branches[0].format_version; - bundle->branches[0].format_version++; + branch_ref saved_branch = bundle->branches[0]; + routing_filter saved_maplet = bundle->maplet; test_splinter_unclaim_page(cc, page); - lookup_result result; - lookup_result_init( - &result, spl.cfg.data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); - platform_status sync_rc = - core_lookup(&spl, key_buffer_key(&keybuf), &result); - platform_status async_rc = async_call_sync_callback(cache_cleanup(cc), - core_lookup_async, - &spl, - key_buffer_key(&keybuf), - &result); - lookup_result_deinit(&result); - core_range_iterator itor; - platform_status range_rc = core_range_iterator_init(&spl, - &itor, - greater_than_or_equal, - NEGATIVE_INFINITY_KEY, - less_than, - POSITIVE_INFINITY_KEY, - greater_than_or_equal, - NEGATIVE_INFINITY_KEY); - if (SUCCESS(range_rc)) { - core_range_iterator_deinit(&itor); - } - page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); - bundle = (test_trunk_bundle *)(page->data + offset % page_size); - bundle->branches[0].format_version = saved_version; - test_splinter_unclaim_page(cc, page); + /* Select the branch for its key, but skip it for a known filter miss. The + * latter must not inspect a branch that the query does not need to read. */ + const routing_config *filter_cfg = spl.cfg.trunk_node_cfg->filter_cfg; + routing_filter null_filter = NULL_ROUTING_FILTER; + routing_filter filter = NULL_ROUTING_FILTER; + uint32 fp = data_key_hash( + spl.cfg.data_cfg, key_buffer_key(&keybuf), filter_cfg->seed); + ASSERT_TRUE(SUCCESS( + routing_filter_add(cc, filter_cfg, &null_filter, &filter, &fp, 1, 0))); + uint64 found_values; + ASSERT_TRUE(SUCCESS(routing_filter_lookup( + cc, filter_cfg, &filter, key_buffer_key(&keybuf), &found_values))); + ASSERT_EQUAL_U(1, found_values); + + DECLARE_AUTO_KEY_BUFFER(miss_keybuf, data->hid); + for (uint64 i = 2; i < 1002; i++) { + test_key( + &miss_keybuf, TEST_RANDOM, i, 0, 0, data->workload_cfg->key_size, 0); + ASSERT_TRUE(SUCCESS(routing_filter_lookup(cc, + filter_cfg, + &filter, + key_buffer_key(&miss_keybuf), + &found_values))); + if (found_values == 0) { + break; + } + } + ASSERT_EQUAL_U(0, found_values); + + platform_status sync_rc[2]; + platform_status async_rc[2]; + platform_status range_rc[2]; + bool32 sync_miss_ok[2]; + bool32 async_miss_ok[2]; + for (uint64 invalid_field = 0; invalid_field < 2; invalid_field++) { + page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); + bundle = (test_trunk_bundle *)(page->data + offset % page_size); + bundle->maplet = filter; + if (invalid_field == 0) { + bundle->branches[0].format_version++; + } else { + bundle->branches[0].addr = 0; + } + test_splinter_unclaim_page(cc, page); + + lookup_result result; + lookup_result_init( + &result, spl.cfg.data_cfg, SPLINTERDB_LOOKUP_VALUE, 0, NULL); + sync_rc[invalid_field] = + core_lookup(&spl, key_buffer_key(&keybuf), &result); + async_rc[invalid_field] = + async_call_sync_callback(cache_cleanup(cc), + core_lookup_async, + &spl, + key_buffer_key(&keybuf), + &result); + platform_status rc = + core_lookup(&spl, key_buffer_key(&miss_keybuf), &result); + sync_miss_ok[invalid_field] = + SUCCESS(rc) && !lookup_result_found(&result); + rc = async_call_sync_callback(cache_cleanup(cc), + core_lookup_async, + &spl, + key_buffer_key(&miss_keybuf), + &result); + async_miss_ok[invalid_field] = + SUCCESS(rc) && !lookup_result_found(&result); + lookup_result_deinit(&result); + + core_range_iterator itor; + range_rc[invalid_field] = core_range_iterator_init(&spl, + &itor, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY, + less_than, + POSITIVE_INFINITY_KEY, + greater_than_or_equal, + NEGATIVE_INFINITY_KEY); + if (SUCCESS(range_rc[invalid_field])) { + core_range_iterator_deinit(&itor); + } + + /* Restore the owning reference before cleanup can release it. */ + page = test_splinter_claim_page(cc, page_addr, PAGE_TYPE_TRUNK); + bundle = (test_trunk_bundle *)(page->data + offset % page_size); + bundle->branches[0] = saved_branch; + bundle->maplet = saved_maplet; + test_splinter_unclaim_page(cc, page); + } + routing_filter_dec_ref(cc, &filter); core_destroy(&spl); - ASSERT_FALSE(SUCCESS(sync_rc)); - ASSERT_FALSE(SUCCESS(async_rc)); - ASSERT_FALSE(SUCCESS(range_rc)); + for (uint64 invalid_field = 0; invalid_field < 2; invalid_field++) { + ASSERT_TRUE(STATUS_IS_EQ(sync_rc[invalid_field], STATUS_INVALID_STATE)); + ASSERT_TRUE(STATUS_IS_EQ(async_rc[invalid_field], STATUS_INVALID_STATE)); + ASSERT_TRUE(STATUS_IS_EQ(range_rc[invalid_field], STATUS_INVALID_STATE)); + ASSERT_TRUE(sync_miss_ok[invalid_field]); + ASSERT_TRUE(async_miss_ok[invalid_field]); + } } /* Persisted pivot statistics are sizing hints, but they still feed arithmetic diff --git a/tests/unit/superblock_test.c b/tests/unit/superblock_test.c index 6649f87be..8f55e86da 100644 --- a/tests/unit/superblock_test.c +++ b/tests/unit/superblock_test.c @@ -715,7 +715,7 @@ CTEST2(superblock, test_bootstrap_geometry_metadata_budget) /* A checksummed newer version is not permission to roll back to its peer. */ CTEST2(superblock, test_unknown_structure_version_rejected) { - for (uint64 component = 0; component < 3; component++) { + for (uint64 component = 0; component < 4; component++) { superblock_context ctx; platform_status rc = superblock_context_init( &ctx, data->ioh, &data->allocator_cfg, data->hid); @@ -725,9 +725,13 @@ CTEST2(superblock, test_unknown_structure_version_rejected) ctx.image->trunk_format_version++; } else if (component == 1) { ctx.image->allocator_format_version++; - } else { + } else if (component == 2) { superblock_log_cut(&ctx, TEST_LOG_L1); ctx.image->tree.live_log.head.format_version++; + } else { + superblock_log_cut(&ctx, TEST_LOG_L1); + superblock_log_cut(&ctx, TEST_LOG_L2); + ctx.image->tree.sealed_log.head.format_version++; } ASSERT_TRUE(SUCCESS(superblock_make_durable(&ctx))); superblock image; diff --git a/tests/unit/trunk_split_test.c b/tests/unit/trunk_split_test.c index ecaa3385c..bf0baebcc 100644 --- a/tests/unit/trunk_split_test.c +++ b/tests/unit/trunk_split_test.c @@ -237,6 +237,7 @@ CTEST2(trunk_split, split_scan_propagates_io_error) data->cfg.btree_cfg, &itor, branch, + BTREE_BRANCH_FORMAT_VERSION, PAGE_TYPE_BRANCH, greater_than_or_equal, NEGATIVE_INFINITY_KEY, From 2ba43edd4b6e7d045f1c2f272d85ed8e26f1149a Mon Sep 17 00:00:00 2001 From: Rob Johnson Date: Tue, 13 Oct 2026 09:29:12 -0700 Subject: [PATCH 6/7] Record main integration tests and real-disk benchmark results --- .../main-7bb5fab/.gitattributes | 3 + .../results-20261013/main-7bb5fab/README.md | 11 + .../main-7bb5fab/benchmarks/audit.json | 90 + .../main-7bb5fab/benchmarks/audit.py | 62 + .../benchmarks/benchmark-jobs.json | 167 ++ .../main-7bb5fab/benchmarks/build_all.py | 44 + .../main-7bb5fab/benchmarks/control-drift.csv | 69 + .../benchmarks/disk-format-build.json | 193 ++ .../main-7bb5fab/benchmarks/disk/00-main.log | 20 + .../benchmarks/disk/01-disk-format.log | 20 + .../main-7bb5fab/benchmarks/disk/11-main.log | 20 + .../benchmarks/disk/21-disk-format.log | 20 + .../main-7bb5fab/benchmarks/disk/22-main.log | 20 + .../benchmarks/disk/manifest.json | 417 ++++ .../main-7bb5fab/benchmarks/disk/results.csv | 21 + .../benchmarks/feature-summaries.json | 354 ++++ .../fixtures/disk-disk-format-ordinary.csv | 2 + .../fixtures/disk-disk-format-ordinary.log | 7 + .../fixtures/disk-main-ordinary.csv | 2 + .../fixtures/disk-main-ordinary.log | 7 + .../benchmarks/fixtures/manifest.json | 271 +++ .../resident-disk-format-ordinary.csv | 2 + .../resident-disk-format-ordinary.log | 7 + .../fixtures/resident-main-ordinary.csv | 2 + .../fixtures/resident-main-ordinary.log | 7 + .../main-7bb5fab/benchmarks/host.json | 7 + .../benchmarks/insert/manifest.json | 426 ++++ .../benchmarks/insert/results.csv | 8 + .../benchmarks/insert/wal0-00-main.csv | 51 + .../benchmarks/insert/wal0-00-main.log | 14 + .../insert/wal0-00-main.summary.csv | 2 + .../benchmarks/insert/wal0-05-main.csv | 51 + .../benchmarks/insert/wal0-05-main.log | 14 + .../insert/wal0-05-main.summary.csv | 2 + .../benchmarks/insert/wal1-00-main.csv | 51 + .../benchmarks/insert/wal1-00-main.log | 14 + .../insert/wal1-00-main.summary.csv | 2 + .../benchmarks/insert/wal1-01-disk-format.csv | 51 + .../benchmarks/insert/wal1-01-disk-format.log | 14 + .../insert/wal1-01-disk-format.summary.csv | 2 + .../benchmarks/insert/wal1-12-main.csv | 51 + .../benchmarks/insert/wal1-12-main.log | 14 + .../insert/wal1-12-main.summary.csv | 2 + .../benchmarks/insert/wal1-23-disk-format.csv | 51 + .../benchmarks/insert/wal1-23-disk-format.log | 14 + .../insert/wal1-23-disk-format.summary.csv | 2 + .../benchmarks/insert/wal1-24-main.csv | 51 + .../benchmarks/insert/wal1-24-main.log | 14 + .../insert/wal1-24-main.summary.csv | 2 + .../benchmarks/insert_benchmark.c | 1002 ++++++++++ .../main-7bb5fab/benchmarks/inventory.json | 69 + .../main-7bb5fab/benchmarks/kvbench.c | 1707 +++++++++++++++++ .../main-7bb5fab/benchmarks/main-build.json | 193 ++ .../main-7bb5fab/benchmarks/package.py | 67 + .../main-7bb5fab/benchmarks/preparation.json | 20 + .../main-7bb5fab/benchmarks/repeat.py | 19 + .../main-7bb5fab/benchmarks/report.md | 237 +++ .../main-7bb5fab/benchmarks/report.py | 127 ++ .../benchmarks/resident/00-main.log | 30 + .../benchmarks/resident/01-disk-format.log | 30 + .../benchmarks/resident/11-main.log | 30 + .../benchmarks/resident/21-disk-format.log | 30 + .../benchmarks/resident/22-main.log | 30 + .../benchmarks/resident/manifest.json | 429 +++++ .../benchmarks/resident/results.csv | 31 + .../main-7bb5fab/benchmarks/run_job.py | 54 + .../benchmarks/run_matrix-fixtures.py | 152 ++ .../main-7bb5fab/benchmarks/run_matrix.py | 156 ++ .../main-7bb5fab/benchmarks/status.json | 33 + .../main-7bb5fab/benchmarks/summary.csv | 10 + .../main-7bb5fab/tests/README.md | 11 + .../main-7bb5fab/tests/archive-manifest.json | 214 +++ .../main-7bb5fab/tests/build-debug-3.log | 38 + .../main-7bb5fab/tests/build-debug-4.log | 14 + .../main-7bb5fab/tests/build-debug-5.log | 8 + .../main-7bb5fab/tests/build-debug-final.log | 84 + .../tests/build-debug-formatted.log | 26 + .../tests/build-debug-resolved.log | 18 + .../main-7bb5fab/tests/build-debug.log | 33 + .../main-7bb5fab/tests/build-release.log | 120 ++ .../main-7bb5fab/tests/debug-btree.log | 35 + .../main-7bb5fab/tests/debug-filter.log | 65 + .../main-7bb5fab/tests/debug-log.log | 11 + .../main-7bb5fab/tests/debug-quick.log | 55 + .../tests/debug-splinter-4gib.log | 89 + .../main-7bb5fab/tests/debug-splinter.log | 88 + .../main-7bb5fab/tests/debug-superblock.log | 43 + .../main-7bb5fab/tests/debug-trunk-async.log | 5 + .../main-7bb5fab/tests/manifest.json | 273 +++ .../main-7bb5fab/tests/release-btree.log | 35 + .../main-7bb5fab/tests/release-filter.log | 65 + .../main-7bb5fab/tests/release-log.log | 11 + .../main-7bb5fab/tests/release-quick.log | 55 + .../tests/release-splinter-4gib.log | 89 + .../main-7bb5fab/tests/release-splinter.log | 88 + .../main-7bb5fab/tests/release-superblock.log | 43 + .../tests/release-trunk-async.log | 5 + .../main-7bb5fab/tests/shared/before.json | 97 + .../main-7bb5fab/tests/shared/run_job.py | 54 + .../tests/shared/test_root_branches.py | 67 + .../main-7bb5fab/tests/test_integration.py | 19 + .../tests/validation-debug-4gib.json | 47 + .../main-7bb5fab/tests/validation-debug.json | 111 ++ .../tests/validation-release-4gib.json | 45 + .../tests/validation-release.json | 111 ++ docs/on-disk-format.md | 10 + 106 files changed, 9386 insertions(+) create mode 100644 benchmarks/results-20261013/main-7bb5fab/.gitattributes create mode 100644 benchmarks/results-20261013/main-7bb5fab/README.md create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/benchmark-jobs.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/build_all.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/control-drift.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk-format-build.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/00-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/01-disk-format.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/11-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/21-disk-format.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/22-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/manifest.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/results.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/feature-summaries.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/manifest.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/host.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/manifest.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/results.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/insert_benchmark.c create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/inventory.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/kvbench.c create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/main-build.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/package.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/preparation.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/repeat.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/report.md create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/report.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/00-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/01-disk-format.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/11-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/21-disk-format.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/22-main.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/manifest.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/results.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/run_job.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix-fixtures.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/status.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/benchmarks/summary.csv create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/README.md create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/archive-manifest.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-debug-3.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-debug-4.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-debug-5.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-debug-final.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-debug-formatted.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-debug-resolved.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-debug.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/build-release.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-btree.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-filter.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-log.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-quick.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter-4gib.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-superblock.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/debug-trunk-async.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/manifest.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-btree.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-filter.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-log.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-quick.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-splinter-4gib.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-splinter.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-superblock.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/release-trunk-async.log create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/shared/before.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/shared/run_job.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/shared/test_root_branches.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/test_integration.py create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/validation-debug-4gib.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/validation-debug.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/validation-release-4gib.json create mode 100644 benchmarks/results-20261013/main-7bb5fab/tests/validation-release.json diff --git a/benchmarks/results-20261013/main-7bb5fab/.gitattributes b/benchmarks/results-20261013/main-7bb5fab/.gitattributes new file mode 100644 index 000000000..b56614d96 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/.gitattributes @@ -0,0 +1,3 @@ +*.csv whitespace=cr-at-eol +*.patch -whitespace +*.log -whitespace diff --git a/benchmarks/results-20261013/main-7bb5fab/README.md b/benchmarks/results-20261013/main-7bb5fab/README.md new file mode 100644 index 000000000..f6967df2b --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/README.md @@ -0,0 +1,11 @@ +# disk-format: main integration evidence + +Source merge: `d21b87f5cfac709fb0fa74c3585fb38936c08852`; main: `7bb5fab305b9c6179e31f75998dabea833537a08`. + +[Correctness checks](tests/README.md) include exact commands, hashes and retained initial failures. [Benchmark report](benchmarks/report.md), [project summary](benchmarks/summary.csv), and [verification status](benchmarks/status.json) describe the fresh real-disk screening. + +Resident point throughput changed -1.21%/-1.92% at one/eight threads, and disk point throughput changed -0.81%. These sample ranges overlap main; substantial control drift prevents precise attribution of small differences. + +The campaign ran 133 comparison processes, 12 fixture creators and one recorded warmup through 11 exclusive jobs. These compare each updated feature with main; they do not isolate merge cost. Main controls drifted; sample-range overlap is descriptive, not a statistical confidence interval. Query/hash/barrier runs validate data, while insert-only runs check API returns, positive counts and successful close without readback. All benchmark databases were on ext4 `/dev/mapper/snafu-root_crypt` with O_DIRECT and fsynced fixture copies. + +`tests/archive-manifest.json` and `benchmarks/inventory.json` verify the small archived files. No database images, executables or core dumps are committed. Archived scripts retain original paths and must be adapted before replay outside the original campaign. diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.json new file mode 100644 index 000000000..188de621f --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.json @@ -0,0 +1,90 @@ +{ + "complete": true, + "scope": "new main versus updated feature; merge cost not isolated", + "counts": { + "fixtures": { + "runs": 12, + "runner": "run_matrix-fixtures.py", + "verified": true + }, + "resident": { + "runs": 23, + "runner": "run_matrix.py", + "verified": true + }, + "disk": { + "runs": 23, + "runner": "run_matrix.py", + "verified": true + }, + "prefix-resident-common": { + "runs": 4, + "runner": "run_matrix.py", + "verified": true + }, + "prefix-resident-random": { + "runs": 4, + "runner": "run_matrix.py", + "verified": true + }, + "prefix-disk-common": { + "runs": 4, + "runner": "run_matrix.py", + "verified": true + }, + "prefix-disk-random": { + "runs": 4, + "runner": "run_matrix.py", + "verified": true + }, + "insert": { + "runs": 31, + "runner": "run_matrix.py", + "verified": true + }, + "hash": { + "runs": 4, + "runner": "run_matrix.py", + "verified": true + }, + "barrier": { + "runs": 24, + "runner": "run_matrix.py", + "verified": true + }, + "repeat-warmup": { + "runs": 1, + "runner": "repeat.py", + "verified": true + }, + "repeat-cache-invalid-addresses": { + "runs": 4, + "runner": "repeat.py", + "verified": true + }, + "repeat-directory-bucket-atomic": { + "runs": 4, + "runner": "repeat.py", + "verified": true + }, + "repeat-async-durable-barrier": { + "runs": 4, + "runner": "repeat.py", + "verified": true + } + }, + "total_processes": 146, + "comparison_processes": 133, + "fixture_processes": 12, + "recorded_warmup_processes": 1, + "exclusive_benchmark_jobs": 11, + "source_and_binary_hashes_verified": true, + "runner_hashes_verified": true, + "all_processes_serial": true, + "no_overlapping_build_test_jobs": true, + "resident_physical_read_bytes": 0, + "errors": 0, + "violations": 0, + "insert_verification": "API return, positive insert count and successful close; no readback", + "storage": "ext4 /dev/mapper/snafu-root_crypt; O_DIRECT; fsynced fixture copies" +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.py new file mode 100644 index 000000000..ad764439d --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/audit.py @@ -0,0 +1,62 @@ +"""Validate final matrix coverage, provenance, row counts and exclusive scheduling.""" +import csv,hashlib,json +from pathlib import Path +C=Path(__file__).resolve().parent +P=C.parents[2] +O=P/'benchmarks/main-7bb5fab' +def sha(p):return hashlib.sha256(p.read_bytes()).hexdigest() +def readcsv(p):return list(csv.DictReader(p.open())) +expected={'fixtures':12,'resident':23,'disk':23,'prefix-resident-common':4,'prefix-resident-random':4,'prefix-disk-common':4,'prefix-disk-random':4,'insert':31,'hash':4,'barrier':24,'repeat-warmup':1,'repeat-cache-invalid-addresses':4,'repeat-directory-bucket-atomic':4,'repeat-async-durable-barrier':4} +known_runner_hashes={sha(C/n):n for n in ['run_matrix.py','run_matrix-fixtures.py','repeat.py']} +builds={p.name.removesuffix('-build.json'):json.loads(p.read_text()) for p in C.glob('*-build.json')} +for profile,b in builds.items(): + assert sha(Path(b['library']))==b['library_sha256'],profile + for f,h in b['source_files_sha256'].items():assert sha(Path(b['tree'])/f)==h,(profile,f) + for exe,v in b['binaries'].items(): + assert sha(Path(v['binary']))==v['binary_sha256'] + assert sha(Path(v['source']))==v['source_sha256'] +runs=[];counts={} +for mode,count in expected.items(): + folder=O/mode;m=json.loads((folder/'manifest.json').read_text()) + assert m['complete'] and len(m['runs'])==count,mode + assert m['runner_sha256'] in known_runner_hashes,mode + if 'shared_runner_sha256' in m:assert m['shared_runner_sha256']==sha(C/'run_matrix.py') + for r in m['runs']: + assert r['verified'] and r['returncode']==0,(mode,r['label']) + log=folder/(r['label']+'.log');assert sha(log)==r['log_sha256'] + assert 'FINDING: splinterdb_close' not in log.read_text(errors='replace') + b=builds[r['profile']];assert r['source_head']==b['head'] and r['library_sha256']==b['library_sha256'] + assert r['binary_sha256'] in {v['binary_sha256'] for v in b['binaries'].values()} + assert r['finished']>=r['started'] and r['process_cpu_s']>0 + runs.append(dict(mode=mode,label=r['label'],started=r['started'],finished=r['finished'])) + counts[mode]=dict(runs=count,runner=known_runner_hashes[m['runner_sha256']],verified=True) + if (folder/'results.csv').exists(): + rows=readcsv(folder/'results.csv') + if 'phase' in rows[0]: + for r in rows: + assert int(r['errors'])==0 and int(r['violations'])==0 and int(r['ops'])>0 + if r['phase']=='warm':assert int(r['ops'])==int(r['n']) + elif mode=='resident' or mode.startswith('repeat-'):assert int(r['rd_bytes'])==0 + if mode=='barrier':assert len(rows)==count and all(int(r['verified'])==1 for r in rows) + if mode=='insert':assert len(rows)==count and all(int(r['errors'])==0 and int(r['violations'])==0 and int(r['inserted'])>0 for r in rows) + if mode=='fixtures': + for r in m['runs']: + rows=readcsv(folder/(r['label']+'.csv'));assert len(rows)==1 and int(rows[0]['ops'])==int(rows[0]['n']) and int(rows[0]['errors'])==int(rows[0]['violations'])==0 + if mode=='hash': + for r in m['runs']:assert [int(x['limit']) for x in r['rows']]==[1,2,10,100] and all(int(x['queries'])==1000 for x in r['rows']) + for i in range(4):assert len({r['rows'][i]['checksum'] for r in m['runs']})==1 +runs.sort(key=lambda r:r['started']) +assert all(a['finished']<=b['started'] for a,b in zip(runs,runs[1:])), 'Benchmark subprocesses overlap' +jobs=[json.loads(line) for line in (P/'coordination/jobs.jsonl').read_text().splitlines()] +ours=[j for j in jobs if j['mode']=='benchmark' and j['name'].startswith('main-7bb5fab-benchmark-')] +assert len(ours)==11 and all(j['mode']=='benchmark' and j['returncode']==0 for j in ours) +for j in ours: + for k in jobs: + if j is k:continue + assert min(j['finished'],k['finished'])<=max(j['started'],k['started']),(j['name'],k['name']) +for r in runs:assert sum(j['started']<=r['started'] and j['finished']>=r['finished'] for j in ours)==1 +(C/'run_job.py').write_bytes((P/'coordination/run_job.py').read_bytes()) +(C/'benchmark-jobs.json').write_text(json.dumps(ours,indent=2)+'\n') +result=dict(complete=True,scope='new main versus updated feature; merge cost not isolated',counts=counts,total_processes=len(runs),comparison_processes=len(runs)-13,fixture_processes=12,recorded_warmup_processes=1,exclusive_benchmark_jobs=len(ours),source_and_binary_hashes_verified=True,runner_hashes_verified=True,all_processes_serial=True,no_overlapping_build_test_jobs=True,resident_physical_read_bytes=0,errors=0,violations=0,insert_verification='API return, positive insert count and successful close; no readback',storage='ext4 /dev/mapper/snafu-root_crypt; O_DIRECT; fsynced fixture copies') +(C/'audit.json').write_text(json.dumps(result,indent=2)+'\n') +print(json.dumps(result,indent=2)) diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/benchmark-jobs.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/benchmark-jobs.json new file mode 100644 index 000000000..079868d9d --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/benchmark-jobs.json @@ -0,0 +1,167 @@ +[ + { + "name": "main-7bb5fab-benchmark-fixtures", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/run_matrix.py", + "fixtures" + ], + "started": 1791905953.791713, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791905996.8644083 + }, + { + "name": "main-7bb5fab-benchmark-resident", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/run_matrix.py", + "resident" + ], + "started": 1791906087.9268048, + "pid": 5, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791906898.088736 + }, + { + "name": "main-7bb5fab-benchmark-disk", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/run_matrix.py", + "disk" + ], + "started": 1791906929.7655535, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791907361.8222167 + }, + { + "name": "main-7bb5fab-benchmark-prefix", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/run_matrix.py", + "prefix" + ], + "started": 1791907361.8244245, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791907746.3817344 + }, + { + "name": "main-7bb5fab-benchmark-insert", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/run_matrix.py", + "insert" + ], + "started": 1791907746.3838046, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791907918.6362858 + }, + { + "name": "main-7bb5fab-benchmark-hash", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/run_matrix.py", + "hash" + ], + "started": 1791907918.6382897, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791907922.146424 + }, + { + "name": "main-7bb5fab-benchmark-barrier", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/run_matrix.py", + "barrier" + ], + "started": 1791908012.5013723, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791908144.8841226 + }, + { + "name": "main-7bb5fab-benchmark-repeat-warmup", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/repeat.py", + "warmup" + ], + "started": 1791908144.8859186, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791908167.146525 + }, + { + "name": "main-7bb5fab-benchmark-repeat-invalid", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/repeat.py", + "cache-invalid-addresses" + ], + "started": 1791908236.5269575, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791908369.5229552 + }, + { + "name": "main-7bb5fab-benchmark-repeat-atomic", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/repeat.py", + "directory-bucket-atomic" + ], + "started": 1791908442.7619998, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791908531.8354392 + }, + { + "name": "main-7bb5fab-benchmark-repeat-async", + "mode": "benchmark", + "cwd": "/home/rob/projects/splinterdb", + "command": [ + "python3", + ".cache/projects-20261011/coordination/main-7bb5fab/benchmarks/repeat.py", + "async-durable-barrier" + ], + "started": 1791908542.3013017, + "pid": 3, + "db_dir": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab", + "returncode": 0, + "finished": 1791908631.2491326 + } +] diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/build_all.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/build_all.py new file mode 100644 index 000000000..f3e172154 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/build_all.py @@ -0,0 +1,44 @@ +import hashlib,json,subprocess,sys +from pathlib import Path +ROOT=Path('/home/rob/projects/splinterdb') +CAMP=ROOT/'.cache/projects-20261011' +OUT=Path(__file__).resolve().parent +SOURCES=OUT/'sources' +def sha(p):return hashlib.sha256(Path(p).read_bytes()).hexdigest() +profiles={'main':(ROOT,'build-main-7bb5fab')} +for name in ['disk-format','filtered-ranges','log-group-limit','branch-prefix','lazy-merge','async-durable-barrier','cache-invalid-addresses']: + profiles[name]=(CAMP/'worktrees'/name,'build-main-7bb5fab') +for opt in ['direct','bucket','bucket-simd','bucket-atomic']: + profiles['directory-'+opt]=(CAMP/'worktrees/cache-directory',f'build-main-7bb5fab/{opt}') +flags=['cc','-D_GNU_SOURCE','-ggdb','-Wall','-pthread','-Werror','-DXXH_STATIC_LINKING_ONLY','-fPIC','-DSPLINTERDB_PLATFORM_DIR=platform_linux','-O3','-ffast-math','-flto','-march=native'] +manifest={} +for name,(tree,buildroot) in profiles.items(): + if len(sys.argv)>1 and name not in sys.argv[1:]:continue + head=subprocess.check_output(['git','rev-parse','HEAD'],cwd=tree,text=True).strip() + subprocess.run(['git','merge-base','--is-ancestor','7bb5fab',head],cwd=tree,check=True) + lib=tree/buildroot/'release/lib/libsplinterdb.a' + header=(tree/'include/splinterdb/splinterdb.h').read_text() + record=dict(tree=str(tree),head=head,library=str(lib),library_sha256=sha(lib),binaries={},source_files_sha256={}) + for filename in subprocess.check_output(['git','ls-files','src','include'],cwd=tree,text=True).splitlines(): + if (tree/filename).is_file():record['source_files_sha256'][filename]=sha(tree/filename) + bins=['kvbench','insert_benchmark'] + if name in ['main','branch-prefix']:bins+=['prefix_kvbench'] + if name in ['main','filtered-ranges']:bins+=['hash_range_benchmark'] + if name in ['main','async-durable-barrier']:bins+=['barrier_bench'] + for binaryname in bins: + source=SOURCES/(binaryname+'.c');binary=OUT/'bin'/name/binaryname;binary.parent.mkdir(parents=True,exist_ok=True) + macros=[] + if binaryname in ['kvbench','prefix_kvbench']:macros+=['-DKVBENCH_CLOSE_HAS_FORCE'] + if binaryname=='prefix_kvbench': + macros+=['-DKVBENCH_SPACE_MODE'] + if name=='branch-prefix':macros+=['-DKVBENCH_PREFIX_SUPPORT'] + if binaryname=='hash_range_benchmark' and name=='filtered-ranges':macros+=['-DHASH_RANGE_FILTER_API'] + if binaryname=='insert_benchmark' and name!='log-group-limit':macros+=['-DKVBENCH_NO_LOG_GROUP_LIMIT'] + if '#define SPLINTERDB_NOTIFICATION_BLOCKING' in header:macros+=['-DHAVE_SENTINEL_NOTIFICATION','-DHAVE_ASYNC_BARRIER'] + command=flags+macros+['-I',str(tree/'include'),'-I',str(tree/'src'),'-I',str(tree/'src/platform_linux'),str(source),str(lib),'-laio','-lxxhash','-lm','-lpthread','-o',str(binary)] + print('build',name,binaryname,flush=True) + subprocess.run(command,check=True) + record['binaries'][binaryname]=dict(command=command,binary=str(binary),binary_sha256=sha(binary),source=str(source),source_sha256=sha(source)) + (OUT/(name+'-build.json')).write_text(json.dumps(record,indent=2)+'\n') + manifest[name]=record +(OUT/'builds-last.json').write_text(json.dumps(manifest,indent=2)+'\n') diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/control-drift.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/control-drift.csv new file mode 100644 index 000000000..cfb613d57 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/control-drift.csv @@ -0,0 +1,69 @@ +workload,phase,order,ops_s,change_from_first_pct,cpu_us_op,read_bytes +repeat-warmup,point:uniform@8,0,5725282.1,0.0,1.397020051521282,0 +repeat-warmup,range:uniform:1@8,0,531558.3,0.0,15.047298523902287,0 +repeat-directory-bucket-atomic,point:uniform@8,0,6058373.4,0.0,1.320094818123589,0 +repeat-directory-bucket-atomic,point:uniform@8,3,5103421.5,-15.762513086433405,1.5670903540638217,0 +repeat-directory-bucket-atomic,range:uniform:1@8,0,566747.0,0.0,14.112342596291334,0 +repeat-directory-bucket-atomic,range:uniform:1@8,3,498451.0,-12.050526954708186,16.044938184918625,0 +disk,point:uniform@8,0,153061.5,0.0,5.361065052728681,2471006208 +disk,point:uniform@8,11,162899.9,6.427743096729088,6.005865909099279,2630258688 +disk,point:uniform@8,22,168110.0,9.831668969662521,6.042270913585652,2714718208 +disk,range:uniform:1@8,0,63458.1,0.0,132.9297233243934,6286577664 +disk,range:uniform:1@8,11,63625.4,0.2636385268389718,133.28995634011335,6302711808 +disk,range:uniform:1@8,22,64975.4,2.391026519861139,130.54799748839613,6439641088 +disk,range:uniform:10000@8,0,841.4,0.0,3017.1143332541005,9990152192 +disk,range:uniform:10000@8,11,835.2,-0.7368671262181992,3448.754789272031,9911496704 +disk,range:uniform:10000@8,22,840.0,-0.16638935108153063,3655.0,9932713984 +repeat-async-durable-barrier,point:uniform@8,0,5568429.8,0.0,1.4363332788847318,0 +repeat-async-durable-barrier,point:uniform@8,3,4940084.1,-11.284073294773334,1.6189229334750765,0 +repeat-async-durable-barrier,range:uniform:1@8,0,506930.4,0.0,15.778718800936517,0 +repeat-async-durable-barrier,range:uniform:1@8,3,489239.4,-3.4898281894319205,16.345419991652534,0 +prefix-resident-common,point:uniform@8,0,3798858.3,0.0,2.105459146011521,0 +prefix-resident-common,point:uniform@8,3,3979950.2,4.767008550963858,2.009371623828547,0 +prefix-resident-common,range:uniform:1@8,0,992358.5,0.0,8.06011032205677,0 +prefix-resident-common,range:uniform:1@8,3,996704.2,0.43791633769447724,8.02499213268852,0 +prefix-resident-common,range:uniform:10000@8,0,3472.3,0.0,2305.0155987521,0 +prefix-resident-common,range:uniform:10000@8,3,3609.2,3.9426316850502463,2216.3834503140006,0 +prefix-resident-common,ow:uniform@8,0,2274316.9,0.0,3.0310520336583586,2005544960 +prefix-resident-common,ow:uniform@8,3,2304559.8,1.3297575197194433,3.037911263890554,2003456000 +resident,point:uniform@1,0,1261757.3,0.0,0.79252082249396,0 +resident,point:uniform@1,11,1152531.0,-8.656680646904125,0.8676282687172,0 +resident,point:uniform@1,22,1148133.5,-9.005202506060407,0.8709404196916726,0 +resident,point:uniform@8,0,6445634.1,0.0,1.2407480739199983,0 +resident,point:uniform@8,11,5270841.2,-18.22618041567081,1.5171993153252339,0 +resident,point:uniform@8,22,5463995.8,-15.229507055015734,1.4638320892600838,0 +resident,range:uniform:1@1,0,114690.1,0.0,8.718775883865906,0 +resident,range:uniform:1@1,11,105447.5,-8.058760084785,9.481407022468359,0 +resident,range:uniform:1@1,22,105112.6,-8.350764364143027,9.519536136660204,0 +resident,range:uniform:1@8,0,600376.6,0.0,13.32092671033598,0 +resident,range:uniform:1@8,11,532530.7,-11.300557017045643,15.020171795914191,0 +resident,range:uniform:1@8,22,549552.3,-8.465403215248546,14.554061799643964,0 +resident,range:uniform:10000@8,0,4305.3,0.0,1857.8562304029729,0 +resident,range:uniform:10000@8,11,3785.8,-12.06652265811906,2112.651875330164,0 +resident,range:uniform:10000@8,22,3929.7,-8.724130722597733,2035.4552780016115,0 +repeat-cache-invalid-addresses,point:uniform@1,0,1237297.9,0.0,0.8081970578394306,0 +repeat-cache-invalid-addresses,point:uniform@1,3,1167884.0,-5.610120246708572,0.8562318264794829,0 +repeat-cache-invalid-addresses,point:uniform@8,0,6207307.6,0.0,1.2885781245675567,0 +repeat-cache-invalid-addresses,point:uniform@8,3,5328062.8,-14.164672619091723,1.5011880792761536,0 +repeat-cache-invalid-addresses,range:uniform:1@8,0,546650.2,0.0,14.631325678521693,0 +repeat-cache-invalid-addresses,range:uniform:1@8,3,516913.3,-5.439840687884134,15.472656757810807,0 +prefix-disk-random,point:uniform@8,0,135623.0,0.0,6.979415663796549,2392637440 +prefix-disk-random,point:uniform@8,3,173077.1,27.616333512752256,6.5078303110978615,3054764032 +prefix-disk-random,range:uniform:1@8,0,56431.4,0.0,148.08803203742426,3998134272 +prefix-disk-random,range:uniform:1@8,3,70766.5,25.402701332945842,118.96633582781307,5010862080 +prefix-disk-random,range:uniform:10000@8,0,813.8,0.0,3910.2973703612683,10312994816 +prefix-disk-random,range:uniform:10000@8,3,912.4,12.11599901695748,3859.710653222271,11551866880 +prefix-disk-common,point:uniform@8,0,166767.1,0.0,6.629322760954632,2942513152 +prefix-disk-common,point:uniform@8,3,137171.0,-17.74696567848215,6.595905989385899,2419056640 +prefix-disk-common,range:uniform:1@8,0,67364.5,0.0,124.38620345685463,4767051776 +prefix-disk-common,range:uniform:1@8,3,56813.9,-15.661958449925406,147.16173499200934,4020363264 +prefix-disk-common,range:uniform:10000@8,0,899.2,0.0,3836.7437722419927,11408752640 +prefix-disk-common,range:uniform:10000@8,3,812.8,-9.608540925266917,3834.399606299213,10284527616 +prefix-resident-random,point:uniform@8,0,4089596.1,0.0,1.955655773286237,0 +prefix-resident-random,point:uniform@8,3,3861855.4,-5.568782208101187,2.0713343968861757,0 +prefix-resident-random,range:uniform:1@8,0,987903.4,0.0,8.096649643019639,0 +prefix-resident-random,range:uniform:1@8,3,1027722.4,4.0306572484718695,7.782304346965907,0 +prefix-resident-random,range:uniform:10000@8,0,4617.6,0.0,1732.1976395856643,0 +prefix-resident-random,range:uniform:10000@8,3,4063.2,-12.006237006237019,1968.7448728465956,0 +prefix-resident-random,ow:uniform@8,0,2181154.0,0.0,3.049805026119027,1722953728 +prefix-resident-random,ow:uniform@8,3,2346979.3,7.602640620515544,3.0134031679362994,2000707584 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk-format-build.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk-format-build.json new file mode 100644 index 000000000..000d7b2f1 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk-format-build.json @@ -0,0 +1,193 @@ +{ + "tree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format", + "head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "library": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-main-7bb5fab/release/lib/libsplinterdb.a", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "binaries": { + "kvbench": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/include", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/src", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/kvbench.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-main-7bb5fab/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench", + "binary_sha256": "733c71174e9356864c326570c69279aacafc92f2dd7286813f9a21cdf9335ddc", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/kvbench.c", + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9" + }, + "insert_benchmark": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_NO_LOG_GROUP_LIMIT", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/include", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/src", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/insert_benchmark.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-main-7bb5fab/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/insert_benchmark" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/insert_benchmark", + "binary_sha256": "1212b5a19188db5d22643b7e875b8cfecd3c664d6616519892e1ce4fed0a1a83", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/insert_benchmark.c", + "source_sha256": "75acbda52b2a8949f1d61710a8d89250d4dac63b12ec9803b89170f9c2cc0b26" + } + }, + "source_files_sha256": { + "include/splinterdb/data.h": "57be61933a057f52f679dfcf21755a8645e6228cd1a7cdee983f5d57768923ab", + "include/splinterdb/default_data_config.h": "179386f3e224d322a9d7ac97afd929292994a1d890a3254705f9ebea525bce32", + "include/splinterdb/platform_linux/public_platform.h": "c5ded82c5b5bb5bbcedf06a66951eb98dc8892f7b3c4f8ce07c474db98ac68a0", + "include/splinterdb/public_platform.h": "4ce9da3fea9a54fb7bb613ec1458f11f043525efb7d0bc55182f7183dd4a4f6e", + "include/splinterdb/public_util.h": "4e8f6b30408fb4174894d4ba809acff4e96f559370c7749b65bb3724a853603f", + "include/splinterdb/splinterdb.h": "364073691c390663f6b732f57c3ee4bfecc9f68aca33ef6750d6106e04462dc8", + "src/PackedArray.c": "97b64eb0a4f550e55f3385d39d729a878fbc581909ea16ec3b8987a7f46604cc", + "src/PackedArray.h": "e46f854b487a8b130f0613efcab11836012d27c8aacf9ca9e76a2de1a7eb6874", + "src/allocator.c": "96c7f0a5e0808719cf114d267c84da1f057b5eed748b0672d60686d7f130b96f", + "src/allocator.h": "06da1d6a491a15ef5229e8e2e4b4ad6468cb3c9ba620e0d956032dc0a1b6aec1", + "src/batch_rwlock.c": "c043d9e5ead75cd9e1266a6f51fb555012269f3822d7f0233dc2abbd962c3aae", + "src/batch_rwlock.h": "0ff5f0bd7fdf5f9a63ca2f77c387d574d8009ad726b47283c93566cf06bc0927", + "src/blob.c": "75e01e92c06d775b1cb36ea63a7cddf191b5179b94c3e9cd6b5b23ab0480485b", + "src/blob.h": "8f68d2f67de8a9d81a422b28d47aa636cc7234fccc3dbabe59f2355bc361e93c", + "src/blob_build.c": "e484e9c14fd8a9b3c1d211485c35931c944fc66ac4ee78ea20ffc5b3f074b10b", + "src/blob_build.h": "38176881cf9159247e6f578bbc7abb725cfa969c493715df7d1388cb7cbfd119", + "src/btree.c": "8b059dfa240564a7fca21379b467afc7635b9128d2ca72555300333ae06dbc97", + "src/btree.h": "2dc405429d4d84f606412d3a16dc07dd2aee2877524d658e946cb1a8bf1d865e", + "src/btree_private.h": "43693dd3048ae050b3358ccfc9b91371a3d9dff4f3d6d5c8b07445897dee09f6", + "src/cache.h": "d4c1e133d6423fff324d0ebcf5c95e13647c0022a3458935156746b90d816d6d", + "src/clockcache.c": "e0ef2e19a8280849df4ee80d9b8a54a0faaf56a13faefe7a91d54f5b7269f933", + "src/clockcache.h": "68f4e7b019d0440f85929bfc0da1891ce941318ee7dfd14a3d4d923e3d4e72d8", + "src/core.c": "b8d5d7e2432841485468bf958c23f27e112e0370e6929715e44fb76b1b06f7a4", + "src/core.h": "7cf4d086f6d4dcaa4cbd161fef786dbe3cc502ced4512b7fe547c2aa2020462e", + "src/data_blob_build.c": "13475faa56bba94ee26ef7fec4e4da8a5045e0efe510a63291fd5e2697cb5d22", + "src/data_blob_build.h": "dd7f263a0b1138325b211c561ba7289183a2cc8472978cba3ca430295e11e5f3", + "src/data_internal.c": "0a42dd477aaa6ad6c4910841045c4c768ad8d248c0126c64c51035f53e769cde", + "src/data_internal.h": "98b4ebe77576c4cbb7dd9984ca5e297851175494062073f1a4c137c5afb8b587", + "src/default_data_config.c": "161352f5247978f3d57033818d05a79d0a1a0d0ffac2bc9f25f4f33eabca2227", + "src/histogram.c": "24e09dfabd6085021d8a06ef9b10f6e85b24914d95831be359df09f8a9729aff", + "src/histogram.h": "7ccfde6d3ada7ead18fe477c06a6b021db83ca8c041ab0998ca023b945a3c1c7", + "src/iterator.h": "2fcddeb9fa6bd49f4e3da933224219bdb830f9e2f11638ea1ed13eba534fa27f", + "src/log.h": "49a2b0aec91a1c40784b8b3b54f016ef5504c79492b1393674123e27656a09bb", + "src/log_data.h": "8b9198c49c95367f5c66cfa23ec1a3b6cab2abd875fe74c4afb4dafb5a5c44fe", + "src/lookup_result.h": "7528b988a28ef91b6bd7a4fa10dfa51b9ce2475e801b3b6a8bf30adf88f1ef16", + "src/memtable.c": "7db3b4754497c73b4071c7e7e591cc76fc226197fb70d98fb46caf139b3953f4", + "src/memtable.h": "244cac7cf304df85a25c8bede2df106d6f840d65bbd1a3551b3d9f2ed9129fc1", + "src/merge.c": "df98421e597d1edeac871755f0473055b590791af8efad495575d20260e46a12", + "src/merge.h": "c17e2a0b6272ec52e6d1e80acd9df7dc85221f186e6c60e323c37b467dc5c41a", + "src/mini_allocator.c": "d620cbdff9867ce64c752992ede58783010fb09366220ad3bd6d8518911205ca", + "src/mini_allocator.h": "39d245b58914d28b26dfe230b9de15582382cec1c2a90fa14db19f5f2c203632", + "src/notification.c": "2f3bcda49d8ce35ad532fef8ec8f4aeca48bec280cdf6210c21d55cf2aa4a661", + "src/notification.h": "47eb73f5813d6f54658f339ce44d6a475f8f64f2b2eb6368261e839a090f083f", + "src/ondisk_ref.c": "2ace8cbbce3f80a54ff7762ad03a79a564665fa660394f1d3446235a3e40ed7b", + "src/ondisk_ref.h": "643325c8b4f16fab41d8c884c97bdfcf6c86e5178398508fa0f81b6b0606926d", + "src/pcq.h": "862cd346b0097014ccae797dc1463e7b3fc387f409319d5adcef7613b6f82f84", + "src/platform_linux/async.h": "41383b57e21f0f51d737749a4ce7c2c5330ac9363e742330625c9aedd4f13a52", + "src/platform_linux/laio.c": "0ce628806e6896070359a9e39d069f326f460ab697c6c17ff17d171614290881", + "src/platform_linux/laio.h": "f3987e22f1fd7a9d6b46c79f84e75559aec5410239a13d52a7a5343ba792b28e", + "src/platform_linux/platform.h": "f1c3d7c5bcd089ed7a7df129d3b8d00d7617de9a7aa6f9d1f7deeb739281e768", + "src/platform_linux/platform_assert.c": "112e803b6db03dd8c14a3bfc3f1139bbcc4b81f1d5e24a232b7b99ce89146c2d", + "src/platform_linux/platform_assert.h": "751dccceac9ce4a97d25d441b5216ced5ee5f079a13c9ccd43551b0994aa7cc2", + "src/platform_linux/platform_backtrace.h": "ef13ccbf4ae4747b0de244a37c026ad8769018d84f9fc4d6493db43d9714358c", + "src/platform_linux/platform_buffer.c": "b790d645b3dc5f638da871bf9611865dc700927f3b594a1d645b0350bd1bd027", + "src/platform_linux/platform_buffer.h": "72e016dbb19b8250378b3a5c79fbebf9c3d72a31d07a232f36431e04f0e4afa6", + "src/platform_linux/platform_condvar.c": "e595f69194accb3f3460d6fc9a4a9144bd7f495dbeb77f11bf3616dc0e9b03ec", + "src/platform_linux/platform_condvar.h": "a4ee02ae5069637bce4cae07f3ed9869925f17a9e6e1b04d69baedfc148fd493", + "src/platform_linux/platform_hash.h": "f69f5ed6218bfe62c62718b387e1c110e9fca46c07603d4d51ccc4fc782b09ab", + "src/platform_linux/platform_heap.c": "72bb2753f525e56f8c398988a787feb38f02efaa749e5d57cb476f967269c479", + "src/platform_linux/platform_heap.h": "921f3113d20b44c36817c284702e965e01769596ff012edd6ceaa70cbdbbd10a", + "src/platform_linux/platform_io.c": "875a7331f8bfa46e9a6961f50acb40b107d40883839c94def52a289a521c66be", + "src/platform_linux/platform_io.h": "69ae9556daf1c7a69e007abd494bf9c86ec81c08bca1e032663009cef95b2fac", + "src/platform_linux/platform_log.c": "318bd17c858475a3a91c5e9641b52808f7c4194e1c6fc9a7d801392b04c49538", + "src/platform_linux/platform_log.h": "569484386150fcfbf5dde722c5b8e34d67d65be3454e3e009cb9df41f11efea6", + "src/platform_linux/platform_machine.h": "c02745313c7268dbe99efa2b93dd4d37640b5a7aa2e88c78fd5b73742f252c7d", + "src/platform_linux/platform_mutex.c": "7ec1dd6a5397217d28fd85c10648d29a70bf66649f59f76d7387236031b15bf3", + "src/platform_linux/platform_mutex.h": "688991d6a7dc57a7a88f565541561344cdfbb1733ca6073f1655d6c436ed6544", + "src/platform_linux/platform_random.h": "184ac9da9c38a6c10275891be61ccb91beaf2234ebb9ce046d059cb16824d68f", + "src/platform_linux/platform_semaphore.h": "9fdb7cfeaf36c8bf9aa346a16466ed6bc50bea76fdb5722c86001563d55bedfe", + "src/platform_linux/platform_sleep.h": "34aeaea35133bf33e47eb1d9ace429297644e13bc100d90ebbd205bceb58d3ab", + "src/platform_linux/platform_sort.h": "d374788510f94a8c3b86ff6bfa1f6a3ff8ac33168eb26f6f7774984252f6d01e", + "src/platform_linux/platform_spinlock.h": "7b3f0409109f3341f2287fe7cd8fb69faac4a74b009a44c1eb83220f71e94de0", + "src/platform_linux/platform_status.h": "2749285ea1322e5c1639019b728fde4b87b0cae6f71a7fc16e420bd592d38c00", + "src/platform_linux/platform_string.h": "fa41faf9c6661c62ece783740250232709656b0a8677f542e390a5f3b0d834a6", + "src/platform_linux/platform_threads.c": "789465a2c6ac7ac501e5878f429cc0dd9b163145a522e247a4833e3de7af8f2a", + "src/platform_linux/platform_threads.h": "653e070115131756e6aaffc3c2764a6437fb39ff96df4afc8a01e6e751c9d622", + "src/platform_linux/platform_time.h": "f212a08cf8432693711cecfd389f9ed3d0a70e0c056dc062321bd11c4a8cbec7", + "src/platform_linux/platform_trace.c": "4a68e89f214920dd2762d072a0ad8e7c80d9b1733a64e4a7e3dd4bb04b91c352", + "src/platform_linux/platform_trace.h": "345f7283b01e5a9b4d5da7bea66ad44ff61f53155417c97990e23bb8fedadf33", + "src/platform_linux/platform_typed_alloc.h": "42c1d2bbd56c183d11768a5f43d440cf3c1bfe329c4352afa6acd3bc2b191a8e", + "src/platform_linux/platform_units.h": "1cb22a07b33ec8b58f9381438f480f561d12b838f315a117cc1f979e95b1ef65", + "src/platform_linux/platform_util.h": "4fa2702b3919eb1a7efcf29457f5ee9a3d7a7b4c38fed4606264562022fa2638", + "src/platform_linux/poison.h": "a32dfa445439eb08913e519117e7b11f72b2c3550577a74769ed935a792c42c5", + "src/platform_linux/shmalloc.c": "e4631d63a9db1ad58472ad3d034d081f2bb9a97f34c5898f7c848dda923727f8", + "src/platform_linux/shmalloc.h": "081a82473f6021f8ed628e79f8ae5dc7104988981f8c3b25749ea42c8158113d", + "src/prefetch.h": "9d55d9dad64e3795c81856bc38688936e4e74e488dfe2d29a1f5438ad000df42", + "src/rc_allocator.c": "23ef0da6fb2d5114fe403403c8e535bf8ba19ed2c8bef4397212933cedfc8b9e", + "src/rc_allocator.h": "5f1fae7e32207a4e0cd5c4f799c9b802ce116dd381ee5c074c21d64a229dc3ad", + "src/routing_filter.c": "a913ee83ca39267b6da882bae0db05b1453f39c863f31ec6fea9844998f72f03", + "src/routing_filter.h": "3104de8135817e33b585ba489bc4821a458f6a9c2ce9e561ea8be2fde28e43ba", + "src/shard_log.c": "551efce92b8e60bbc29d0a7724c0382f3d4e41c84e6e6915c2252e52410fae86", + "src/shard_log.h": "bde00a5cbe72e72ac7133006796e61f046ae758e8e4667aa610e184ed6ef4fd7", + "src/splinterdb.c": "1de96ba09714d40ee673ebdf05654ba85a447548667681476c7bd311025b67ca", + "src/splinterdb_tests_private.h": "579031767b8869f49cf1e3fc94caeec13a1362dfd047782538df40b281c93414", + "src/superblock.c": "e3fbe387ac4fef197c11977e35e3aecf0d31c27c9d4932e4ec54b1bfc391620a", + "src/superblock.h": "cec63cb28426e2d23140b3ac310a86ae9635d6b3ec429abe7c469bc0de06aa73", + "src/task.c": "35886e598b4464af16e76497917955d3d55c6c077fbcf80fb3fd2284c8ddc08d", + "src/task.h": "35405e04fbdcef9bc5c5d6ef43debc4b0889d6bc4d8d530649b0c45a34012825", + "src/trunk.c": "cff7a5eef6945c36b7784c888bdf6ab2510d132d87d963ba2c909322c13faf36", + "src/trunk.h": "5d8674472258da1be0a84e20a20869c84dc846254d7604e5799297491e6f91e1", + "src/util.c": "969df408c4c393f73feb05dfa37ebe1ed8be627cef630061830e4992c4c1fd9c", + "src/util.h": "9fa455f5a6268fa4a6864d444849b0872391af00512932c1587363cfd38304ef", + "src/vector.h": "6855dd0430f1a868d098797cbd4f3453ab542b5f6fbe716c79871124bee200a9", + "src/writeback_set.c": "cb485716f57a3183ce82543848edaf61f6568c9ff0445ea5ff989f95c00848ff", + "src/writeback_set.h": "68d6f589d24719a09e2680b05a1eb7e36c22b1fab978768f5b54f958f667aec9" + }, + "library_build_configuration": { + "build-config.fb6bc8eeb8390429253883c09491f5bd": { + "sha256": "d7ab1b7d5e3c610072b971a8c578b11ad3d1c79ad075ae2703dc0ca240a6d800", + "text": "BUILD_ROOT=build-main-7bb5fab\nMAKEFLAGS= -j6 --jobserver-auth=fifo:/tmp/splinter-mainmerge-disk-format-release-build-4n8c6t_q/GMfifo7 -- BUILD_MODE=release BUILD_ROOT=build-main-7bb5fab\nBUILD_MODE=release\nQT_ACCESSIBILITY=1\nCC = cc\nDEPFLAGS = -MMD -MP\nCFLAGS = -D_GNU_SOURCE -ggdb -Wall -pthread -Wfatal-errors -Werror -Wvla -DXXH_STATIC_LINKING_ONLY -fPIC -DSPLINTERDB_PLATFORM_DIR=platform_linux -O3 -ffast-math -flto -DPLATFORM_MEMORY_FAULT_INJECTION=0\nTEST_MEMORY_FAULT_DISABLED_CFLAGS = -DPLATFORM_MEMORY_FAULT_INJECTION_DISABLED_IN_THIS_FILE\nINCLUDE = -I include -I src -I src/platform_linux -I tests\nTARGET_ARCH =\nLD = cc\nLDFLAGS = -ggdb -pthread -O3 -ffast-math -flto\nLIBS = -lm -lpthread -laio -lxxhash\nAR = ar\n" + } + } +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/00-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/00-main.log new file mode 100644 index 000000000..d44e58a68 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/00-main.log @@ -0,0 +1,20 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=5.0 warmup=1.0 rep=0 sabotage=0 +open took 0.006 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.11 ops/s= 18349899 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.43s ops/cpu-s=4662168 rd=146.1 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=1 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=765333 notfound=0 +== point:uniform@8 T=8 ops=765333 secs= 5.00 ops/s= 153061 lat us p50=60.93 p99=105.47 p99.9=161.79 max=3031.5 mean=52.23 + cpu=4.10s ops/cpu-s=186512 rd=3228.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 310411 ops (+31039 ops/s) +== range:uniform:1@8 T=8 ops=317303 secs= 5.00 ops/s= 63458 lat us p50=123.90 p99=251.90 p99.9=331.78 max=2166.9 mean=126.01 + cpu=42.18s ops/cpu-s=7523 rd=19812.5 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=121.86 p99=251.90 mean=125.27 per-tuple=nan ns deinit mean=0.70 us tuples/s=63458 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) +close took 0.076 s +== range:uniform:10000@8 T=8 ops=4207 secs= 5.00 ops/s= 841 lat us p50=9830.40 p99=23330.82 p99.9=28049.41 max=32120.6 mean=9504.55 + cpu=12.69s ops/cpu-s=331 rd=2374649.9 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=274.43 p99=909.31 mean=318.56 per-tuple=920.2 ns deinit mean=1.09 us tuples/s=8399365 + insert/run=5.00s drain(close)=0.08s total=5.08s drain_cpu=0.01s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=472989696 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/01-disk-format.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/01-disk-format.log new file mode 100644 index 000000000..dd0549be5 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/01-disk-format.log @@ -0,0 +1,20 @@ +kvbench version=disk-format splinter=splinterdb_build_version fb878f13-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=5.0 warmup=1.0 rep=1 sabotage=0 +open took 0.006 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.11 ops/s= 18254307 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.44s ops/cpu-s=4525030 rd=146.2 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=762416 notfound=0 +== point:uniform@8 T=8 ops=762416 secs= 5.00 ops/s= 152478 lat us p50=60.93 p99=103.42 p99.9=149.50 max=1495.0 mean=52.43 + cpu=4.18s ops/cpu-s=182614 rd=3228.8 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 305015 ops (+30500 ops/s) +== range:uniform:1@8 T=8 ops=311589 secs= 5.00 ops/s= 62315 lat us p50=125.95 p99=256.00 p99.9=323.58 max=1698.4 mean=128.31 + cpu=42.27s ops/cpu-s=7371 rd=19804.3 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=123.90 p99=256.00 mean=127.52 per-tuple=nan ns deinit mean=0.75 us tuples/s=62315 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) +close took 0.097 s +== range:uniform:10000@8 T=8 ops=4105 secs= 5.00 ops/s= 821 lat us p50=10092.54 p99=23330.82 p99.9=27525.12 max=29640.1 mean=9743.22 + cpu=13.08s ops/cpu-s=314 rd=2378060.8 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=274.43 p99=942.08 mean=320.51 per-tuple=944.1 ns deinit mean=1.11 us tuples/s=8195330 + insert/run=5.00s drain(close)=0.10s total=5.10s drain_cpu=0.01s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=475049984 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/11-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/11-main.log new file mode 100644 index 000000000..5198b3d27 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/11-main.log @@ -0,0 +1,20 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=5.0 warmup=1.0 rep=11 sabotage=0 +open took 0.007 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.13 ops/s= 15684927 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.52s ops/cpu-s=3869834 rd=146.6 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=814537 notfound=0 +== point:uniform@8 T=8 ops=814537 secs= 5.00 ops/s= 162900 lat us p50=59.90 p99=99.33 p99.9=128.00 max=1609.7 mean=49.07 + cpu=4.89s ops/cpu-s=166490 rd=3229.1 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 308561 ops (+30854 ops/s) +== range:uniform:1@8 T=8 ops=318141 secs= 5.00 ops/s= 63625 lat us p50=121.86 p99=243.71 p99.9=315.39 max=2542.9 mean=125.66 + cpu=42.41s ops/cpu-s=7502 rd=19811.1 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=121.86 p99=243.71 mean=124.77 per-tuple=nan ns deinit mean=0.84 us tuples/s=63625 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) +close took 0.077 s +== range:uniform:10000@8 T=8 ops=4176 secs= 5.00 ops/s= 835 lat us p50=10092.54 p99=22806.53 p99.9=26476.54 max=27125.7 mean=9579.22 + cpu=14.40s ops/cpu-s=290 rd=2373442.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=260.10 p99=909.31 mean=305.46 per-tuple=928.6 ns deinit mean=1.29 us tuples/s=8336860 + insert/run=5.00s drain(close)=0.08s total=5.08s drain_cpu=0.01s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=472989696 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/21-disk-format.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/21-disk-format.log new file mode 100644 index 000000000..a8e858111 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/21-disk-format.log @@ -0,0 +1,20 @@ +kvbench version=disk-format splinter=splinterdb_build_version fb878f13-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=5.0 warmup=1.0 rep=21 sabotage=0 +open took 0.007 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.12 ops/s= 17342745 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.54s ops/cpu-s=3723584 rd=146.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=3 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=838144 notfound=0 +== point:uniform@8 T=8 ops=838144 secs= 5.00 ops/s= 167622 lat us p50=51.71 p99=99.33 p99.9=133.12 max=4010.9 mean=47.69 + cpu=5.07s ops/cpu-s=165174 rd=3229.5 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 314939 ops (+31492 ops/s) +== range:uniform:1@8 T=8 ops=323674 secs= 5.00 ops/s= 64732 lat us p50=119.81 p99=239.62 p99.9=471.04 max=3711.5 mean=123.51 + cpu=42.40s ops/cpu-s=7634 rd=19790.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=117.76 p99=239.62 mean=122.61 per-tuple=nan ns deinit mean=0.85 us tuples/s=64732 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) +close took 0.094 s +== range:uniform:10000@8 T=8 ops=4249 secs= 5.00 ops/s= 850 lat us p50=9830.40 p99=23330.82 p99.9=27525.12 max=32955.7 mean=9412.89 + cpu=14.92s ops/cpu-s=285 rd=2373012.6 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=256.00 p99=909.31 mean=298.06 per-tuple=913.0 ns deinit mean=1.28 us tuples/s=8483329 + insert/run=5.00s drain(close)=0.09s total=5.09s drain_cpu=0.01s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=475049984 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/22-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/22-main.log new file mode 100644 index 000000000..46f35d5fe --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/22-main.log @@ -0,0 +1,20 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=5.0 warmup=1.0 rep=22 sabotage=0 +open took 0.006 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.13 ops/s= 15968853 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.55s ops/cpu-s=3624521 rd=146.6 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=840578 notfound=0 +== point:uniform@8 T=8 ops=840578 secs= 5.00 ops/s= 168110 lat us p50=51.71 p99=99.33 p99.9=130.05 max=2039.3 mean=47.55 + cpu=5.08s ops/cpu-s=165502 rd=3229.6 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 315837 ops (+31582 ops/s) +== range:uniform:1@8 T=8 ops=324892 secs= 5.00 ops/s= 64975 lat us p50=119.81 p99=239.62 p99.9=348.16 max=2233.5 mean=123.05 + cpu=42.41s ops/cpu-s=7660 rd=19820.9 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=117.76 p99=239.62 mean=122.13 per-tuple=nan ns deinit mean=0.87 us tuples/s=64975 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) +close took 0.083 s +== range:uniform:10000@8 T=8 ops=4200 secs= 5.00 ops/s= 840 lat us p50=9830.40 p99=21757.95 p99.9=27000.83 max=28803.9 mean=9525.33 + cpu=15.35s ops/cpu-s=274 rd=2364931.9 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=256.00 p99=892.93 mean=301.69 per-tuple=923.9 ns deinit mean=1.41 us tuples/s=8385217 + insert/run=5.00s drain(close)=0.08s total=5.08s drain_cpu=0.01s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=472985600 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/manifest.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/manifest.json new file mode 100644 index 000000000..9253b951d --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/manifest.json @@ -0,0 +1,417 @@ +{ + "mode": "disk", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "scope": "Updated feature branches versus fresh main; not an isolation of merge cost", + "runner_sha256": "536f10a1a7fd4b1e8e0250ea01cfe36793c7d6e2345f141092616ec06f64ee50", + "runs": [ + { + "label": "00-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/results.csv", + "--vtag", + "main", + "--stag", + "disk", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791906930.3914676, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791906948.6450965, + "log_sha256": "31d7f4877605590f45d53a31f1da36cfd3464ed056f9aea89f6083b962487883", + "process_cpu_s": 71.248714, + "process_read_bytes": 22774968320, + "process_write_bytes": 319488, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-main-ordinary.db", + "fixture_sha256": "b074d0238b42f37eb1be4ea4994e5b4dc0252e2c4a840727193f3c9629ba142a", + "allocated_bytes": 473116672, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 472989696 + }, + { + "label": "01-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/results.csv", + "--vtag", + "disk-format", + "--stag", + "disk", + "--otag", + "1", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791906949.0863597, + "binary_sha256": "733c71174e9356864c326570c69279aacafc92f2dd7286813f9a21cdf9335ddc", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791906967.3374197, + "log_sha256": "33d0fa616187da01c016c5cbd39e028762a63f877ace9c9e3c84a1297eaf6269", + "process_cpu_s": 71.95582199999998, + "process_read_bytes": 22369202176, + "process_write_bytes": 319488, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-disk-format-ordinary.db", + "fixture_sha256": "122ad4abc27ade97ea7cca2c1acb1ac7e4b876401a76f0f2c2f22d6bc8be235a", + "allocated_bytes": 475205632, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 475049984 + }, + { + "label": "11-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/results.csv", + "--vtag", + "main", + "--stag", + "disk", + "--otag", + "11", + "--rep", + "11", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791907136.6043751, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791907154.9016702, + "log_sha256": "7c21a91723c434998d5c9b8a8dd15c86bbea62e8b787cc23f4ed16acea5a5edf", + "process_cpu_s": 74.70139499999988, + "process_read_bytes": 22838931456, + "process_write_bytes": 323584, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-main-ordinary.db", + "fixture_sha256": "b074d0238b42f37eb1be4ea4994e5b4dc0252e2c4a840727193f3c9629ba142a", + "allocated_bytes": 473116672, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 472989696 + }, + { + "label": "21-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/results.csv", + "--vtag", + "disk-format", + "--stag", + "disk", + "--otag", + "21", + "--rep", + "21", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791907324.704277, + "binary_sha256": "733c71174e9356864c326570c69279aacafc92f2dd7286813f9a21cdf9335ddc", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791907343.0033288, + "log_sha256": "e9a3eed5b11581ea5e2fc9a837325a51ebb80ab369c33222d4e17a7bb562b266", + "process_cpu_s": 75.60325399999999, + "process_read_bytes": 23315443712, + "process_write_bytes": 315392, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-disk-format-ordinary.db", + "fixture_sha256": "122ad4abc27ade97ea7cca2c1acb1ac7e4b876401a76f0f2c2f22d6bc8be235a", + "allocated_bytes": 475205632, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 475049984 + }, + { + "label": "22-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "5", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk/results.csv", + "--vtag", + "main", + "--stag", + "disk", + "--otag", + "22", + "--rep", + "22", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791907343.500252, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791907361.755288, + "log_sha256": "5086e61f0aaac72a1061e180c0877301383d9148d4822a6439cb0a2db936c1bd", + "process_cpu_s": 76.16407100000004, + "process_read_bytes": 23167078400, + "process_write_bytes": 319488, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-main-ordinary.db", + "fixture_sha256": "b074d0238b42f37eb1be4ea4994e5b4dc0252e2c4a840727193f3c9629ba142a", + "allocated_bytes": 473116672, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 472985600 + } + ], + "order": [ + "main", + "disk-format", + "filtered-ranges", + "log-group-limit", + "lazy-merge", + "async-durable-barrier", + "cache-invalid-addresses", + "directory-direct", + "directory-bucket", + "directory-bucket-simd", + "directory-bucket-atomic", + "main", + "directory-bucket-atomic", + "directory-bucket-simd", + "directory-bucket", + "directory-direct", + "cache-invalid-addresses", + "async-durable-barrier", + "lazy-merge", + "log-group-limit", + "filtered-ranges", + "disk-format", + "main" + ], + "phases": [ + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "complete": true +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/results.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/results.csv new file mode 100644 index 000000000..df27f9da6 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/disk/results.csv @@ -0,0 +1,21 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,disk,0,0,warm,8,0,2000000,2000000,0.109,18349899.4,nan,nan,nan,0.000,nan,0.429,4662167.7,292171776,32768,146.09,0.02,nan,nan,nan,nan,nan,2000000,18349899.4,0.109,0.000,0.109,0.000,0,0,nan,0,0,0.006,0,0,0,0,1 +main,disk,0,0,point:uniform@8,8,0,2000000,765333,5.000,153061.5,60.928,105.472,161.792,3031.537,52.230,4.103,186511.7,2471006208,0,3228.67,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,disk,0,0,range:uniform:1@8,8,0,2000000,317303,5.000,63458.1,123.904,251.904,331.776,2166.861,126.006,42.179,7522.8,6286577664,0,19812.54,0.00,121.856,251.904,125.268,nan,0.697,317303,63458.1,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,disk,0,0,range:uniform:10000@8,8,0,2000000,4207,5.000,841.4,9830.400,23330.816,28049.408,32120.565,9504.553,12.693,331.4,9990152192,0,2374649.91,0.00,274.432,909.312,318.556,920.15,1.094,41998346,8399364.6,5.000,0.076,5.076,0.007,0,274432,nan,2513178624,472989696,0.006,0,0,0,0,0 +disk-format,disk,1,0,warm,8,1,2000000,2000000,0.110,18254306.9,nan,nan,nan,0.000,nan,0.442,4525030.2,292478976,32768,146.24,0.02,nan,nan,nan,nan,nan,2000000,18254306.9,0.110,0.000,0.110,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +disk-format,disk,1,0,point:uniform@8,8,1,2000000,762416,5.000,152478.2,60.928,103.424,149.504,1495.004,52.431,4.175,182613.6,2461700096,0,3228.81,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +disk-format,disk,1,0,range:uniform:1@8,8,1,2000000,311589,5.000,62315.3,125.952,256.000,323.584,1698.373,128.313,42.273,7370.9,6170816512,0,19804.35,0.00,123.904,256.000,127.518,nan,0.750,311589,62315.3,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +disk-format,disk,1,0,range:uniform:10000@8,8,1,2000000,4105,5.000,821.0,10092.544,23330.816,27525.120,29640.111,9743.220,13.081,313.8,9761939456,0,2378060.77,0.00,274.432,942.080,320.510,944.15,1.107,40978346,8195330.2,5.000,0.097,5.097,0.007,0,274432,nan,2513178624,475049984,0.006,0,0,0,0,0 +main,disk,11,0,warm,8,11,2000000,2000000,0.128,15684927.3,nan,nan,nan,0.000,nan,0.517,3869834.3,293187584,32768,146.59,0.02,nan,nan,nan,nan,nan,2000000,15684927.3,0.128,0.000,0.128,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,disk,11,0,point:uniform@8,8,11,2000000,814537,5.000,162899.9,59.904,99.328,128.000,1609.669,49.069,4.892,166489.9,2630258688,0,3229.15,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,disk,11,0,range:uniform:1@8,8,11,2000000,318141,5.000,63625.4,121.856,243.712,315.392,2542.922,125.665,42.405,7502.4,6302711808,0,19811.06,0.00,121.856,243.712,124.774,nan,0.841,318141,63625.4,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,disk,11,0,range:uniform:10000@8,8,11,2000000,4176,5.000,835.2,10092.544,22806.528,26476.544,27125.723,9579.215,14.402,290.0,9911496704,0,2373442.70,0.00,260.096,909.312,305.463,928.60,1.295,41685785,8336859.6,5.000,0.077,5.077,0.007,0,274432,nan,2513178624,472989696,0.007,0,0,0,0,0 +disk-format,disk,21,0,warm,8,21,2000000,2000000,0.115,17342744.7,nan,nan,nan,0.000,nan,0.537,3723583.5,292892672,32768,146.45,0.02,nan,nan,nan,nan,nan,2000000,17342744.7,0.115,0.000,0.115,0.000,0,0,nan,0,0,0.007,0,0,0,0,3 +disk-format,disk,21,0,point:uniform@8,8,21,2000000,838144,5.000,167622.5,51.712,99.328,133.120,4010.904,47.685,5.074,165173.7,2706763776,0,3229.47,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +disk-format,disk,21,0,range:uniform:1@8,8,21,2000000,323674,5.000,64731.6,119.808,239.616,471.040,3711.498,123.513,42.397,7634.4,6405734400,0,19790.70,0.00,117.760,239.616,122.614,nan,0.847,323674,64731.6,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +disk-format,disk,21,0,range:uniform:10000@8,8,21,2000000,4249,5.000,849.8,9830.400,23330.816,27525.120,32955.652,9412.894,14.924,284.7,10082930688,0,2373012.64,0.00,256.000,909.312,298.060,912.97,1.279,42418346,8483329.3,5.000,0.094,5.094,0.007,0,274432,nan,2513178624,475049984,0.007,0,0,0,0,0 +main,disk,22,0,warm,8,22,2000000,2000000,0.125,15968853.5,nan,nan,nan,0.000,nan,0.552,3624521.3,293216256,32768,146.61,0.02,nan,nan,nan,nan,nan,2000000,15968853.5,0.125,0.000,0.125,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,disk,22,0,point:uniform@8,8,22,2000000,840578,5.000,168110.0,51.712,99.328,130.048,2039.305,47.547,5.079,165502.3,2714718208,0,3229.59,0.00,nan,nan,nan,nan,nan,0,0.0,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,disk,22,0,range:uniform:1@8,8,22,2000000,324892,5.000,64975.4,119.808,239.616,348.160,2233.541,123.048,42.414,7660.0,6439641088,0,19820.87,0.00,117.760,239.616,122.130,nan,0.867,324892,64975.4,5.000,0.000,5.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,disk,22,0,range:uniform:10000@8,8,22,2000000,4200,5.000,840.0,9830.400,21757.952,27000.832,28803.887,9525.329,15.351,273.6,9932713984,0,2364931.90,0.00,256.000,892.928,301.690,923.91,1.413,41928346,8385216.8,5.000,0.083,5.083,0.011,0,274432,nan,2513178624,472985600,0.006,0,0,0,0,0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/feature-summaries.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/feature-summaries.json new file mode 100644 index 000000000..4d9a23e69 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/feature-summaries.json @@ -0,0 +1,354 @@ +{ + "hash": [ + { + "profile": "main", + "limit": 1, + "samples": 2, + "ns_per_query_mean": 27787.25, + "ns_per_query_min": 26622.0, + "ns_per_query_max": 28952.5, + "latency_delta_main_pct": 0.0, + "visited_mean": 16404.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + }, + { + "profile": "filtered-ranges", + "limit": 1, + "samples": 2, + "ns_per_query_mean": 26204.55, + "ns_per_query_min": 25561.8, + "ns_per_query_max": 26847.3, + "latency_delta_main_pct": -5.6957777397907305, + "visited_mean": 1000.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + }, + { + "profile": "main", + "limit": 2, + "samples": 2, + "ns_per_query_mean": 24743.4, + "ns_per_query_min": 23851.0, + "ns_per_query_max": 25635.8, + "latency_delta_main_pct": 0.0, + "visited_mean": 48404.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + }, + { + "profile": "filtered-ranges", + "limit": 2, + "samples": 2, + "ns_per_query_mean": 20609.85, + "ns_per_query_min": 20525.1, + "ns_per_query_max": 20694.6, + "latency_delta_main_pct": -16.70566696573633, + "visited_mean": 2000.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + }, + { + "profile": "main", + "limit": 10, + "samples": 2, + "ns_per_query_mean": 33161.2, + "ns_per_query_min": 32551.2, + "ns_per_query_max": 33771.2, + "latency_delta_main_pct": 0.0, + "visited_mean": 304404.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + }, + { + "profile": "filtered-ranges", + "limit": 10, + "samples": 2, + "ns_per_query_mean": 33346.8, + "ns_per_query_min": 33010.8, + "ns_per_query_max": 33682.8, + "latency_delta_main_pct": 0.5596902404014559, + "visited_mean": 10000.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + }, + { + "profile": "main", + "limit": 100, + "samples": 2, + "ns_per_query_mean": 152074.95, + "ns_per_query_min": 150518.6, + "ns_per_query_max": 153631.3, + "latency_delta_main_pct": 0.0, + "visited_mean": 3184404.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + }, + { + "profile": "filtered-ranges", + "limit": 100, + "samples": 2, + "ns_per_query_mean": 137488.2, + "ns_per_query_min": 136036.1, + "ns_per_query_max": 138940.3, + "latency_delta_main_pct": -9.59181640368778, + "visited_mean": 100000.0, + "verified": true, + "verification_kind": "successful process with harness assertions and matching result checksums" + } + ], + "barrier": [ + { + "workload": "barrier", + "phase": "wal0-workers0-sync", + "version": "main", + "samples": 2, + "ops_s_mean": 35984.341, + "ops_s_min": 31071.613, + "ops_s_max": 40897.069, + "baseline_ops_s_mean": 35984.341, + "baseline_ops_s_min": 31071.613, + "baseline_ops_s_max": 40897.069, + "delta_pct": 0.0, + "ranges_overlap": true, + "launch_p99_us": 18830.339, + "completion_p99_us": 18830.339, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 9.038776397705078, + "read_bytes_update": 0.0, + "write_bytes_update": 5924.65625 + }, + { + "workload": "barrier", + "phase": "wal0-workers0-sync", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 58274.859500000006, + "ops_s_min": 57551.641, + "ops_s_max": 58998.078, + "baseline_ops_s_mean": 35984.341, + "baseline_ops_s_min": 31071.613, + "baseline_ops_s_max": 40897.069, + "delta_pct": 61.945051321073265, + "ranges_overlap": false, + "launch_p99_us": 15769.339, + "completion_p99_us": 15769.339, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 8.89599609375, + "read_bytes_update": 0.03125, + "write_bytes_update": 5924.65625 + }, + { + "workload": "barrier", + "phase": "wal0-workers0-async", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 59874.4315, + "ops_s_min": 59568.233, + "ops_s_max": 60180.63, + "baseline_ops_s_mean": 35984.341, + "baseline_ops_s_min": 31071.613, + "baseline_ops_s_max": 40897.069, + "delta_pct": 66.39024041040517, + "ranges_overlap": false, + "launch_p99_us": 0.9764999999999999, + "completion_p99_us": 14594.8035, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 8.942153930664062, + "read_bytes_update": 0.0, + "write_bytes_update": 5924.65625 + }, + { + "workload": "barrier", + "phase": "wal0-workers1-sync", + "version": "main", + "samples": 2, + "ops_s_mean": 32877.4345, + "ops_s_min": 32685.772, + "ops_s_max": 33069.097, + "baseline_ops_s_mean": 32877.4345, + "baseline_ops_s_min": 32685.772, + "baseline_ops_s_max": 33069.097, + "delta_pct": 0.0, + "ranges_overlap": true, + "launch_p99_us": 21590.944499999998, + "completion_p99_us": 21590.944499999998, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 9.591472625732422, + "read_bytes_update": 0.046875, + "write_bytes_update": 5944.890625 + }, + { + "workload": "barrier", + "phase": "wal0-workers1-sync", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 32816.2485, + "ops_s_min": 32566.888, + "ops_s_max": 33065.609, + "baseline_ops_s_mean": 32877.4345, + "baseline_ops_s_min": 32685.772, + "baseline_ops_s_max": 33069.097, + "delta_pct": -0.18610332871319413, + "ranges_overlap": true, + "launch_p99_us": 21671.515, + "completion_p99_us": 21671.515, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 9.660575866699219, + "read_bytes_update": 0.015625, + "write_bytes_update": 5946.984375 + }, + { + "workload": "barrier", + "phase": "wal0-workers1-async", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 32623.6835, + "ops_s_min": 31925.049, + "ops_s_max": 33322.318, + "baseline_ops_s_mean": 32877.4345, + "baseline_ops_s_min": 32685.772, + "baseline_ops_s_max": 33069.097, + "delta_pct": -0.7718090047445836, + "ranges_overlap": true, + "launch_p99_us": 2.8975, + "completion_p99_us": 23050.2955, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 9.739543914794922, + "read_bytes_update": 0.03125, + "write_bytes_update": 5946.421875 + }, + { + "workload": "barrier", + "phase": "wal1-workers0-sync", + "version": "main", + "samples": 2, + "ops_s_mean": 63924.082500000004, + "ops_s_min": 63287.752, + "ops_s_max": 64560.413, + "baseline_ops_s_mean": 63924.082500000004, + "baseline_ops_s_min": 63287.752, + "baseline_ops_s_max": 64560.413, + "delta_pct": 0.0, + "ranges_overlap": true, + "launch_p99_us": 4813.8705, + "completion_p99_us": 4813.8705, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 1.2700424194335938, + "read_bytes_update": 0.0, + "write_bytes_update": 206.3828125 + }, + { + "workload": "barrier", + "phase": "wal1-workers0-sync", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 127121.751, + "ops_s_min": 60090.385, + "ops_s_max": 194153.117, + "baseline_ops_s_mean": 63924.082500000004, + "baseline_ops_s_min": 63287.752, + "baseline_ops_s_max": 64560.413, + "delta_pct": 98.86363015065565, + "ranges_overlap": true, + "launch_p99_us": 4118.6775, + "completion_p99_us": 4118.6775, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 1.2528772354125977, + "read_bytes_update": 0.0, + "write_bytes_update": 206.3828125 + }, + { + "workload": "barrier", + "phase": "wal1-workers0-async", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 236974.144, + "ops_s_min": 200297.973, + "ops_s_max": 273650.315, + "baseline_ops_s_mean": 63924.082500000004, + "baseline_ops_s_min": 63287.752, + "baseline_ops_s_max": 64560.413, + "delta_pct": 270.7118424421656, + "ranges_overlap": false, + "launch_p99_us": 3.5060000000000002, + "completion_p99_us": 2190.8725, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 1.2635889053344727, + "read_bytes_update": 0.0, + "write_bytes_update": 206.3828125 + }, + { + "workload": "barrier", + "phase": "wal1-workers1-sync", + "version": "main", + "samples": 2, + "ops_s_mean": 62594.769, + "ops_s_min": 61606.519, + "ops_s_max": 63583.019, + "baseline_ops_s_mean": 62594.769, + "baseline_ops_s_min": 61606.519, + "baseline_ops_s_max": 63583.019, + "delta_pct": 0.0, + "ranges_overlap": true, + "launch_p99_us": 5467.197, + "completion_p99_us": 5467.197, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 1.2778234481811523, + "read_bytes_update": 0.0, + "write_bytes_update": 206.6484375 + }, + { + "workload": "barrier", + "phase": "wal1-workers1-sync", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 68395.2785, + "ops_s_min": 67544.975, + "ops_s_max": 69245.582, + "baseline_ops_s_mean": 62594.769, + "baseline_ops_s_min": 61606.519, + "baseline_ops_s_max": 63583.019, + "delta_pct": 9.266763968727165, + "ranges_overlap": false, + "launch_p99_us": 4940.577, + "completion_p99_us": 4940.577, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 1.2629365921020508, + "read_bytes_update": 0.0, + "write_bytes_update": 206.6484375 + }, + { + "workload": "barrier", + "phase": "wal1-workers1-async", + "version": "async-durable-barrier", + "samples": 2, + "ops_s_mean": 79283.9895, + "ops_s_min": 78489.093, + "ops_s_max": 80078.886, + "baseline_ops_s_mean": 62594.769, + "baseline_ops_s_min": 61606.519, + "baseline_ops_s_max": 63583.019, + "delta_pct": 26.66232461054372, + "ranges_overlap": false, + "launch_p99_us": 6.167, + "completion_p99_us": 4128.316, + "verified": true, + "verification_kind": "crash after completed final barrier, reopen and full key/value readback", + "cpu_us_update": 1.3590545654296875, + "read_bytes_update": 0.0, + "write_bytes_update": 206.6484375 + } + ] +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.csv new file mode 100644 index 000000000..d7c08d44a --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +disk-format,disk,0,0,load:random,1,0,2000000,2000000,2.698,741339.6,0.508,1.520,5.568,64126.874,1.299,2.164,924147.4,147542016,437133312,73.77,218.57,nan,nan,nan,nan,nan,0,0.0,2.698,0.087,2.785,0.018,12288,37859328,1.9153,2513178624,475205632,0.011,0,0,0,0,0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.log new file mode 100644 index 000000000..8deede8e8 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-disk-format-ordinary.log @@ -0,0 +1,7 @@ +kvbench version=disk-format splinter=splinterdb_build_version fb878f13-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-disk-format-ordinary.db mode=create n=2000000 cache_mib=64 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=1.0 warmup=1.0 rep=0 sabotage=0 +create took 0.011 s +phase load:random: T=1 dist=random (fixed count) +close took 0.087 s +== load:random T=1 ops=2000000 secs= 2.70 ops/s= 741340 lat us p50=0.51 p99=1.52 p99.9=5.57 max=64126.9 mean=1.30 + cpu=2.16s ops/cpu-s=924147 rd=73.8 B/op wr=218.6 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=2.70s drain(close)=0.09s total=2.78s drain_cpu=0.02s drain_rd=12288 drain_wr=37859328 wamp=1.915 db_apparent=2513178624 db_alloc=475205632 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.csv new file mode 100644 index 000000000..d85a97f5e --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,disk,0,0,load:random,1,0,2000000,2000000,2.631,760037.0,0.508,1.488,5.696,60421.601,1.266,2.151,929612.1,146448384,434933760,73.22,217.47,nan,nan,nan,nan,nan,0,0.0,2.631,0.111,2.743,0.014,28672,37990400,1.9070,2513178624,473116672,0.009,0,0,0,0,0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.log new file mode 100644 index 000000000..c8965f88c --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/disk-main-ordinary.log @@ -0,0 +1,7 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-main-ordinary.db mode=create n=2000000 cache_mib=64 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=1.0 warmup=1.0 rep=0 sabotage=0 +create took 0.009 s +phase load:random: T=1 dist=random (fixed count) +close took 0.111 s +== load:random T=1 ops=2000000 secs= 2.63 ops/s= 760037 lat us p50=0.51 p99=1.49 p99.9=5.70 max=60421.6 mean=1.27 + cpu=2.15s ops/cpu-s=929612 rd=73.2 B/op wr=217.5 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=2.63s drain(close)=0.11s total=2.74s drain_cpu=0.01s drain_rd=28672 drain_wr=37990400 wamp=1.907 db_apparent=2513178624 db_alloc=473116672 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/manifest.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/manifest.json new file mode 100644 index 000000000..52806385e --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/manifest.json @@ -0,0 +1,271 @@ +{ + "mode": "fixtures", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "scope": "Updated feature branches versus fresh main; not an isolation of merge cost", + "runner_sha256": "776d2ee23705f3a60a54f4e78b4f305a24d4a124eeecfe75dfacf5a90e4329c5", + "runs": [ + { + "label": "resident-main-ordinary", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-main-ordinary.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "1", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-main-ordinary.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures", + "load:random" + ], + "started": 1791905953.8715267, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791905954.186633, + "log_sha256": "e55092c3e8d86fa11a0c8308f2951ca7d933e1437c34285fc696792625d8eea3", + "process_cpu_s": 0.199626, + "process_read_bytes": 28672, + "process_write_bytes": 28803072, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-main-ordinary.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + { + "label": "resident-disk-format-ordinary", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-disk-format-ordinary.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "1", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-disk-format-ordinary.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures", + "load:random" + ], + "started": 1791905954.3706686, + "binary_sha256": "733c71174e9356864c326570c69279aacafc92f2dd7286813f9a21cdf9335ddc", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791905954.7357728, + "log_sha256": "80b8a6d6db1250cf8b95ab86ef8c7105f10b8560ed9e0203b3919dbff00d90c7", + "process_cpu_s": 0.21097700000000003, + "process_read_bytes": 32768, + "process_write_bytes": 28803072, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-disk-format-ordinary.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + { + "label": "disk-main-ordinary", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-main-ordinary.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "1", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-main-ordinary.csv", + "--vtag", + "main", + "--stag", + "disk", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures", + "load:random" + ], + "started": 1791905957.5532956, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791905960.3251648, + "log_sha256": "df659510c3e45ad4982e8a4d87cf79bfc17eee6e29d823cfb16a86fa83d91419", + "process_cpu_s": 2.188361, + "process_read_bytes": 146481152, + "process_write_bytes": 472977408, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-main-ordinary.db", + "fixture_sha256": "b074d0238b42f37eb1be4ea4994e5b4dc0252e2c4a840727193f3c9629ba142a", + "allocated_bytes": 473116672, + "apparent_bytes": 2513178624 + }, + { + "label": "disk-disk-format-ordinary", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-disk-format-ordinary.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "1", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-disk-format-ordinary.csv", + "--vtag", + "disk-format", + "--stag", + "disk", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures", + "load:random" + ], + "started": 1791905962.445601, + "binary_sha256": "733c71174e9356864c326570c69279aacafc92f2dd7286813f9a21cdf9335ddc", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791905965.2699206, + "log_sha256": "5e21dfcd258b47ef7d95ab24a76f428a25495b9aa00cca100f45cf9661d4a646", + "process_cpu_s": 2.2052849999999995, + "process_read_bytes": 147554304, + "process_write_bytes": 475045888, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/disk-disk-format-ordinary.db", + "fixture_sha256": "122ad4abc27ade97ea7cca2c1acb1ac7e4b876401a76f0f2c2f22d6bc8be235a", + "allocated_bytes": 475205632, + "apparent_bytes": 2513178624 + } + ], + "complete": true +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.csv new file mode 100644 index 000000000..d78ff4da0 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +disk-format,resident,0,0,load:random,1,0,200000,200000,0.187,1070181.9,0.520,2.976,5.952,3076.069,0.898,0.187,1071042.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.187,0.122,0.309,0.016,28672,28749824,1.1593,162664448,28905472,0.008,0,0,0,0,0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.log new file mode 100644 index 000000000..00cb73a2d --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-disk-format-ordinary.log @@ -0,0 +1,7 @@ +kvbench version=disk-format splinter=splinterdb_build_version fb878f13-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-disk-format-ordinary.db mode=create n=200000 cache_mib=128 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=1.0 warmup=1.0 rep=0 sabotage=0 +create took 0.008 s +phase load:random: T=1 dist=random (fixed count) +close took 0.122 s +== load:random T=1 ops=200000 secs= 0.19 ops/s= 1070182 lat us p50=0.52 p99=2.98 p99.9=5.95 max=3076.1 mean=0.90 + cpu=0.19s ops/cpu-s=1071042 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=0.19s drain(close)=0.12s total=0.31s drain_cpu=0.02s drain_rd=28672 drain_wr=28749824 wamp=1.159 db_apparent=162664448 db_alloc=28905472 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.csv new file mode 100644 index 000000000..f98a0ab0a --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,resident,0,0,load:random,1,0,200000,200000,0.172,1162108.8,0.492,2.400,4.928,2951.309,0.827,0.172,1162756.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.172,0.123,0.295,0.019,20480,28749824,1.1593,162664448,28905472,0.009,0,0,0,0,0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.log new file mode 100644 index 000000000..833416902 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/fixtures/resident-main-ordinary.log @@ -0,0 +1,7 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-main-ordinary.db mode=create n=200000 cache_mib=128 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=1.0 warmup=1.0 rep=0 sabotage=0 +create took 0.009 s +phase load:random: T=1 dist=random (fixed count) +close took 0.123 s +== load:random T=1 ops=200000 secs= 0.17 ops/s= 1162109 lat us p50=0.49 p99=2.40 p99.9=4.93 max=2951.3 mean=0.83 + cpu=0.17s ops/cpu-s=1162757 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=0.17s drain(close)=0.12s total=0.30s drain_cpu=0.02s drain_rd=20480 drain_wr=28749824 wamp=1.159 db_apparent=162664448 db_alloc=28905472 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/host.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/host.json new file mode 100644 index 000000000..989b69784 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/host.json @@ -0,0 +1,7 @@ +{ + "cpu": "{\n \"lscpu\": [\n {\n \"field\": \"Architecture:\",\n \"data\": \"x86_64\"\n },{\n \"field\": \"CPU op-mode(s):\",\n \"data\": \"32-bit, 64-bit\"\n },{\n \"field\": \"Address sizes:\",\n \"data\": \"46 bits physical, 48 bits virtual\"\n },{\n \"field\": \"Byte Order:\",\n \"data\": \"Little Endian\"\n },{\n \"field\": \"CPU(s):\",\n \"data\": \"20\"\n },{\n \"field\": \"On-line CPU(s) list:\",\n \"data\": \"0-19\"\n },{\n \"field\": \"Vendor ID:\",\n \"data\": \"GenuineIntel\"\n },{\n \"field\": \"Model name:\",\n \"data\": \"12th Gen Intel(R) Core(TM) i7-12800H\"\n },{\n \"field\": \"CPU family:\",\n \"data\": \"6\"\n },{\n \"field\": \"Model:\",\n \"data\": \"154\"\n },{\n \"field\": \"Thread(s) per core:\",\n \"data\": \"2\"\n },{\n \"field\": \"Core(s) per socket:\",\n \"data\": \"14\"\n },{\n \"field\": \"Socket(s):\",\n \"data\": \"1\"\n },{\n \"field\": \"Stepping:\",\n \"data\": \"3\"\n },{\n \"field\": \"CPU(s) scaling MHz:\",\n \"data\": \"31%\"\n },{\n \"field\": \"CPU max MHz:\",\n \"data\": \"4800.0000\"\n },{\n \"field\": \"CPU min MHz:\",\n \"data\": \"400.0000\"\n },{\n \"field\": \"BogoMIPS:\",\n \"data\": \"5606.40\"\n },{\n \"field\": \"Flags:\",\n \"data\": \"fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities\"\n },{\n \"field\": \"Virtualization:\",\n \"data\": \"VT-x\"\n },{\n \"field\": \"L1d cache:\",\n \"data\": \"544 KiB (14 instances)\"\n },{\n \"field\": \"L1i cache:\",\n \"data\": \"704 KiB (14 instances)\"\n },{\n \"field\": \"L2 cache:\",\n \"data\": \"11.5 MiB (8 instances)\"\n },{\n \"field\": \"L3 cache:\",\n \"data\": \"24 MiB (1 instance)\"\n },{\n \"field\": \"NUMA node(s):\",\n \"data\": \"1\"\n },{\n \"field\": \"NUMA node0 CPU(s):\",\n \"data\": \"0-19\"\n },{\n \"field\": \"Vulnerability Gather data sampling:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Indirect target selection:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Itlb multihit:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability L1tf:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Mds:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Meltdown:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Mmio stale data:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Reg file data sampling:\",\n \"data\": \"Mitigation; Clear Register File\"\n },{\n \"field\": \"Vulnerability Retbleed:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Spec rstack overflow:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Spec store bypass:\",\n \"data\": \"Mitigation; Speculative Store Bypass disabled via prctl\"\n },{\n \"field\": \"Vulnerability Spectre v1:\",\n \"data\": \"Mitigation; usercopy/swapgs barriers and __user pointer sanitization\"\n },{\n \"field\": \"Vulnerability Spectre v2:\",\n \"data\": \"Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\"\n },{\n \"field\": \"Vulnerability Srbds:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Tsa:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Tsx async abort:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Vmscape:\",\n \"data\": \"Mitigation; IBPB before exit to userspace\"\n }\n ]\n}", + "kernel": "Linux snafu 6.12.107+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.107-1 (2026-08-29) x86_64 GNU/Linux", + "compiler": "cc (Debian 14.2.0-19) 14.2.0\nCopyright (C) 2024 Free Software Foundation, Inc.\nThis is free software; see the source for copying conditions. There is NO\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "space": "Filesystem 1B-blocks Used Available Use% Mounted on\n/dev/mapper/snafu-root_crypt 830432124928 706181464064 81991790592 90% /home/rob/projects/splinterdb" +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/manifest.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/manifest.json new file mode 100644 index 000000000..65eeb938a --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/manifest.json @@ -0,0 +1,426 @@ +{ + "mode": "insert", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "scope": "Updated feature branches versus fresh main; not an isolation of merge cost", + "runner_sha256": "536f10a1a7fd4b1e8e0250ea01cfe36793c7d6e2345f141092616ec06f64ee50", + "runs": [ + { + "label": "wal1-00-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/insert_benchmark", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/scratch.db", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-00-main.csv", + "--summary", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-00-main.summary.csv", + "--records", + "100000000", + "--seconds", + "5", + "--report-ms", + "100", + "--cache-mib", + "256", + "--disk-gib", + "16", + "--direct", + "--inserters", + "8", + "--readers", + "0", + "--memtable-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "8", + "--key-size", + "16", + "--value-size", + "16", + "--min-free-gib", + "12", + "--seed", + "42" + ], + "started": 1791907746.4460592, + "binary_sha256": "9c39506eca1e6dc3088f26df34ce0c3af5920baa767533552e4d723579947c6c", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791907751.6709034, + "log_sha256": "d2d2e8354ebcb144c19981057836ad942f500f144aac30bc60c6975892496363", + "process_cpu_s": 31.113331, + "process_read_bytes": 148025344, + "process_write_bytes": 1466130432, + "errors": 0, + "verification_kind": "API return, positive insert count, successful close; no readback", + "allocated_bytes": 1466564608 + }, + { + "label": "wal1-01-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/insert_benchmark", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/scratch.db", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-01-disk-format.csv", + "--summary", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-01-disk-format.summary.csv", + "--records", + "100000000", + "--seconds", + "5", + "--report-ms", + "100", + "--cache-mib", + "256", + "--disk-gib", + "16", + "--direct", + "--inserters", + "8", + "--readers", + "0", + "--memtable-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "8", + "--key-size", + "16", + "--value-size", + "16", + "--min-free-gib", + "12", + "--seed", + "42" + ], + "started": 1791907751.7014997, + "binary_sha256": "1212b5a19188db5d22643b7e875b8cfecd3c664d6616519892e1ce4fed0a1a83", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791907756.9260163, + "log_sha256": "55633a0fd575838aff77f7c9cca060d127da54a96ec9e6452898dc3aaf1bc007", + "process_cpu_s": 30.966324, + "process_read_bytes": 144490496, + "process_write_bytes": 1479114752, + "errors": 0, + "verification_kind": "API return, positive insert count, successful close; no readback", + "allocated_bytes": 1479471104 + }, + { + "label": "wal1-12-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/insert_benchmark", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/scratch.db", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-12-main.csv", + "--summary", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-12-main.summary.csv", + "--records", + "100000000", + "--seconds", + "5", + "--report-ms", + "100", + "--cache-mib", + "256", + "--disk-gib", + "16", + "--direct", + "--inserters", + "8", + "--readers", + "0", + "--memtable-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "8", + "--key-size", + "16", + "--value-size", + "16", + "--min-free-gib", + "12", + "--seed", + "42" + ], + "started": 1791907810.6097565, + "binary_sha256": "9c39506eca1e6dc3088f26df34ce0c3af5920baa767533552e4d723579947c6c", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791907815.8845568, + "log_sha256": "dfdc27d0b401b30c49872cd01233f3282c693df04abe9a7e47d84ad9c6bf553d", + "process_cpu_s": 30.143582999999985, + "process_read_bytes": 143224832, + "process_write_bytes": 1447534592, + "errors": 0, + "verification_kind": "API return, positive insert count, successful close; no readback", + "allocated_bytes": 1447948288 + }, + { + "label": "wal1-23-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/insert_benchmark", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/scratch.db", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-23-disk-format.csv", + "--summary", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-23-disk-format.summary.csv", + "--records", + "100000000", + "--seconds", + "5", + "--report-ms", + "100", + "--cache-mib", + "256", + "--disk-gib", + "16", + "--direct", + "--inserters", + "8", + "--readers", + "0", + "--memtable-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "8", + "--key-size", + "16", + "--value-size", + "16", + "--min-free-gib", + "12", + "--seed", + "42" + ], + "started": 1791907869.5002658, + "binary_sha256": "1212b5a19188db5d22643b7e875b8cfecd3c664d6616519892e1ce4fed0a1a83", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791907874.9249175, + "log_sha256": "d91b330e4a66204ecdd0794c0f82f7f92441f91f8746df2a59ee9e054f2a5b39", + "process_cpu_s": 29.000805000000042, + "process_read_bytes": 144715776, + "process_write_bytes": 1417142272, + "errors": 0, + "verification_kind": "API return, positive insert count, successful close; no readback", + "allocated_bytes": 1417527296 + }, + { + "label": "wal1-24-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/insert_benchmark", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/scratch.db", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-24-main.csv", + "--summary", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal1-24-main.summary.csv", + "--records", + "100000000", + "--seconds", + "5", + "--report-ms", + "100", + "--cache-mib", + "256", + "--disk-gib", + "16", + "--direct", + "--inserters", + "8", + "--readers", + "0", + "--memtable-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "8", + "--key-size", + "16", + "--value-size", + "16", + "--min-free-gib", + "12", + "--seed", + "42" + ], + "started": 1791907874.9669132, + "binary_sha256": "9c39506eca1e6dc3088f26df34ce0c3af5920baa767533552e4d723579947c6c", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791907880.2412379, + "log_sha256": "0e3644505976ec7c8a8e4877dab5b0b9493a5f78984892223bd3e7f074c4557b", + "process_cpu_s": 30.366426000000104, + "process_read_bytes": 145027072, + "process_write_bytes": 1465266176, + "errors": 0, + "verification_kind": "API return, positive insert count, successful close; no readback", + "allocated_bytes": 1465667584 + }, + { + "label": "wal0-00-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/insert_benchmark", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/scratch.db", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal0-00-main.csv", + "--summary", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal0-00-main.summary.csv", + "--records", + "100000000", + "--seconds", + "5", + "--report-ms", + "100", + "--cache-mib", + "256", + "--disk-gib", + "16", + "--direct", + "--inserters", + "8", + "--readers", + "0", + "--memtable-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "8", + "--key-size", + "16", + "--value-size", + "16", + "--min-free-gib", + "12", + "--seed", + "42", + "--no-log" + ], + "started": 1791907880.2742877, + "binary_sha256": "9c39506eca1e6dc3088f26df34ce0c3af5920baa767533552e4d723579947c6c", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791907886.9553814, + "log_sha256": "84c9d5144c6b034941a876f264e74f49cdb3449c4b6b75df09d3f774ae93270a", + "process_cpu_s": 46.78557299999993, + "process_read_bytes": 785420288, + "process_write_bytes": 1788174336, + "errors": 0, + "verification_kind": "API return, positive insert count, successful close; no readback", + "allocated_bytes": 1788436480 + }, + { + "label": "wal0-05-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/insert_benchmark", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/scratch.db", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal0-05-main.csv", + "--summary", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/insert/wal0-05-main.summary.csv", + "--records", + "100000000", + "--seconds", + "5", + "--report-ms", + "100", + "--cache-mib", + "256", + "--disk-gib", + "16", + "--direct", + "--inserters", + "8", + "--readers", + "0", + "--memtable-bg", + "1", + "--normal-bg", + "3", + "--memtable-mib", + "8", + "--key-size", + "16", + "--value-size", + "16", + "--min-free-gib", + "12", + "--seed", + "42", + "--no-log" + ], + "started": 1791907912.0325544, + "binary_sha256": "9c39506eca1e6dc3088f26df34ce0c3af5920baa767533552e4d723579947c6c", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791907918.6131651, + "log_sha256": "e561f2425c9264f1abdedba855ac90a73514a661943399e896c390652cd8749c", + "process_cpu_s": 46.674325000000024, + "process_read_bytes": 783392768, + "process_write_bytes": 1782444032, + "errors": 0, + "verification_kind": "API return, positive insert count, successful close; no readback", + "allocated_bytes": 1782677504 + } + ], + "complete": true +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/results.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/results.csv new file mode 100644 index 000000000..7bb436631 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/results.csv @@ -0,0 +1,8 @@ +wal,version,index,errors,violations,verification_kind,ops_s,inserted,close_s,drain_ops_s,cpu_us_op,p99_us,max_ms,read_B_op,write_B_op +1,main,0,0,0,"API return, positive insert count, successful close; no readback",2292714.900132,11482220,0.141,2229933.122708837,2.700349912386281,61.952,146.300317,12.888511223826617,117.8628768179576 +1,disk-format,1,0,0,"API return, positive insert count, successful close; no readback",2318818.891143,11616290,0.126,2261927.201098534,2.65399804162947,61.952,119.79783,12.435411719554146,118.30987688845424 +1,main,12,0,0,"API return, positive insert count, successful close; no readback",2261423.232769,11326742,0.218,2167101.1363047166,2.647612962756634,64.0,106.089198,12.641600761251093,118.37450773484346 +1,disk-format,23,0,0,"API return, positive insert count, successful close; no readback",2207885.214204,11105082,0.336,2069628.4592405478,2.5526451593063424,65.024,158.302842,10.877803717212368,118.31872953633729 +1,main,24,0,0,"API return, positive insert count, successful close; no readback",2296488.128709,11499824,0.179,2217231.0409384235,2.6297198407558238,64.0,117.3139,12.60767769905495,117.53355121080892 +0,main,0,0,0,"API return, positive insert count, successful close; no readback",2735106.653056,17209225,0.316,2604311.0628731097,2.5080690860279877,4.288,1874.51952,18.882187172383276,70.38843200252148 +0,main,5,0,0,"API return, positive insert count, successful close; no readback",2783453.909562,17244009,0.297,2656118.341685941,2.502063120240775,4.288,1776.242964,19.846489632301147,70.79921517504536 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.csv new file mode 100644 index 000000000..0c2a767fd --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.csv @@ -0,0 +1,51 @@ +elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s,hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib,read_mib_s,write_mib_s,image_gib,free_gib,inserters_running,barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s,offered_insert_ops_s,insert_backlog,response_max_ms,i0_ops_s,i1_ops_s,i2_ops_s,i3_ops_s,i4_ops_s,i5_ops_s,i6_ops_s,i7_ops_s +0.100073,0.100073,515580,0.015365,5152016.256,0.000,0.000,17.818,0.000,0.000,8.380,91.910,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,17.818,638620.984,671566.788,748570.235,687275.251,650482.273,503300.366,639430.390,612769.969 +0.200273,0.100199,985174,0.029360,4686610.926,0.000,0.000,18.319,0.000,0.001,9.000,163.051,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,18.319,487230.129,718130.504,662381.477,588348.845,603049.582,572989.419,590754.057,463726.914 +0.300456,0.100183,1430537,0.042633,4445491.683,0.000,0.000,19.249,0.000,12.076,9.310,234.926,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,19.249,574108.984,700258.045,518530.731,596198.545,584889.249,554544.800,464868.969,452092.359 +0.400633,0.100177,1870357,0.055741,4390418.510,0.000,0.000,20.382,0.000,0.001,9.572,290.762,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,20.382,567034.999,693381.065,531977.134,546571.268,577276.846,543776.222,383580.150,546820.826 +0.504015,0.103382,2301658,0.068595,4171919.399,0.000,0.000,23.182,0.000,15.914,9.978,299.074,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,23.182,594243.223,567352.639,463098.423,617032.510,496527.869,567642.825,465990.612,400031.298 +0.604206,0.100191,2469587,0.073599,1676080.540,0.000,0.000,102.999,0.000,57.981,10.629,307.668,0.000,79.418,0.423950,71.056,8,0.000,0.000,0.000,0,0.000,0,102.999,489283.091,84079.000,40961.564,252456.581,37308.559,432481.858,268086.652,71423.235 +0.704376,0.100170,2816342,0.083934,3461662.819,0.000,0.000,101.921,0.000,18.206,9.205,307.668,12.596,1024.469,0.592041,70.956,8,0.000,0.000,0.000,0,0.000,0,101.921,469970.730,594169.508,422641.223,294638.913,489856.910,535579.151,249276.061,405530.323 +0.804548,0.100172,3227737,0.096194,4106902.596,0.000,0.000,19.307,0.000,4.260,9.471,307.668,75.495,818.907,0.592041,70.876,8,0.000,0.000,0.000,0,0.000,0,19.307,435512.665,539703.874,565180.157,401421.165,520626.610,579415.728,511262.678,553779.720 +0.904715,0.100167,3656265,0.108965,4278132.310,0.000,0.000,13.840,0.000,0.001,9.564,307.668,86.808,188.123,0.593544,70.858,8,0.000,0.000,0.000,0,0.000,0,13.840,506823.225,506124.393,602732.984,456207.791,556310.544,569009.328,534117.623,546806.423 +1.004874,0.100159,4027565,0.120031,3707115.251,0.000,0.000,13.729,0.000,21.131,9.305,307.668,115.208,162.554,0.661865,70.842,8,0.000,0.000,0.000,0,0.000,0,13.729,340359.706,408132.123,643827.975,438244.322,448737.665,463803.749,424166.669,539843.042 +1.105043,0.100170,4250231,0.126667,2222885.474,0.000,0.000,78.799,0.000,0.001,10.125,307.668,95.268,15.793,0.776371,70.840,8,0.000,0.000,0.000,0,0.000,0,78.799,166127.910,232764.758,540631.991,102356.196,274374.105,99081.756,235879.470,571669.288 +1.205217,0.100174,4718486,0.140622,4674438.680,0.000,0.000,12.359,0.000,0.002,8.326,307.668,0.000,0.000,0.776371,70.840,8,0.000,0.000,0.000,0,0.000,0,12.359,544844.558,582648.958,606916.848,553000.406,705845.182,607186.380,546611.492,527384.855 +1.305382,0.100165,5163804,0.153893,4445837.255,0.000,0.000,11.999,0.000,11.019,8.462,307.668,0.000,34.942,1.018188,70.837,8,0.000,0.000,0.000,0,0.000,0,11.999,501152.297,511435.313,653560.579,562111.616,599749.454,511105.858,575549.423,531172.715 +1.405537,0.100155,5539134,0.165079,3747509.199,0.000,0.000,17.721,0.000,6.502,8.061,307.668,0.000,445.562,1.069336,70.793,8,0.000,0.000,0.000,0,0.000,0,17.721,488704.834,436195.973,525238.381,481765.557,489683.322,465151.230,461946.182,398823.722 +1.505943,0.100406,5809138,0.173126,2689116.834,0.000,0.000,19.596,0.000,33.624,8.112,307.668,0.000,357.610,1.151428,70.758,8,0.000,0.000,0.000,0,0.000,0,19.596,268798.145,315050.269,494780.208,393113.178,238043.071,337827.747,359290.565,282213.651 +1.606142,0.100199,5940116,0.177029,1307179.667,0.000,0.000,113.676,0.000,20.204,8.520,307.668,0.000,6.666,1.287846,70.757,8,0.000,0.000,0.000,0,0.000,0,113.676,224683.045,9.980,344674.350,472759.554,3792.456,9.980,228475.501,32774.802 +1.706318,0.100177,6295340,0.187616,3545980.104,0.000,0.000,35.677,0.000,0.414,7.897,307.668,0.000,0.039,1.287846,70.757,8,0.000,0.000,0.000,0,0.000,0,35.677,416255.164,396669.739,422064.908,467714.321,433265.135,429721.391,524434.190,455855.256 +1.806492,0.100174,6721350,0.200312,4252699.028,0.000,0.000,12.707,0.000,7.975,8.169,307.668,0.000,0.000,1.287846,70.757,8,0.000,0.000,0.000,0,0.000,0,12.707,515512.853,570786.660,614400.758,469353.185,478936.507,634026.604,487821.045,481861.417 +1.906684,0.100192,7137922,0.212727,4157733.991,0.000,0.000,13.141,0.000,12.989,8.483,307.668,0.000,142.578,1.622803,70.743,8,0.000,0.000,0.000,0,0.000,0,13.141,471155.024,464727.370,622214.875,462192.240,441312.345,607034.033,556570.961,532527.143 +2.006864,0.100179,7415170,0.220989,2767518.973,0.000,0.000,15.137,0.000,1.469,8.326,307.668,0.000,540.204,1.634155,70.690,8,0.000,0.000,0.000,0,0.000,0,15.137,211670.561,422332.930,401679.953,357229.634,323020.956,375506.870,331545.675,344532.395 +2.107030,0.100166,7697782,0.229412,2821427.881,0.000,0.000,16.958,0.000,70.073,9.801,307.668,0.000,231.529,1.833252,70.668,8,0.000,0.000,0.000,0,0.000,0,16.958,166443.200,604245.122,168040.544,164656.172,518308.038,164656.172,451728.762,583349.872 +2.207206,0.100176,7980357,0.237833,2820777.590,0.000,0.000,79.758,0.000,0.001,9.403,307.668,21.837,7.838,1.884647,70.667,8,0.000,0.000,0.000,0,0.000,0,79.758,523457.260,322521.466,557716.868,314355.860,371784.625,257416.232,320215.530,153309.749 +2.307496,0.100290,8438168,0.251477,4564870.597,0.000,0.000,15.960,0.000,0.001,8.416,307.668,0.000,0.000,1.884647,70.667,8,0.000,0.000,0.000,0,0.000,0,15.960,546375.358,681832.495,567952.778,660504.352,586100.145,571751.760,494685.274,455668.434 +2.407659,0.100162,8847879,0.263687,4090472.268,0.000,0.000,20.976,0.000,0.002,7.523,307.668,0.000,150.342,2.116089,70.652,8,0.000,0.000,0.000,0,0.000,0,20.976,459114.980,570693.918,535191.529,488796.815,519516.964,461850.541,564264.351,491043.170 +2.507826,0.100167,9235775,0.275248,3872494.135,0.000,0.000,22.429,0.000,0.001,8.374,307.668,0.000,639.518,2.433105,70.590,8,0.000,0.000,0.000,0,0.000,0,22.429,581059.810,426927.164,378647.776,477981.918,389160.223,616690.318,471372.953,530653.972 +2.608030,0.100204,9429595,0.281024,1934251.052,0.000,0.000,76.166,0.000,12.186,10.325,307.668,2.534,116.442,2.433105,70.578,8,0.000,0.000,0.000,0,0.000,0,76.166,170302.313,552551.916,145492.963,89357.568,79876.924,585913.804,158067.291,152688.273 +2.708217,0.100187,9813781,0.292473,3834673.477,0.000,0.000,43.958,0.000,0.001,8.990,307.668,0.000,10.410,2.433105,70.577,8,0.000,0.000,0.000,0,0.000,0,43.958,395738.351,386335.972,594076.647,450136.404,396666.611,500492.033,527411.583,583815.877 +2.808388,0.100171,10275380,0.306230,4608116.250,0.000,0.000,12.466,0.000,1.031,8.371,307.668,0.000,0.000,2.433105,70.577,8,0.000,0.000,0.000,0,0.000,0,12.466,566042.820,533089.127,558725.323,630602.508,558365.937,572951.016,633507.545,554831.975 +2.908550,0.100162,10691099,0.318620,4150475.444,0.000,0.000,13.241,0.000,4.749,8.674,307.668,0.000,554.455,2.704956,70.523,8,0.000,0.000,0.000,0,0.000,0,13.241,622333.202,480233.089,484336.450,534814.787,505172.742,586201.655,468811.566,468571.954 +3.008709,0.100159,11102448,0.330879,4106965.469,0.000,0.000,15.523,0.000,11.488,8.562,307.668,0.000,383.648,2.818970,70.486,8,0.000,0.000,0.000,0,0.000,0,15.523,468945.011,514273.000,449625.702,599487.624,494963.676,470652.298,572370.703,536647.455 +3.108871,0.100162,11280849,0.336195,1781125.627,0.000,0.000,77.292,0.000,36.276,10.888,307.668,91.336,21.801,3.202148,70.484,8,0.000,0.000,0.000,0,0.000,0,77.292,187646.124,15285.247,43479.589,381961.448,187546.286,555989.625,382840.025,26377.284 +3.209068,0.100197,11692418,0.348461,4107586.922,0.000,0.000,58.175,0.000,0.002,9.140,307.668,0.000,0.000,3.202148,70.484,8,0.000,0.000,0.000,0,0.000,0,58.175,583977.981,428804.094,417696.007,465152.389,621214.524,656604.709,493137.183,441000.035 +3.309252,0.100185,12167543,0.362621,4742496.016,0.000,0.000,12.405,0.000,0.001,8.516,307.668,0.000,5.342,3.202148,70.483,8,0.000,0.000,0.000,0,0.000,0,12.405,537657.559,666978.850,486352.262,579250.784,582255.238,562970.835,663255.723,663774.765 +3.409424,0.100171,12574713,0.374756,4064738.242,0.000,0.000,16.033,0.000,0.001,8.408,307.668,0.000,690.068,3.478092,70.416,8,0.000,0.000,0.000,0,0.000,0,16.033,499414.643,551355.682,476523.848,506083.221,540763.823,522355.352,441453.913,526787.760 +3.509596,0.100172,12948733,0.385902,3733782.710,0.000,0.000,11.428,0.000,41.907,9.512,307.668,0.000,254.757,3.478092,70.391,8,0.000,0.000,0.000,0,0.000,0,11.428,565807.541,405652.800,574871.962,280018.729,587659.983,511251.307,278780.857,529739.531 +3.609763,0.100168,13187859,0.393029,2387253.949,0.000,0.000,73.311,0.000,3.756,10.865,307.668,61.264,54.908,3.716675,70.385,8,0.000,0.000,0.000,0,0.000,0,73.311,165871.651,208210.602,110274.948,452001.496,265234.909,172111.180,441229.573,572319.590 +3.709940,0.100176,13625897,0.406083,4372678.402,0.000,0.000,12.842,0.000,0.142,9.291,307.668,0.000,1.053,3.716675,70.385,8,0.000,0.000,0.000,0,0.000,0,12.842,625677.993,554932.597,520822.675,506348.169,553395.305,620866.468,453810.703,536824.491 +3.810098,0.100159,14063649,0.419129,4370572.538,0.000,0.000,16.226,0.000,0.001,8.716,307.668,0.000,436.728,4.025757,70.342,8,0.000,0.000,0.000,0,0.000,0,16.226,597859.642,564253.063,593516.546,525484.690,486776.221,600455.516,452031.451,550195.409 +3.910269,0.100170,14497825,0.432069,4334381.669,0.000,0.000,12.645,0.000,0.003,8.699,307.668,0.039,417.804,4.025757,70.302,8,0.000,0.000,0.000,0,0.000,0,12.645,579733.132,608773.697,559028.377,535538.364,541388.405,515682.165,529608.458,464629.071 +4.016017,0.105749,14801436,0.441117,2871064.856,0.000,0.000,76.673,0.000,20.793,10.428,307.668,156.141,191.529,4.025757,70.282,8,0.000,0.000,0.000,0,0.000,0,76.673,524158.393,163803.635,553463.754,440336.928,345602.785,557738.044,125136.445,160824.871 +4.116206,0.100188,15073162,0.449215,2712148.143,0.000,0.000,77.477,0.000,0.003,10.048,307.668,65.502,10.410,4.127808,70.281,8,0.000,0.000,0.000,0,0.000,0,77.477,246924.597,221382.738,659966.096,264890.734,247353.788,246914.615,223858.072,600857.504 +4.216374,0.100169,15561945,0.463782,4879605.912,0.000,0.000,10.166,0.000,6.834,8.639,307.668,0.000,15.560,4.470825,70.279,8,0.000,0.000,0.000,0,0.000,0,10.166,538642.172,706319.569,661674.813,574691.415,704941.891,580182.161,550112.840,563041.051 +4.316539,0.100165,16010302,0.477144,4476182.777,0.000,0.000,14.998,0.000,3.838,8.700,307.668,0.039,625.998,4.470825,70.218,8,0.000,0.000,0.000,0,0.000,0,14.998,508171.345,635381.405,478560.213,461787.893,675455.269,625337.980,545270.119,546218.554 +4.416697,0.100158,16465289,0.490704,4542693.952,0.000,0.000,14.005,0.000,32.506,8.336,307.668,11.388,306.937,4.470825,70.188,8,0.000,0.000,0.000,0,0.000,0,14.005,593851.898,430929.265,605273.855,452944.488,673206.542,639629.584,589189.264,557669.056 +4.516869,0.100171,16645690,0.496080,1800923.254,0.000,0.000,35.701,0.000,99.541,4.907,366.672,315.046,201.530,4.578491,70.168,8,0.000,0.000,0.000,0,0.000,0,35.701,4252.711,346276.489,289633.574,565261.154,4851.684,4582.146,582212.100,3853.395 +4.617018,0.100149,16771088,0.499817,1252114.212,0.000,0.000,17.214,0.000,199.690,6.479,366.672,473.240,159.879,4.685425,70.153,8,0.000,0.000,0.000,0,0.000,0,17.214,0.000,324825.974,0.000,461781.894,0.000,0.000,465506.344,0.000 +4.717164,0.100146,16898712,0.503621,1274380.119,0.000,0.000,9.688,0.000,299.836,5.404,366.672,382.489,783.661,5.028687,70.076,8,0.000,0.000,0.000,0,0.000,0,9.688,0.000,392656.941,0.000,406676.480,0.000,0.000,475046.698,0.000 +4.817305,0.100141,17004111,0.506762,1052504.201,0.000,0.000,49.807,0.000,399.977,5.549,366.672,382.039,553.359,5.028687,70.022,8,0.000,0.000,0.000,0,0.000,0,49.807,0.000,433258.378,0.000,487631.620,0.000,0.000,131614.203,0.000 +4.917478,0.100173,17137624,0.510741,1332821.047,0.000,0.000,44.936,0.000,500.150,7.070,366.672,549.088,385.270,5.126831,69.984,8,0.000,0.000,0.000,0,0.000,0,44.936,0.000,497927.401,0.000,521097.262,0.000,0.000,313796.385,0.000 +5.017650,0.100172,17208981,0.512868,712343.067,0.000,0.000,26.629,0.000,600.323,3.024,366.672,187.919,493.915,5.143188,69.936,8,0.000,0.000,0.000,0,0.000,0,26.629,0.000,224213.816,0.000,216377.314,0.000,0.000,271751.937,0.000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.log new file mode 100644 index 000000000..3b7e4532c --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.log @@ -0,0 +1,14 @@ +version=splinterdb_build_version 7bb5fab3 records=100000000 key=16 value=16 cache_mib=256 disk_gib=16 inserters=8 readers=0 memtable_bg=1 normal_bg=3 memtable_mib=8 queue_scale=100 log=0 checkpoints=disabled direct=1 query_range=inserted seed=42 report_ms=100 barrier_ms=0 target_insert_ops_s=0 id_base=0 mode=create +Splinter task system created 1 background thread of type 'TASK_TYPE_MEMTABLE'. +Splinter task system created 3 background threads of type 'TASK_TYPE_NORMAL'. +worker=4 kind=insert tid=5 os_tid=367 +worker=5 kind=insert tid=6 os_tid=368 +worker=7 kind=insert tid=7 os_tid=370 +worker=6 kind=insert tid=8 os_tid=369 +worker=0 kind=insert tid=9 os_tid=363 +start_monotonic_ns=1819390647997410 main_address=0x55c006f5ca80 +worker=3 kind=insert tid=10 os_tid=366 +worker=2 kind=insert tid=11 os_tid=365 +worker=1 kind=insert tid=12 os_tid=364 +load_end reason=time_limit inserted=17209225 elapsed_s=6.292 error=0 barrier_count=0 barrier_avg_ms=0.000 barrier_max_ms=0.000 +close_s=0.316 close_rc=0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.summary.csv new file mode 100644 index 000000000..04bc43c52 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-00-main.summary.csv @@ -0,0 +1,2 @@ +target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted,insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms,insert_service_p999_ms,insert_service_max_ms,insert_response_mean_ms,insert_response_p50_ms,insert_response_p99_ms,insert_response_p999_ms,insert_response_max_ms,query_service_mean_ms,query_service_p99_ms,query_service_max_ms +0,2735106.653056,6.291976,17209225,0.002639,0.001424,0.004288,0.010880,1874.519520,0.002639,0.001424,0.004288,0.010880,1874.519520,0.000000,0.000000,0.000000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.csv new file mode 100644 index 000000000..4ba3d8e14 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.csv @@ -0,0 +1,51 @@ +elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s,hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib,read_mib_s,write_mib_s,image_gib,free_gib,inserters_running,barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s,offered_insert_ops_s,insert_backlog,response_max_ms,i0_ops_s,i1_ops_s,i2_ops_s,i3_ops_s,i4_ops_s,i5_ops_s,i6_ops_s,i7_ops_s +0.100075,0.100075,514236,0.015325,5138521.268,0.000,0.000,18.323,0.000,0.002,8.002,90.289,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,18.323,536599.134,628080.792,634316.134,679082.691,777489.177,533381.537,635675.119,713896.683 +0.200249,0.100174,967483,0.028833,4524599.821,0.000,0.000,18.654,0.000,0.002,9.086,162.695,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,18.654,595424.307,565486.382,456565.840,592549.308,523279.797,597021.529,567782.388,626490.270 +0.300411,0.100162,1409205,0.041998,4410065.727,0.000,0.000,18.584,0.000,7.902,9.609,231.980,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,18.584,475808.116,565133.207,460023.722,547701.486,620153.949,582365.252,558923.281,599956.714 +0.400590,0.100179,1878746,0.055991,4687011.064,0.000,0.000,19.041,0.000,0.001,9.516,290.457,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,19.041,601032.973,490890.344,679741.930,534182.766,530469.420,602360.594,562052.824,686280.213 +0.500762,0.100171,2271688,0.067702,3922694.266,0.000,0.000,17.291,0.000,20.373,10.034,300.020,0.000,0.000,0.005013,71.064,8,0.000,0.000,0.000,0,0.000,0,17.291,390171.024,496718.342,636937.925,476483.036,460690.115,468097.414,450597.419,542998.990 +0.600941,0.100179,2422733,0.072203,1507744.473,0.000,0.000,99.402,0.000,0.308,10.801,309.238,0.000,49.910,0.214844,71.059,8,0.000,0.000,0.000,0,0.000,0,99.402,58135.680,542995.638,537445.597,66959.846,41066.310,75793.994,58804.480,126542.929 +0.701106,0.100165,2724929,0.081209,3016991.347,0.000,0.000,40.570,0.000,0.050,7.991,309.238,0.741,720.493,0.415283,70.989,8,0.000,0.000,0.000,0,0.000,0,40.570,394150.877,410124.570,255349.468,388040.939,392483.623,373105.536,371568.068,432168.266 +0.801265,0.100159,3049748,0.090890,3243035.001,0.000,0.000,27.668,0.000,0.306,8.329,309.238,49.297,900.990,0.495728,70.901,8,0.000,0.000,0.000,0,0.000,0,27.668,425593.494,448357.309,421509.985,405794.965,447269.039,423436.922,333100.517,337972.772 +0.901606,0.100341,3443717,0.102631,3926303.778,0.000,0.000,15.985,0.000,0.001,9.182,309.238,90.356,353.716,0.603157,70.866,8,0.000,0.000,0.000,0,0.000,0,15.985,502915.375,490437.916,511795.101,556672.099,566757.713,389033.640,509722.169,398969.764 +1.001776,0.100171,3872628,0.115413,4281801.735,0.000,0.000,13.090,0.000,0.001,9.628,309.238,115.116,115.389,0.783936,70.855,8,0.000,0.000,0.000,0,0.000,0,13.090,588016.362,529975.427,589773.363,552277.362,665943.355,412625.722,536843.704,406346.440 +1.101949,0.100173,4111280,0.122526,2382395.478,0.000,0.000,19.483,0.000,66.393,9.166,309.238,88.792,70.815,0.783936,70.848,8,0.000,0.000,0.000,0,0.000,0,19.483,238347.361,192836.152,194473.318,204955.171,664918.859,493575.497,199903.916,193385.202 +1.202124,0.100175,4476517,0.133411,3646004.259,0.000,0.000,81.864,0.000,0.001,9.403,309.238,42.426,3.276,0.783936,70.847,8,0.000,0.000,0.000,0,0.000,0,81.864,571302.534,304248.797,427323.914,457022.062,450193.984,534476.830,495464.943,405971.195 +1.302289,0.100165,4941177,0.147259,4638933.050,0.000,0.000,9.750,0.000,2.726,8.446,309.238,0.000,0.000,0.783936,70.847,8,0.000,0.000,0.000,0,0.000,0,9.750,624318.164,562031.109,534776.154,677420.400,597452.566,514000.491,556819.722,572114.444 +1.402439,0.100150,5345450,0.159307,4036676.922,0.000,0.000,12.270,0.000,7.002,8.353,309.238,0.000,234.960,1.044312,70.824,8,0.000,0.000,0.000,0,0.000,0,12.270,530774.094,445881.392,550903.909,467818.497,575906.417,458412.601,488817.009,518163.004 +1.502608,0.100169,5703007,0.169963,3569526.078,0.000,0.000,15.805,0.000,0.002,7.968,309.238,0.000,490.966,1.106567,70.776,8,0.000,0.000,0.000,0,0.000,0,15.805,424531.184,419070.430,440124.781,506372.610,460180.822,490888.827,388751.766,439605.660 +1.604455,0.101846,5949850,0.177319,2423685.184,0.000,0.000,78.928,0.000,12.812,10.421,309.238,3.183,139.955,1.245071,70.762,8,0.000,0.000,0.000,0,0.000,0,78.928,508247.008,95202.422,100700.912,124580.068,461234.921,540177.523,495620.119,97922.211 +1.704642,0.100188,6331981,0.188708,3814158.186,0.000,0.000,73.198,0.000,5.540,8.754,309.238,10.956,4.055,1.298183,70.762,8,0.000,0.000,0.000,0,0.000,0,73.198,225726.747,404641.269,326487.812,569801.582,491787.863,659034.265,689427.276,447251.372 +1.804814,0.100171,6781066,0.202092,4483167.689,0.000,0.000,12.114,0.000,0.001,8.683,309.238,0.000,0.000,1.298183,70.762,8,0.000,0.000,0.000,0,0.000,0,12.114,461998.324,535532.317,551634.724,682889.807,533176.354,548430.215,644086.300,525419.646 +1.904965,0.100151,7181387,0.214022,3997166.324,0.000,0.000,17.743,0.000,6.132,7.990,309.238,0.000,337.576,1.595947,70.729,8,0.000,0.000,0.000,0,0.000,0,17.743,387344.339,487313.187,463698.892,601730.190,498196.732,506813.703,591855.121,460214.161 +2.005115,0.100150,7542236,0.224776,3603078.644,0.000,0.000,19.104,0.000,14.547,8.043,309.238,0.000,444.801,1.676514,70.685,8,0.000,0.000,0.000,0,0.000,0,19.104,479649.629,469534.820,361796.628,482095.955,433708.626,474137.907,403384.169,498770.911 +2.105271,0.100156,7713691,0.229886,1711875.862,0.000,0.000,77.885,0.000,19.932,10.742,309.238,0.000,121.568,1.883957,70.674,8,0.000,0.000,0.000,0,0.000,0,77.885,486869.456,86255.260,442199.236,92315.793,103698.012,21466.467,28904.847,450166.790 +2.205454,0.100183,8140906,0.242618,4264338.797,0.000,0.000,36.519,0.000,2.417,8.739,309.238,0.000,1.053,1.884159,70.674,8,0.000,0.000,0.000,0,0.000,0,36.519,390624.474,427227.426,414131.415,675023.526,587992.944,549493.465,544822.022,675023.526 +2.305634,0.100180,8624600,0.257033,4828271.370,0.000,0.000,11.704,0.000,0.001,8.528,309.238,0.000,2.651,2.097534,70.673,8,0.000,0.000,0.000,0,0.000,0,11.704,546139.467,517550.794,410043.811,720486.446,619397.939,705603.167,650611.898,658437.847 +2.405774,0.100140,9030139,0.269119,4049714.487,0.000,0.000,16.612,0.000,0.001,8.107,309.238,0.000,514.786,2.432739,70.623,8,0.000,0.000,0.000,0,0.000,0,16.612,471139.717,487007.478,418553.414,532483.745,541680.856,447622.675,561023.748,590202.854 +2.505939,0.100165,9398483,0.280097,3677388.783,0.000,0.000,16.843,0.000,34.097,9.156,309.238,0.000,380.390,2.432739,70.586,8,0.000,0.000,0.000,0,0.000,0,16.843,519355.390,246893.731,419749.294,444668.289,340329.980,429034.016,657997.252,619360.829 +2.606113,0.100175,9640233,0.287301,2413288.305,0.000,0.000,73.943,0.000,0.003,10.555,309.238,32.521,39.267,2.432739,70.582,8,0.000,0.000,0.000,0,0.000,0,73.943,485043.497,121398.135,545118.653,127617.281,139526.497,128166.323,301403.987,565013.932 +2.706270,0.100157,10097772,0.300937,4568232.539,0.000,0.000,9.610,0.000,4.062,8.807,309.238,0.000,3.783,2.679077,70.582,8,0.000,0.000,0.000,0,0.000,0,9.610,553812.292,537737.478,550677.204,680653.559,566702.097,564425.664,606070.415,508153.830 +2.806424,0.100154,10572333,0.315080,4738321.277,0.000,0.000,10.941,0.000,0.001,8.502,309.238,0.000,282.651,2.679077,70.554,8,0.000,0.000,0.000,0,0.000,0,10.941,541756.643,582713.641,558191.362,606077.701,583113.026,631678.321,636201.363,598589.220 +2.906594,0.100171,10959365,0.326615,3863714.941,0.000,0.000,21.894,0.000,0.002,8.157,309.238,0.000,513.770,2.763184,70.504,8,0.000,0.000,0.000,0,0.000,0,21.894,498597.642,467640.564,485090.733,599734.747,434447.308,483982.627,345239.809,548981.511 +3.006757,0.100162,11161422,0.332636,2017296.392,0.000,0.000,74.845,0.000,3.904,8.829,309.238,27.494,150.576,2.869511,70.489,8,0.000,0.000,0.000,0,0.000,0,74.845,261106.285,172919.392,154629.073,175984.418,173109.084,256893.122,155887.031,666767.989 +3.106918,0.100161,11502407,0.342798,3404368.082,0.000,0.000,54.567,0.000,0.001,9.532,309.238,21.177,1.911,2.963745,70.489,8,0.000,0.000,0.000,0,0.000,0,54.567,415800.453,538203.353,309641.397,343906.222,580235.670,620211.299,310290.352,286079.338 +3.207084,0.100166,11993844,0.357444,4906227.380,0.000,0.000,11.684,0.000,0.281,8.312,309.238,0.000,0.000,2.963745,70.489,8,0.000,0.000,0.000,0,0.000,0,11.684,601551.635,573478.227,549767.578,726644.025,603268.785,664127.780,521544.419,665844.930 +3.307252,0.100168,12445665,0.370910,4510625.699,0.000,0.000,14.485,0.000,13.381,8.471,309.238,0.000,539.523,3.253662,70.436,8,0.000,0.000,0.000,0,0.000,0,14.485,604084.275,386160.698,559229.694,667747.230,445600.754,615674.786,596067.754,636060.509 +3.407418,0.100166,12850380,0.382971,4040459.161,0.000,0.000,15.908,0.000,6.254,8.652,309.238,0.000,373.990,3.324219,70.399,8,0.000,0.000,0.000,0,0.000,0,15.908,461875.153,511373.186,484717.327,546884.381,479446.057,545866.068,465638.921,544658.068 +3.507574,0.100156,13057854,0.389154,2071503.876,0.000,0.000,76.844,0.000,35.428,10.752,309.238,94.189,84.477,3.748657,70.391,8,0.000,0.000,0.000,0,0.000,0,76.844,9.984,9.984,9.984,468528.061,9.984,476315.895,554553.671,572066.312 +3.607740,0.100167,13447610,0.400770,3891077.387,0.000,0.000,61.776,0.000,0.001,9.859,309.238,22.853,0.078,3.748657,70.391,8,0.000,0.000,0.000,0,0.000,0,61.776,400542.688,336519.352,393554.331,436223.243,429973.655,647311.566,592113.527,654839.025 +3.707923,0.100182,13927686,0.415077,4792025.910,0.000,0.000,9.790,0.000,1.269,8.504,309.238,0.000,70.263,3.748657,70.384,8,0.000,0.000,0.000,0,0.000,0,9.790,571987.478,586710.644,569821.426,565559.195,595364.870,636559.787,561087.345,704935.164 +3.808104,0.100182,14245573,0.424551,3173103.519,0.000,0.000,28.093,0.000,5.148,7.610,309.238,0.000,579.416,3.838379,70.327,8,0.000,0.000,0.000,0,0.000,0,28.093,393444.992,423580.228,431725.425,240273.351,361702.678,398326.122,473040.344,451010.379 +3.908263,0.100158,14634781,0.436150,3885926.558,0.000,0.000,17.800,0.000,8.243,8.710,309.238,0.000,289.425,3.883099,70.299,8,0.000,0.000,0.000,0,0.000,0,17.800,533115.800,520905.136,467549.626,475916.377,479221.144,510222.053,530240.354,368756.067 +4.008420,0.100157,14824675,0.441810,1895954.895,0.000,0.000,76.768,0.000,37.119,10.823,309.238,145.825,81.746,4.031456,70.291,8,0.000,0.000,0.000,0,0.000,0,76.768,358625.359,297970.857,9.984,9.984,9.984,374240.773,280278.712,584809.241 +4.108594,0.100173,15192420,0.452769,3671082.288,0.000,0.000,71.012,0.000,2.700,9.408,309.238,30.728,5.303,4.085453,70.291,8,0.000,0.000,0.000,0,0.000,0,71.012,367312.874,674400.212,378972.649,393936.693,402451.923,430253.699,459922.237,563832.000 +4.208770,0.100176,15594812,0.464762,4016843.366,0.000,0.000,11.280,0.000,0.014,8.408,309.238,0.000,124.078,4.494995,70.278,8,0.000,0.000,0.000,0,0.000,0,11.280,436161.597,485215.177,478766.538,581255.978,517448.390,470750.660,592506.158,454738.868 +4.308931,0.100161,16019887,0.477430,4243920.132,0.000,0.000,16.362,0.000,0.002,8.668,309.238,0.000,571.736,4.494995,70.223,8,0.000,0.000,0.000,0,0.000,0,16.362,555376.216,589181.812,377103.117,635856.697,487814.945,505975.718,602170.908,490440.719 +4.409097,0.100166,16442257,0.490017,4216695.142,0.000,0.000,13.611,0.000,31.469,8.179,309.238,7.410,258.594,4.494995,70.197,8,0.000,0.000,0.000,0,0.000,0,13.611,394025.437,572139.550,418245.203,660532.710,505919.556,541879.818,573657.029,550295.838 +4.509250,0.100153,16630833,0.495637,1882882.090,0.000,0.000,41.665,0.000,90.368,6.241,365.613,425.521,193.025,4.578003,70.178,8,0.000,0.000,0.000,0,0.000,0,41.665,375336.313,544258.123,33009.546,45790.012,36524.174,296626.618,519206.414,32130.889 +4.609416,0.100166,16764792,0.499630,1337367.576,0.000,0.000,31.634,0.000,190.534,7.161,365.613,494.725,291.274,4.947021,70.150,8,0.000,0.000,0.000,0,0.000,0,31.634,508455.054,509024.109,0.000,0.000,0.000,0.000,319888.413,0.000 +4.709563,0.100147,16901707,0.503710,1367142.065,0.000,0.000,6.366,0.000,290.681,6.505,365.613,465.137,927.427,4.968750,70.059,8,0.000,0.000,0.000,0,0.000,0,6.366,470868.431,451556.794,0.000,0.000,0.000,0.000,444716.840,0.000 +4.809727,0.100164,17027850,0.507469,1259361.612,0.000,0.000,33.297,0.000,390.845,6.587,365.613,574.720,350.635,5.021118,70.025,8,0.000,0.000,0.000,0,0.000,0,33.297,516302.020,458167.501,0.000,0.000,0.000,0.000,284892.090,0.000 +4.909912,0.100185,17145903,0.510988,1178349.441,0.000,0.000,5.276,0.000,491.030,4.881,365.613,289.152,600.568,5.137573,69.966,8,0.000,0.000,0.000,0,0.000,0,5.276,265907.932,440614.634,0.000,0.000,0.000,0.000,471826.875,0.000 +5.010075,0.100163,17243822,0.513906,977600.939,0.000,0.000,79.603,0.000,591.193,3.766,365.613,225.961,400.326,5.157227,69.927,8,0.000,0.000,0.000,0,0.000,0,79.603,253028.709,355831.606,0.000,0.000,0.000,0.000,368740.623,0.000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.log new file mode 100644 index 000000000..b08808cc3 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.log @@ -0,0 +1,14 @@ +version=splinterdb_build_version 7bb5fab3 records=100000000 key=16 value=16 cache_mib=256 disk_gib=16 inserters=8 readers=0 memtable_bg=1 normal_bg=3 memtable_mib=8 queue_scale=100 log=0 checkpoints=disabled direct=1 query_range=inserted seed=42 report_ms=100 barrier_ms=0 target_insert_ops_s=0 id_base=0 mode=create +Splinter task system created 1 background thread of type 'TASK_TYPE_MEMTABLE'. +Splinter task system created 3 background threads of type 'TASK_TYPE_NORMAL'. +worker=4 kind=insert tid=5 os_tid=437 +worker=2 kind=insert tid=6 os_tid=435 +worker=7 kind=insert tid=7 os_tid=440 +worker=0 kind=insert tid=8 os_tid=433 +worker=6 kind=insert tid=10 os_tid=439 +worker=1 kind=insert tid=11 os_tid=434 +worker=5 kind=insert tid=9 os_tid=438 +start_monotonic_ns=1819422407558688 main_address=0x564ba1439a80 +worker=3 kind=insert tid=12 os_tid=436 +load_end reason=time_limit inserted=17244009 elapsed_s=6.195 error=0 barrier_count=0 barrier_avg_ms=0.000 barrier_max_ms=0.000 +close_s=0.297 close_rc=0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.summary.csv new file mode 100644 index 000000000..c8682a6bf --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal0-05-main.summary.csv @@ -0,0 +1,2 @@ +target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted,insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms,insert_service_p999_ms,insert_service_max_ms,insert_response_mean_ms,insert_response_p50_ms,insert_response_p99_ms,insert_response_p999_ms,insert_response_max_ms,query_service_mean_ms,query_service_p99_ms,query_service_max_ms +0,2783453.909562,6.195184,17244009,0.002604,0.001392,0.004288,0.011136,1776.242964,0.002604,0.001392,0.004288,0.011136,1776.242964,0.000000,0.000000,0.000000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.csv new file mode 100644 index 000000000..6f7caa0f1 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.csv @@ -0,0 +1,51 @@ +elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s,hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib,read_mib_s,write_mib_s,image_gib,free_gib,inserters_running,barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s,offered_insert_ops_s,insert_backlog,response_max_ms,i0_ops_s,i1_ops_s,i2_ops_s,i3_ops_s,i4_ops_s,i5_ops_s,i6_ops_s,i7_ops_s +0.100061,0.100061,278861,0.008311,2786911.628,0.000,0.000,16.732,0.000,0.153,5.525,75.812,0.000,146.981,0.058071,71.046,8,0.000,0.000,0.000,0,0.000,0,16.732,356242.902,317216.685,371253.754,369774.656,324662.148,323582.805,360320.417,363858.261 +0.200226,0.100165,544947,0.016241,2656473.869,0.000,0.000,13.519,0.000,0.150,5.797,129.250,0.000,140.315,0.115852,71.033,8,0.000,0.000,0.000,0,0.000,0,13.519,343013.647,344730.812,339838.889,335585.911,325372.774,298067.857,338810.586,331053.394 +0.300379,0.100153,814201,0.024265,2688438.277,0.000,0.000,12.787,0.000,12.324,5.804,186.406,0.000,141.815,0.177822,71.019,8,0.000,0.000,0.000,0,0.000,0,12.787,304605.267,346541.285,346531.301,352911.566,344005.155,304425.541,340750.121,348668.041 +0.400540,0.100161,1081003,0.032216,2663728.653,0.000,0.000,14.999,0.000,0.052,6.228,245.469,0.000,140.594,0.244431,71.005,8,0.000,0.000,0.000,0,0.000,0,14.999,338694.353,334261.495,336158.439,340491.458,336897.248,334740.723,349047.674,293437.264 +0.500694,0.100154,1336235,0.039823,2548384.530,0.000,0.000,16.604,0.000,0.122,6.058,284.926,0.000,134.519,0.301167,70.992,8,0.000,0.000,0.000,0,0.000,0,16.604,326066.456,322881.374,321703.194,325387.504,324359.092,328492.709,327723.896,271770.305 +0.600849,0.100155,1594354,0.047515,2577200.828,0.000,0.000,11.884,0.000,8.507,5.882,286.020,0.000,136.078,0.363194,70.978,8,0.000,0.000,0.000,0,0.000,0,11.884,327313.361,337956.887,286236.942,332385.510,328791.074,335670.426,302941.087,325905.541 +0.701010,0.100161,1837467,0.054761,2427220.428,0.000,0.000,11.615,0.000,0.042,5.704,292.043,0.000,255.682,0.423321,70.953,8,0.000,0.000,0.000,0,0.000,0,11.615,265971.594,299437.690,303121.756,315471.863,315941.107,313075.723,302902.110,311298.585 +0.801171,0.100161,2070242,0.061698,2324002.476,0.000,0.000,13.081,0.000,0.188,6.184,293.137,0.039,261.726,0.484028,70.928,8,0.000,0.000,0.000,0,0.000,0,13.081,293307.034,293526.679,293975.955,296721.528,254819.097,300874.830,294784.651,295992.703 +0.901329,0.100158,2275113,0.067804,2045482.821,0.000,0.000,31.560,0.000,30.012,7.100,300.355,53.704,215.753,0.559036,70.907,8,0.000,0.000,0.000,0,0.000,0,31.560,316221.095,215460.066,211436.415,322511.171,213642.933,227491.085,214701.264,324018.792 +1.001490,0.100161,2392654,0.071307,1173524.639,0.000,0.000,107.515,0.000,10.854,9.231,306.762,78.819,453.373,0.647739,70.862,8,0.000,0.000,0.000,0,0.000,0,107.515,290063.989,48162.623,19818.161,266611.667,27555.730,260641.259,46105.927,214565.284 +1.101654,0.100165,2567971,0.076531,1750286.298,0.000,0.000,1.373,0.000,111.018,6.410,306.762,134.076,709.027,0.730396,70.793,8,0.000,0.000,0.000,0,0.000,0,1.373,252214.462,247092.900,253791.863,0.000,238197.555,254440.794,251186.156,253362.570 +1.201807,0.100153,2776706,0.082752,2084162.732,0.000,0.000,146.300,0.000,12.656,6.444,306.762,3.432,469.399,0.813663,70.747,8,0.000,0.000,0.000,0,0.000,0,146.300,288319.079,277845.097,264126.077,190548.598,250267.271,279053.249,277345.861,256657.499 +1.301966,0.100158,3001089,0.089439,2240279.247,0.000,0.000,15.533,0.000,0.123,6.576,306.762,0.078,318.636,0.920559,70.716,8,0.000,0.000,0.000,0,0.000,0,15.533,281423.954,281264.207,306304.520,276401.914,290259.950,283550.584,285657.245,235416.874 +1.402118,0.100153,3242201,0.096625,2407446.886,0.000,0.000,13.689,0.000,0.163,5.347,306.762,0.000,217.442,0.988525,70.695,8,0.000,0.000,0.000,0,0.000,0,13.689,295279.483,289638.090,250787.366,321539.418,305893.289,308439.405,321848.945,314020.889 +1.502258,0.100139,3475008,0.103563,2324828.329,0.000,0.000,11.411,0.000,0.030,5.269,306.762,0.000,184.976,1.056438,70.676,8,0.000,0.000,0.000,0,0.000,0,11.411,296945.947,294109.902,283035.344,288028.382,291433.634,274087.820,299382.550,297804.750 +1.602411,0.100153,3716203,0.110751,2408273.361,0.000,0.000,14.363,0.000,0.091,5.663,306.762,0.000,225.164,1.125404,70.654,8,0.000,0.000,0.000,0,0.000,0,14.363,293631.721,292583.322,263078.366,324404.741,302548.109,317505.274,296806.874,317714.954 +1.702585,0.100174,3945162,0.117575,2285607.170,0.000,0.000,14.738,0.000,5.125,5.322,306.762,0.000,239.076,1.198109,70.631,8,0.000,0.000,0.000,0,0.000,0,14.738,253458.331,289226.003,288786.769,301115.286,276727.782,295325.375,287778.526,293189.097 +1.802945,0.100360,4101013,0.122220,1552918.546,0.000,0.000,82.748,0.000,11.506,8.217,306.762,93.997,172.776,1.290985,70.614,8,0.000,0.000,0.000,0,0.000,0,82.748,46373.029,295296.752,55928.623,38680.726,265553.844,291241.353,282213.858,277630.361 +1.903113,0.100168,4304167,0.128274,2028122.775,0.000,0.000,85.049,0.000,0.105,7.411,306.762,63.799,123.347,1.380692,70.602,8,0.000,0.000,0.000,0,0.000,0,85.049,283672.035,326370.093,331810.925,78986.913,330792.641,80424.491,301601.825,294463.852 +2.003276,0.100163,4565318,0.136057,2607265.971,0.000,0.000,12.068,0.000,0.099,5.674,306.762,0.000,145.115,1.488125,70.588,8,0.000,0.000,0.000,0,0.000,0,12.068,338009.798,327446.992,284437.002,305862.127,338978.221,344828.698,331720.036,335983.097 +2.103412,0.100136,4821963,0.143706,2562973.685,0.000,0.000,12.455,0.000,0.416,5.217,306.762,0.000,395.128,1.558376,70.549,8,0.000,0.000,0.000,0,0.000,0,12.455,337172.672,328294.714,317778.977,339938.920,306534.229,328114.958,320345.496,284793.717 +2.203560,0.100149,5076779,0.151300,2544379.764,0.000,0.000,15.982,0.000,0.111,5.413,306.762,0.000,449.137,1.627785,70.505,8,0.000,0.000,0.000,0,0.000,0,15.982,265984.821,324298.184,326584.786,326594.772,323169.860,325116.967,321062.990,331567.384 +2.303722,0.100161,5330614,0.158865,2534258.010,0.000,0.000,12.855,0.000,10.432,5.477,306.762,0.000,360.005,1.705448,70.470,8,0.000,0.000,0.000,0,0.000,0,12.855,312715.068,288124.774,323248.061,342876.368,284909.964,321430.995,335817.765,325135.014 +2.403876,0.100155,5576906,0.166205,2459117.688,0.000,0.000,15.420,0.000,0.141,6.017,306.762,0.000,257.297,1.807991,70.445,8,0.000,0.000,0.000,0,0.000,0,15.420,315492.183,293975.452,304019.921,324048.952,317409.219,302142.824,254077.143,347951.993 +2.504112,0.100236,5803235,0.172950,2257966.370,0.000,0.000,17.778,0.000,39.263,6.036,306.762,46.570,194.035,1.879467,70.426,8,0.000,0.000,0.000,0,0.000,0,17.778,200068.297,208927.410,340826.430,280847.842,201844.110,345754.810,334122.237,345575.234 +2.604278,0.100166,5967443,0.177844,1639365.882,0.000,0.000,81.837,0.000,0.154,8.593,306.762,127.679,137.702,1.986797,70.412,8,0.000,0.000,0.000,0,0.000,0,81.837,160284.634,74446.747,215233.661,228950.951,159575.808,220934.224,273347.449,306592.409 +2.704425,0.100148,6226795,0.185573,2589698.408,0.000,0.000,10.592,0.000,0.109,6.554,306.762,44.505,139.092,2.069885,70.399,8,0.000,0.000,0.000,0,0.000,0,10.592,323672.360,336403.573,328075.862,329473.799,319428.622,326048.853,294086.020,332509.319 +2.804561,0.100136,6485003,0.193268,2578575.716,0.000,0.000,12.319,0.000,0.057,5.225,306.762,0.000,276.811,2.139957,70.372,8,0.000,0.000,0.000,0,0.000,0,12.319,323600.227,320124.950,331070.076,293141.621,341196.314,327634.744,311506.662,330301.121 +2.904708,0.100146,6724427,0.200404,2390741.461,0.000,0.000,15.761,0.000,1.147,5.235,306.762,0.000,519.123,2.236160,70.321,8,0.000,0.000,0.000,0,0.000,0,15.761,300819.789,297145.167,309666.843,323226.999,271672.443,261726.997,322158.563,304324.660 +3.004857,0.100149,6977138,0.207935,2523346.303,0.000,0.000,13.697,0.000,0.153,5.639,306.762,0.000,349.518,2.316200,70.287,8,0.000,0.000,0.000,0,0.000,0,13.697,323637.279,268689.237,336458.156,331765.156,315020.132,309767.966,317925.798,320082.581 +3.105003,0.100146,7235895,0.215646,2583796.572,0.000,0.000,11.712,0.000,0.108,5.299,306.762,0.000,248.504,2.387169,70.262,8,0.000,0.000,0.000,0,0.000,0,11.712,315519.209,337686.835,320741.583,319952.735,330297.627,340322.986,276276.520,342999.077 +3.205154,0.100151,7444927,0.221876,2087159.685,0.000,0.000,14.033,0.000,0.101,4.849,306.762,0.000,207.303,2.453114,70.242,8,0.000,0.000,0.000,0,0.000,0,14.033,262602.375,273515.851,275972.131,255652.898,210581.144,273835.367,277270.166,257729.753 +3.305325,0.100171,7593977,0.226318,1487958.552,0.000,0.000,21.362,0.000,69.199,6.376,306.762,66.839,140.931,2.526840,70.233,8,0.000,0.000,0.000,0,0.000,0,21.362,90565.314,265815.984,251170.997,198171.521,247477.306,88438.945,255323.904,90994.580 +3.405510,0.100185,7764736,0.231407,1704444.363,0.000,0.000,93.388,0.000,0.263,8.455,306.902,135.219,135.492,2.628395,70.220,8,0.000,0.000,0.000,0,0.000,0,93.388,198693.302,179388.929,313920.644,305605.989,182443.292,234646.947,169477.221,120268.039 +3.505650,0.100141,8004265,0.238546,2391924.132,0.000,0.000,13.575,0.000,0.063,6.260,307.371,26.759,127.204,2.705452,70.207,8,0.000,0.000,0.000,0,0.000,0,13.575,291889.259,305270.430,309105.034,296323.020,307087.872,320459.056,256419.172,305370.289 +3.605788,0.100138,8255480,0.246032,2508686.933,0.000,0.000,10.275,0.000,0.321,5.032,307.371,0.000,294.281,2.774193,70.179,8,0.000,0.000,0.000,0,0.000,0,10.275,306866.392,268509.342,316063.696,323673.192,326459.346,320687.314,312588.493,333839.158 +3.705931,0.100143,8496427,0.253213,2406038.532,0.000,0.000,16.813,0.000,1.087,5.356,307.371,0.000,505.412,2.865833,70.129,8,0.000,0.000,0.000,0,0.000,0,16.813,311765.363,304905.147,293781.012,320243.272,314751.105,225378.567,313293.184,321920.880 +3.806078,0.100147,8747409,0.260693,2506138.583,0.000,0.000,12.043,0.000,0.091,5.584,307.371,0.039,365.284,2.950428,70.093,8,0.000,0.000,0.000,0,0.000,0,12.043,330913.901,326939.739,298841.015,307108.869,292739.977,329695.690,302625.455,317273.937 +3.906231,0.100153,9002655,0.268300,2548565.206,0.000,0.000,12.062,0.000,0.054,5.458,307.371,0.000,232.340,3.020954,70.071,8,0.000,0.000,0.000,0,0.000,0,12.062,337873.651,304854.113,328058.651,300720.430,324534.037,317115.375,314758.976,320649.973 +4.006374,0.100144,9267628,0.276197,2645932.056,0.000,0.000,13.224,0.000,9.770,5.654,307.371,0.195,241.450,3.102116,70.047,8,0.000,0.000,0.000,0,0.000,0,13.224,279848.309,351735.123,308886.627,338723.799,355519.691,339912.094,344695.228,326611.186 +4.106521,0.100147,9428005,0.280976,1601419.213,0.000,0.000,80.281,0.000,18.463,8.646,307.684,142.408,167.801,3.194740,70.031,8,0.000,0.000,0.000,0,0.000,0,80.281,322905.993,50186.329,271890.881,53651.243,279309.990,59592.522,282585.182,281297.073 +4.206688,0.100167,9626586,0.286895,1982501.997,0.000,0.000,84.906,0.000,0.132,7.903,307.996,96.948,136.608,3.287861,70.017,8,0.000,0.000,0.000,0,0.000,0,84.906,337367.069,328811.345,236715.017,217297.418,238022.835,286901.277,95720.281,241666.755 +4.306834,0.100146,9885239,0.294603,2582756.773,0.000,0.000,12.827,0.000,0.162,5.719,307.996,0.000,270.464,3.395813,69.991,8,0.000,0.000,0.000,0,0.000,0,12.827,348281.186,315519.049,332224.643,296806.387,332094.833,336947.743,336208.822,284674.111 +4.406981,0.100147,10140028,0.302196,2544142.148,0.000,0.000,15.534,0.000,0.086,5.214,307.996,0.000,469.075,3.465572,69.945,8,0.000,0.000,0.000,0,0.000,0,15.534,316234.146,315774.823,325121.054,340039.078,330712.817,277830.719,323643.231,314786.279 +4.507117,0.100136,10397894,0.309881,2575162.337,0.000,0.000,10.254,0.000,0.240,5.366,307.996,0.000,325.925,3.535496,69.913,8,0.000,0.000,0.000,0,0.000,0,10.254,322961.344,327854.698,319246.390,319246.390,303727.468,332548.323,307482.368,342095.356 +4.607253,0.100136,10649121,0.317369,2508866.647,0.000,0.000,17.775,0.000,0.811,5.254,307.996,0.078,373.790,3.614136,69.877,8,0.000,0.000,0.000,0,0.000,0,17.775,314972.730,339199.865,264231.562,333517.573,332918.386,288079.212,319926.011,316021.308 +4.707403,0.100150,10898230,0.324793,2487361.097,0.000,0.000,11.276,0.000,0.049,6.095,307.996,0.039,256.451,3.715839,69.851,8,0.000,0.000,0.000,0,0.000,0,11.276,326919.901,289865.451,313619.840,314937.864,333609.872,283924.358,314638.313,309845.498 +4.807542,0.100140,11102251,0.330873,2037361.931,0.000,0.000,20.166,0.000,39.547,5.096,307.996,46.107,160.908,3.781422,69.836,8,0.000,0.000,0.000,0,0.000,0,20.166,302177.580,235510.775,303585.612,200599.578,196874.785,206111.872,293609.557,298892.173 +4.907700,0.100158,11237000,0.334889,1345366.366,0.000,0.000,85.475,0.000,45.999,9.664,308.934,198.944,165.676,3.889313,69.820,8,0.000,0.000,0.000,0,0.000,0,85.475,161375.272,189141.444,339304.415,131742.048,9295.328,178058.938,187014.801,149434.121 +5.007910,0.100210,11481964,0.342189,2444512.049,0.000,0.000,79.000,0.000,0.121,7.030,309.090,48.531,151.752,3.976528,69.805,8,0.000,0.000,0.000,0,0.000,0,79.000,334518.268,300010.656,325237.736,330596.495,319749.249,325776.605,211546.231,297076.810 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.log new file mode 100644 index 000000000..bb1b4f4a4 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.log @@ -0,0 +1,14 @@ +version=splinterdb_build_version 7bb5fab3 records=100000000 key=16 value=16 cache_mib=256 disk_gib=16 inserters=8 readers=0 memtable_bg=1 normal_bg=3 memtable_mib=8 queue_scale=100 log=1 checkpoints=disabled direct=1 query_range=inserted seed=42 report_ms=100 barrier_ms=0 target_insert_ops_s=0 id_base=0 mode=create +Splinter task system created 1 background thread of type 'TASK_TYPE_MEMTABLE'. +Splinter task system created 3 background threads of type 'TASK_TYPE_NORMAL'. +worker=5 kind=insert tid=5 os_tid=18 +worker=7 kind=insert tid=7 os_tid=20 +worker=1 kind=insert tid=6 os_tid=14 +worker=6 kind=insert tid=8 os_tid=19 +worker=0 kind=insert tid=9 os_tid=13 +start_monotonic_ns=1819256838842837 main_address=0x55bae9f4fa80 +worker=2 kind=insert tid=10 os_tid=15 +worker=4 kind=insert tid=11 os_tid=17 +worker=3 kind=insert tid=12 os_tid=16 +load_end reason=time_limit inserted=11482220 elapsed_s=5.008 error=0 barrier_count=0 barrier_avg_ms=0.000 barrier_max_ms=0.000 +close_s=0.141 close_rc=0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.summary.csv new file mode 100644 index 000000000..42dd6e34d --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-00-main.summary.csv @@ -0,0 +1,2 @@ +target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted,insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms,insert_service_p999_ms,insert_service_max_ms,insert_response_mean_ms,insert_response_p50_ms,insert_response_p99_ms,insert_response_p999_ms,insert_response_max_ms,query_service_mean_ms,query_service_p99_ms,query_service_max_ms +0,2292714.900132,5.008132,11482220,0.003431,0.001424,0.061952,0.198656,146.300317,0.003431,0.001424,0.061952,0.198656,146.300317,0.000000,0.000000,0.000000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.csv new file mode 100644 index 000000000..c644476c9 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.csv @@ -0,0 +1,51 @@ +elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s,hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib,read_mib_s,write_mib_s,image_gib,free_gib,inserters_running,barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s,offered_insert_ops_s,insert_backlog,response_max_ms,i0_ops_s,i1_ops_s,i2_ops_s,i3_ops_s,i4_ops_s,i5_ops_s,i6_ops_s,i7_ops_s +0.100061,0.100061,312191,0.009304,3120008.386,0.000,0.000,12.368,0.000,0.126,5.529,82.828,0.000,164.509,0.065773,71.049,8,0.000,0.000,0.000,0,0.000,0,12.368,366076.880,406012.539,395069.209,399356.596,374951.471,416805.961,394919.301,366816.429 +0.200198,0.100137,590541,0.017599,2779698.096,0.000,0.000,15.970,0.000,0.099,5.782,135.000,0.273,146.830,0.121986,71.035,8,0.000,0.000,0.000,0,0.000,0,15.970,357620.868,372510.502,352757.519,353236.864,300109.528,350161.071,341472.954,351828.790 +0.300339,0.100141,851035,0.025363,2601269.037,0.000,0.000,16.462,0.000,0.086,5.815,195.449,0.000,137.267,0.191074,71.021,8,0.000,0.000,0.000,0,0.000,0,16.462,342436.746,340010.171,339001.594,337633.525,222346.221,332950.134,339860.382,347030.264 +0.400495,0.100156,1111517,0.033126,2600770.626,0.000,0.000,16.893,0.000,0.073,5.992,251.387,0.000,137.247,0.250275,71.008,8,0.000,0.000,0.000,0,0.000,0,16.893,329926.308,332891.691,332991.536,336985.317,336835.550,331144.411,332482.328,267513.484 +0.500642,0.100147,1359816,0.040526,2479333.083,0.000,0.000,16.194,0.000,0.160,5.630,284.363,0.000,130.862,0.304821,70.995,8,0.000,0.000,0.000,0,0.000,0,16.194,311909.945,314955.453,268963.290,305908.797,321605.645,310172.508,313887.029,331930.416 +0.600781,0.100139,1612082,0.048044,2519150.395,0.000,0.000,11.914,0.000,0.063,5.756,286.238,0.000,133.057,0.371906,70.982,8,0.000,0.000,0.000,0,0.000,0,11.914,324637.724,291893.343,323669.074,329351.158,324038.559,287679.214,319385.042,318496.281 +0.700922,0.100141,1852808,0.055218,2403871.335,0.000,0.000,11.290,0.000,0.055,5.623,292.176,0.000,276.953,0.426682,70.955,8,0.000,0.000,0.000,0,0.000,0,11.290,299787.399,263498.554,308185.560,300246.751,304720.445,320518.175,310362.491,296551.960 +0.801070,0.100148,2101958,0.062643,2487819.793,0.000,0.000,14.857,0.000,0.010,6.152,293.113,3.744,258.719,0.490177,70.930,8,0.000,0.000,0.000,0,0.000,0,14.857,319028.065,313536.189,269661.093,312417.843,327455.598,313376.425,320316.160,312028.419 +0.901232,0.100162,2293953,0.068365,1916841.305,0.000,0.000,29.823,0.000,49.286,7.795,300.488,67.235,191.252,0.576370,70.911,8,0.000,0.000,0.000,0,0.000,0,29.823,248856.410,341446.249,335925.203,340248.192,159251.729,160769.268,172510.226,157834.028 +1.001383,0.100150,2422878,0.072207,1287312.646,0.000,0.000,108.783,0.000,53.861,8.722,306.738,118.143,593.638,0.689472,70.853,8,0.000,0.000,0.000,0,0.000,0,108.783,267777.006,268905.308,233248.970,169085.533,69665.157,87398.469,107907.603,83324.600 +1.101545,0.100162,2616021,0.077964,1928300.291,0.000,0.000,119.798,0.000,0.070,7.040,306.738,75.931,738.645,0.777550,70.781,8,0.000,0.000,0.000,0,0.000,0,119.798,259548.742,263672.050,266018.242,95904.343,265768.647,275572.734,272787.255,229028.278 +1.201694,0.100148,2844107,0.084761,2277478.741,0.000,0.000,17.153,0.000,0.100,6.137,306.738,0.000,429.909,0.856079,70.739,8,0.000,0.000,0.000,0,0.000,0,17.153,304408.061,286035.338,298197.281,280783.135,251426.719,284387.784,282061.238,290179.185 +1.301842,0.100148,3084213,0.091917,2397504.070,0.000,0.000,16.600,0.000,0.166,5.881,306.738,0.078,288.907,0.930954,70.710,8,0.000,0.000,0.000,0,0.000,0,16.600,293135.228,302950.670,306645.190,321323.420,259215.537,304009.100,298197.719,312027.207 +1.401997,0.100155,3333022,0.099332,2484247.193,0.000,0.000,11.527,0.000,0.036,5.235,306.738,0.000,186.079,0.999813,70.692,8,0.000,0.000,0.000,0,0.000,0,11.527,320993.465,313664.801,296441.444,305247.821,323020.330,291618.903,328112.453,305147.976 +1.502158,0.100161,3565781,0.106269,2323839.741,0.000,0.000,13.826,0.000,1.752,5.261,306.738,0.000,232.086,1.094200,70.670,8,0.000,0.000,0.000,0,0.000,0,13.826,319574.265,294524.690,289552.714,225216.541,300884.427,307723.390,294993.933,291369.782 +1.602310,0.100152,3816023,0.113726,2498633.097,0.000,0.000,14.787,0.000,0.047,5.816,306.738,0.000,269.630,1.177788,70.643,8,0.000,0.000,0.000,0,0.000,0,14.787,334263.394,321752.355,307933.299,266935.435,316050.996,346784.417,313335.112,291578.087 +1.702460,0.100150,4037065,0.120314,2207106.251,0.000,0.000,15.778,0.000,48.677,6.533,306.738,59.988,185.737,1.255341,70.625,8,0.000,0.000,0.000,0,0.000,0,15.778,337613.108,345760.875,347358.476,180209.434,288776.431,182765.596,184732.642,339889.690 +1.802633,0.100174,4213083,0.125560,1757130.588,0.000,0.000,80.201,0.000,0.052,8.441,306.738,87.543,134.337,1.359257,70.612,8,0.000,0.000,0.000,0,0.000,0,80.201,228074.189,306877.430,290136.483,113762.571,221425.727,188932.118,190150.004,217772.067 +1.902789,0.100156,4462319,0.132987,2488477.701,0.000,0.000,12.357,0.000,0.157,6.196,306.738,19.072,132.294,1.437416,70.599,8,0.000,0.000,0.000,0,0.000,0,12.357,322836.340,271895.813,313301.216,322347.103,312113.069,301479.659,328128.084,316376.418 +2.002955,0.100165,4707518,0.140295,2447940.837,0.000,0.000,12.901,0.000,2.077,5.287,306.738,0.000,243.698,1.505848,70.575,8,0.000,0.000,0.000,0,0.000,0,12.901,309977.264,315757.702,312692.772,321927.497,271231.354,305205.157,304196.825,306952.267 +2.103119,0.100165,4930994,0.146955,2231084.979,0.000,0.000,16.332,0.000,14.130,5.048,306.738,0.390,469.617,1.578518,70.529,8,0.000,0.000,0.000,0,0.000,0,16.332,294355.141,202516.417,280438.065,296661.342,294155.470,302791.245,283483.050,276684.248 +2.203284,0.100164,5155697,0.153652,2243342.192,0.000,0.000,18.992,0.000,0.138,5.746,306.738,0.000,351.960,1.676525,70.495,8,0.000,0.000,0.000,0,0.000,0,18.992,260042.519,288475.778,268568.503,274199.247,304898.780,260811.255,298219.760,288126.352 +2.303441,0.100158,5398528,0.160889,2424490.530,0.000,0.000,13.830,0.000,0.115,5.457,306.738,0.000,275.308,1.745564,70.468,8,0.000,0.000,0.000,0,0.000,0,13.830,309222.860,296383.087,310920.186,287896.457,302183.948,323610.194,317210.276,277063.522 +2.403595,0.100153,5645579,0.168251,2466730.746,0.000,0.000,16.393,0.000,0.155,5.426,306.738,0.078,216.192,1.816158,70.447,8,0.000,0.000,0.000,0,0.000,0,16.393,324712.511,329834.664,314188.635,321846.902,299241.535,251824.183,325840.783,299241.535 +2.503747,0.100153,5838500,0.174001,1926267.607,0.000,0.000,15.678,0.000,68.814,7.144,306.738,69.425,196.692,1.903992,70.427,8,0.000,0.000,0.000,0,0.000,0,15.678,101964.249,341208.800,106307.614,336246.380,331014.372,275728.821,102323.700,331473.671 +2.603912,0.100164,6023036,0.179500,1842329.626,0.000,0.000,83.112,0.000,0.116,8.079,306.738,140.394,121.675,2.005547,70.415,8,0.000,0.000,0.000,0,0.000,0,83.112,211392.289,240514.388,282445.417,240095.077,319035.232,203904.605,294415.727,50526.890 +2.704062,0.100150,6289895,0.187453,2664592.073,0.000,0.000,11.411,0.000,2.240,5.547,306.738,0.000,140.687,2.080345,70.402,8,0.000,0.000,0.000,0,0.000,0,11.411,333879.051,352670.856,334408.257,327548.549,342965.418,294827.644,341367.815,336924.482 +2.804210,0.100148,6540947,0.194935,2506818.382,0.000,0.000,15.324,0.000,0.235,5.432,306.738,0.000,475.158,2.185081,70.355,8,0.000,0.000,0.000,0,0.000,0,15.324,318469.741,310191.964,285957.749,313806.627,326717.562,291120.126,321425.377,339129.235 +2.904362,0.100153,6798576,0.202613,2572355.454,0.000,0.000,16.503,0.000,0.139,5.135,306.738,0.000,361.166,2.254745,70.320,8,0.000,0.000,0.000,0,0.000,0,16.503,259213.520,326580.479,331752.568,316196.362,325721.792,332631.224,341357.876,338901.633 +3.004516,0.100154,7046871,0.210013,2479143.325,0.000,0.000,14.949,0.000,0.123,5.260,306.738,0.039,384.410,2.324677,70.282,8,0.000,0.000,0.000,0,0.000,0,14.949,311421.818,317622.297,279351.062,301457.119,314017.832,312400.316,325839.680,317033.202 +3.104659,0.100143,7304074,0.217678,2568361.558,0.000,0.000,13.377,0.000,1.894,5.566,306.738,0.312,263.959,2.423759,70.256,8,0.000,0.000,0.000,0,0.000,0,13.377,289436.592,343768.988,347004.367,323158.426,342440.885,337028.615,322988.669,262535.016 +3.204816,0.100157,7540148,0.224714,2357028.670,0.000,0.000,15.813,0.000,27.276,5.826,306.738,39.118,190.715,2.500996,70.238,8,0.000,0.000,0.000,0,0.000,0,15.813,255198.170,349399.842,340843.315,293338.116,319606.754,325697.164,239313.182,233632.128 +3.304989,0.100172,7684973,0.229030,1445760.025,0.000,0.000,89.741,0.000,46.532,9.408,306.738,142.411,128.295,2.580597,70.225,8,0.000,0.000,0.000,0,0.000,0,89.741,124874.933,309976.140,178163.157,194025.837,298445.998,173630.962,65087.902,101555.096 +3.405174,0.100185,7916577,0.235932,2311762.153,0.000,0.000,82.619,0.000,0.143,6.892,306.738,39.497,134.205,2.694336,70.212,8,0.000,0.000,0.000,0,0.000,0,82.619,313589.712,280970.073,210380.697,325737.233,233408.085,304326.852,330528.367,312821.134 +3.505323,0.100149,8185798,0.243956,2688200.925,0.000,0.000,9.868,0.000,0.092,5.240,306.738,0.000,283.757,2.765713,70.184,8,0.000,0.000,0.000,0,0.000,0,9.868,346543.179,345065.383,331765.219,340721.861,300471.888,326872.515,346103.835,350657.044 +3.605474,0.100151,8427022,0.251145,2408597.358,0.000,0.000,16.377,0.000,0.105,5.114,306.738,0.000,446.707,2.833328,70.141,8,0.000,0.000,0.000,0,0.000,0,16.377,301464.081,294075.255,264519.953,318807.851,310021.140,305877.406,313286.202,300545.470 +3.705622,0.100148,8679909,0.258681,2525142.612,0.000,0.000,13.670,0.000,1.599,5.503,306.738,0.000,403.428,2.932915,70.101,8,0.000,0.000,0.000,0,0.000,0,13.670,328075.724,321445.511,327866.034,327177.051,308035.306,342933.792,247115.231,322493.963 +3.805772,0.100151,8941459,0.266476,2611559.705,0.000,0.000,13.857,0.000,0.453,5.635,306.738,0.000,289.602,3.013401,70.073,8,0.000,0.000,0.000,0,0.000,0,13.857,310671.262,311050.690,341574.636,284191.215,352368.350,351389.827,340386.428,319927.297 +3.905911,0.100139,9202094,0.274244,2602744.860,0.000,0.000,11.313,0.000,0.127,5.393,306.738,0.039,227.224,3.085800,70.051,8,0.000,0.000,0.000,0,0.000,0,11.313,320046.694,321344.896,346579.942,311848.050,331061.437,308153.168,317839.751,345870.924 +4.006062,0.100151,9392153,0.279908,1897731.409,0.000,0.000,76.203,0.000,76.810,7.977,306.738,73.834,166.741,3.179951,70.034,8,0.000,0.000,0.000,0,0.000,0,76.203,83903.614,337561.524,337571.509,79799.796,80538.683,342104.681,283572.849,352678.753 +4.106213,0.100152,9562779,0.284993,1703675.286,0.000,0.000,85.813,0.000,0.129,7.777,306.789,155.272,133.313,3.277744,70.021,8,0.000,0.000,0.000,0,0.000,0,85.813,198169.349,45041.665,48126.985,339275.272,330278.921,285207.301,235063.376,222512.418 +4.206364,0.100151,9829564,0.292944,2663824.934,0.000,0.000,11.098,0.000,8.523,5.841,306.789,0.000,170.212,3.356316,70.005,8,0.000,0.000,0.000,0,0.000,0,11.098,333745.707,347524.886,339886.428,348623.227,318498.745,300725.601,342961.781,331858.558 +4.306511,0.100147,10070114,0.300113,2401974.622,0.000,0.000,17.509,0.000,0.139,5.439,306.789,0.039,511.476,3.457218,69.955,8,0.000,0.000,0.000,0,0.000,0,17.509,309745.387,315516.916,309815.284,313809.422,283054.561,304682.817,265050.985,300299.251 +4.406661,0.100149,10303661,0.307073,2331985.990,0.000,0.000,14.200,0.000,0.157,5.398,306.789,0.000,325.061,3.524296,69.923,8,0.000,0.000,0.000,0,0.000,0,14.200,299961.854,284474.992,299592.406,238763.285,310246.489,303416.692,297755.151,297775.121 +4.506803,0.100142,10553899,0.314531,2498822.601,0.000,0.000,15.926,0.000,0.121,5.186,306.789,0.156,354.963,3.594215,69.888,8,0.000,0.000,0.000,0,0.000,0,15.926,321702.015,320493.736,314981.583,265492.038,317008.697,307422.344,321622.129,330100.060 +4.606976,0.100173,10821776,0.322514,2674151.660,0.000,0.000,12.626,0.000,9.771,5.523,306.789,0.000,244.578,3.673813,69.864,8,0.000,0.000,0.000,0,0.000,0,12.626,342588.340,346671.288,332176.322,296158.525,351632.720,366816.497,286545.128,351562.840 +4.707119,0.100144,11050954,0.329344,2288488.718,0.000,0.000,17.341,0.000,16.754,5.503,306.789,22.585,202.170,3.763023,69.844,8,0.000,0.000,0.000,0,0.000,0,17.341,251368.484,253705.124,295824.548,321008.329,329566.021,309944.241,278509.450,248562.520 +4.807283,0.100163,11196040,0.333668,1448494.970,0.000,0.000,87.549,0.000,34.181,8.698,306.945,153.694,166.603,3.841423,69.828,8,0.000,0.000,0.000,0,0.000,0,87.549,197178.058,86438.867,209817.421,294459.222,291504.047,196449.247,90122.852,82525.257 +4.907450,0.100167,11373416,0.338954,1770797.880,0.000,0.000,83.431,0.000,0.133,8.067,308.195,106.112,166.674,3.925884,69.812,8,0.000,0.000,0.000,0,0.000,0,83.431,271625.636,144158.857,148731.208,309542.210,160691.202,152255.313,308753.529,275039.924 +5.007625,0.100175,11616037,0.346185,2421961.202,0.000,0.000,12.732,0.000,12.990,4.941,308.195,0.000,272.374,4.000412,69.785,8,0.000,0.000,0.000,0,0.000,0,12.732,315536.461,310645.042,314098.982,308039.612,323492.504,223837.327,312362.030,313949.245 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.log new file mode 100644 index 000000000..c52651679 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.log @@ -0,0 +1,14 @@ +version=splinterdb_build_version fb878f13-dirty records=100000000 key=16 value=16 cache_mib=256 disk_gib=16 inserters=8 readers=0 memtable_bg=1 normal_bg=3 memtable_mib=8 queue_scale=100 log=1 checkpoints=disabled direct=1 query_range=inserted seed=42 report_ms=100 barrier_ms=0 target_insert_ops_s=0 id_base=0 mode=create +Splinter task system created 1 background thread of type 'TASK_TYPE_MEMTABLE'. +Splinter task system created 3 background threads of type 'TASK_TYPE_NORMAL'. +worker=4 kind=insert tid=5 os_tid=31 +worker=7 kind=insert tid=6 os_tid=34 +worker=5 kind=insert tid=7 os_tid=32 +start_monotonic_ns=1819262080039972 main_address=0x55fd25c0faa0 +worker=6 kind=insert tid=8 os_tid=33 +worker=0 kind=insert tid=9 os_tid=27 +worker=2 kind=insert tid=10 os_tid=29 +worker=3 kind=insert tid=11 os_tid=30 +worker=1 kind=insert tid=12 os_tid=28 +load_end reason=time_limit inserted=11616290 elapsed_s=5.010 error=0 barrier_count=0 barrier_avg_ms=0.000 barrier_max_ms=0.000 +close_s=0.126 close_rc=0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.summary.csv new file mode 100644 index 000000000..fce3a282b --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-01-disk-format.summary.csv @@ -0,0 +1,2 @@ +target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted,insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms,insert_service_p999_ms,insert_service_max_ms,insert_response_mean_ms,insert_response_p50_ms,insert_response_p99_ms,insert_response_p999_ms,insert_response_max_ms,query_service_mean_ms,query_service_p99_ms,query_service_max_ms +0,2318818.891143,5.009572,11616290,0.003394,0.001392,0.061952,0.198656,119.797830,0.003394,0.001392,0.061952,0.198656,119.797830,0.000000,0.000000,0.000000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.csv new file mode 100644 index 000000000..208050aa8 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.csv @@ -0,0 +1,51 @@ +elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s,hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib,read_mib_s,write_mib_s,image_gib,free_gib,inserters_running,barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s,offered_insert_ops_s,insert_backlog,response_max_ms,i0_ops_s,i1_ops_s,i2_ops_s,i3_ops_s,i4_ops_s,i5_ops_s,i6_ops_s,i7_ops_s +0.100270,0.100270,289207,0.008619,2884288.277,0.000,0.000,14.908,0.000,0.078,5.369,78.270,0.000,152.090,0.060547,71.050,8,0.000,0.000,0.000,0,0.000,0,14.908,362641.604,355460.977,376474.284,361624.348,325332.263,367528.419,372255.665,362970.716 +0.200421,0.100151,558290,0.016638,2686766.051,0.000,0.000,16.529,0.000,0.167,5.799,131.230,0.000,141.817,0.117874,71.036,8,0.000,0.000,0.000,0,0.000,0,16.529,339796.031,334973.326,353834.797,343580.307,347234.780,341363.660,281953.523,344029.628 +0.300567,0.100146,823942,0.024555,2652658.260,0.000,0.000,15.899,0.000,0.133,5.693,189.203,0.000,140.031,0.184914,71.022,8,0.000,0.000,0.000,0,0.000,0,15.899,348582.534,339735.413,345077.636,338427.318,232321.786,358677.837,350250.106,339585.631 +0.400703,0.100136,1083607,0.032294,2593118.199,0.000,0.000,13.746,0.000,0.092,6.095,247.484,0.000,136.884,0.246639,71.009,8,0.000,0.000,0.000,0,0.000,0,13.746,333286.068,323848.921,336411.810,336242.042,277462.099,330380.026,325157.139,330330.094 +0.500844,0.100141,1335546,0.039802,2515833.442,0.000,0.000,14.655,0.000,0.131,5.809,284.277,0.000,132.820,0.301865,70.996,8,0.000,0.000,0.000,0,0.000,0,14.655,322184.537,319228.716,317630.975,319228.716,319148.829,320636.725,328835.136,268939.808 +0.600983,0.100139,1594791,0.047528,2588844.413,0.000,0.000,11.771,0.000,10.159,5.782,285.996,0.000,136.763,0.365288,70.982,8,0.000,0.000,0.000,0,0.000,0,11.771,337839.477,336511.327,330270.020,337599.811,322540.785,340505.764,337450.020,246127.209 +0.701138,0.100154,1856903,0.055340,2617083.202,0.000,0.000,11.699,0.000,0.051,5.684,291.164,0.000,293.532,0.427124,70.954,8,0.000,0.000,0.000,0,0.000,0,11.699,335312.787,290092.538,334444.127,332487.145,329212.197,333845.051,333645.359,328043.999 +0.801278,0.100140,2114183,0.063008,2569190.955,0.000,0.000,13.171,0.000,0.048,6.226,292.258,3.745,260.767,0.492393,70.928,8,0.000,0.000,0.000,0,0.000,0,13.171,327040.593,326621.182,322536.919,332612.766,325802.332,328608.391,321288.673,284680.098 +0.901437,0.100159,2275069,0.067802,1606313.527,0.000,0.000,37.248,0.000,46.948,7.194,301.039,88.610,182.835,0.578201,70.910,8,0.000,0.000,0.000,0,0.000,0,37.248,149353.232,151120.430,314651.187,151060.525,314401.582,221838.322,151669.560,152218.689 +1.001606,0.100169,2392023,0.071288,1167566.754,0.000,0.000,106.089,0.000,29.510,8.848,305.422,69.804,589.356,0.662106,70.853,8,0.000,0.000,0.000,0,0.000,0,106.089,73346.042,90467.106,279807.112,77998.179,196078.617,278888.664,94350.543,76630.491 +1.101769,0.100163,2569690,0.076583,1773770.152,0.000,0.000,32.146,0.000,9.657,6.199,305.422,66.337,684.428,0.737427,70.786,8,0.000,0.000,0.000,0,0.000,0,32.146,199713.498,225331.617,229205.284,227857.487,233887.629,216715.702,223854.032,217204.903 +1.202201,0.100432,2781514,0.082896,2109135.432,0.000,0.000,21.592,0.000,1.862,6.093,305.422,23.220,446.743,0.816708,70.742,8,0.000,0.000,0.000,0,0.000,0,21.592,250279.610,261013.276,273359.979,242791.933,272742.644,276347.084,262845.368,269755.538 +1.302359,0.100158,3015971,0.089883,2340869.624,0.000,0.000,13.605,0.000,0.044,6.268,305.422,0.078,318.364,0.922192,70.711,8,0.000,0.000,0.000,0,0.000,0,13.605,294055.166,282942.733,252331.123,305646.843,304957.932,312635.795,300974.229,287325.804 +1.402503,0.100144,3282027,0.097812,2656745.206,0.000,0.000,10.195,0.000,0.140,5.273,305.422,0.000,163.125,0.993137,70.695,8,0.000,0.000,0.000,0,0.000,0,10.195,327349.961,326990.478,336566.727,306939.269,349578.044,335478.290,336147.329,337695.107 +1.502637,0.100135,3532982,0.105291,2506168.152,0.000,0.000,12.088,0.000,0.045,5.418,305.422,0.000,232.031,1.063091,70.672,8,0.000,0.000,0.000,0,0.000,0,12.088,329695.106,308353.904,328117.235,312558.231,274589.466,312917.746,322205.213,317731.250 +1.602913,0.100276,3775153,0.112508,2415051.438,0.000,0.000,12.564,0.000,2.544,5.739,305.422,0.000,213.162,1.166519,70.651,8,0.000,0.000,0.000,0,0.000,0,12.564,291396.588,317913.478,298138.001,298177.891,312179.287,294567.844,330219.548,272458.801 +1.703072,0.100159,3980966,0.118642,2054870.236,0.000,0.000,20.852,0.000,16.656,5.079,305.422,28.002,218.404,1.228031,70.630,8,0.000,0.000,0.000,0,0.000,0,20.852,248395.955,235117.020,265978.063,292555.901,283270.631,230943.640,250422.739,248186.287 +1.803230,0.100158,4126684,0.122985,1454882.101,0.000,0.000,82.392,0.000,0.728,9.240,305.422,84.086,137.166,1.310722,70.616,8,0.000,0.000,0.000,0,0.000,0,82.392,90726.703,16993.160,335430.208,332494.844,246031.408,248118.112,89828.122,95259.543 +1.903390,0.100160,4359600,0.129926,2325434.468,0.000,0.000,13.706,0.000,1.343,6.060,305.422,32.565,123.981,1.407841,70.604,8,0.000,0.000,0.000,0,0.000,0,13.706,290704.269,300149.137,302565.266,306608.788,297084.048,250738.297,307307.669,270276.995 +2.003743,0.100353,4602387,0.137162,2419341.651,0.000,0.000,12.644,0.000,0.033,5.557,305.422,0.000,154.183,1.492485,70.589,8,0.000,0.000,0.000,0,0.000,0,12.644,300510.679,256446.010,310505.447,299384.648,321656.140,317889.418,311960.318,300988.993 +2.103897,0.100155,4809729,0.143341,2070212.020,0.000,0.000,18.235,0.000,0.175,5.098,305.422,0.000,432.143,1.556400,70.547,8,0.000,0.000,0.000,0,0.000,0,18.235,269682.103,260057.018,264060.814,271898.669,263831.170,213838.638,255414.213,271429.396 +2.204058,0.100160,5048728,0.150464,2386168.932,0.000,0.000,16.620,0.000,0.122,5.215,305.422,0.000,386.490,1.623894,70.509,8,0.000,0.000,0.000,0,0.000,0,16.620,293310.310,305131.381,298791.533,294787.944,296684.907,268320.328,310982.012,318160.517 +2.304207,0.100149,5263696,0.156870,2146480.049,0.000,0.000,16.120,0.000,0.173,5.151,305.422,0.000,288.125,1.689674,70.481,8,0.000,0.000,0.000,0,0.000,0,16.120,271175.734,268220.140,231704.577,269697.937,285214.805,275958.604,271045.927,273462.325 +2.404356,0.100150,5459362,0.162702,1953737.716,0.000,0.000,12.385,0.000,0.118,5.106,305.422,0.000,177.937,1.752968,70.463,8,0.000,0.000,0.000,0,0.000,0,12.385,244574.181,234019.967,251214.249,231274.074,238293.575,271513.886,243515.764,239332.022 +2.504504,0.100147,5716508,0.170365,2567673.067,0.000,0.000,15.439,0.000,5.456,5.467,305.422,0.000,214.332,1.828972,70.442,8,0.000,0.000,0.000,0,0.000,0,15.439,325649.716,312029.803,341356.550,320687.035,341366.536,317581.615,302214.279,306787.534 +2.604647,0.100144,5865062,0.174792,1483410.594,0.000,0.000,83.831,0.000,89.384,6.714,305.422,103.523,172.526,1.911102,70.426,8,0.000,0.000,0.000,0,0.000,0,83.831,40731.531,286259.081,32543.285,276303.372,270142.216,260346.278,43607.402,273477.429 +2.704808,0.100161,6043093,0.180098,1777448.202,0.000,0.000,91.566,0.000,1.706,6.891,305.422,99.800,123.707,2.004269,70.413,8,0.000,0.000,0.000,0,0.000,0,91.566,269785.629,274547.961,275576.306,212837.319,187637.892,285090.986,206118.137,65853.971 +2.804941,0.100133,6299898,0.187752,2564648.455,0.000,0.000,12.052,0.000,2.906,5.720,305.422,10.845,141.531,2.081463,70.400,8,0.000,0.000,0.000,0,0.000,0,12.052,293390.870,327225.994,329602.841,304006.790,322951.663,333387.821,336613.543,317468.935 +2.905100,0.100159,6534213,0.194735,2339425.634,0.000,0.000,14.047,0.000,3.154,5.298,305.422,0.039,436.454,2.182644,70.357,8,0.000,0.000,0.000,0,0.000,0,14.047,307350.698,290337.782,297745.988,286803.409,306581.922,302827.898,245818.657,301959.281 +3.005251,0.100151,6788063,0.202300,2534684.767,0.000,0.000,12.678,0.000,0.041,5.276,305.422,0.000,375.958,2.253109,70.320,8,0.000,0.000,0.000,0,0.000,0,12.678,317721.762,326768.145,338680.215,315834.603,277112.887,335814.529,302804.216,319948.410 +3.105390,0.100140,7032532,0.209586,2441284.043,0.000,0.000,14.600,0.000,0.122,5.107,305.422,0.000,351.463,2.322357,70.286,8,0.000,0.000,0.000,0,0.000,0,14.600,328102.248,298583.431,267237.164,293979.854,318735.316,322839.590,305184.222,306622.216 +3.205530,0.100140,7293428,0.217361,2605307.671,0.000,0.000,11.307,0.000,9.014,5.343,305.422,0.000,248.480,2.399879,70.261,8,0.000,0.000,0.000,0,0.000,0,11.307,292070.552,350967.985,321449.367,328229.861,346334.481,325373.865,329518.055,311363.506 +3.305688,0.100157,7516111,0.223998,2223332.742,0.000,0.000,19.391,0.000,12.266,4.954,305.422,5.382,183.656,2.484131,70.244,8,0.000,0.000,0.000,0,0.000,0,19.391,268018.412,304401.183,264503.941,289674.348,244555.319,277303.807,283773.630,291102.102 +3.405830,0.100143,7666066,0.228467,1497415.090,0.000,0.000,81.551,0.000,32.750,8.188,305.422,157.978,180.485,2.566315,70.226,8,0.000,0.000,0.000,0,0.000,0,81.551,311096.460,309388.895,1717.551,242054.895,247776.737,240227.500,67563.672,77589.378 +3.506228,0.100398,7894111,0.235263,2271418.567,0.000,0.000,86.351,0.000,0.596,7.241,305.422,64.470,129.213,2.676224,70.213,8,0.000,0.000,0.000,0,0.000,0,86.351,316013.492,278104.224,290156.304,307646.760,335635.473,141696.597,315146.938,287018.779 +3.606384,0.100156,8151153,0.242923,2566415.135,0.000,0.000,9.424,0.000,0.198,5.181,305.422,0.000,251.366,2.760498,70.189,8,0.000,0.000,0.000,0,0.000,0,9.424,309027.765,345231.270,319191.904,315877.077,286842.385,334448.097,339869.637,315926.999 +3.706521,0.100137,8398782,0.250303,2472906.470,0.000,0.000,16.185,0.000,0.101,5.107,305.422,0.039,429.529,2.829426,70.147,8,0.000,0.000,0.000,0,0.000,0,16.185,322938.143,302745.771,308677.655,299620.048,314399.826,267563.908,331067.021,325894.099 +3.806656,0.100135,8650783,0.257813,2516612.875,0.000,0.000,13.948,0.000,1.754,5.157,305.422,0.000,353.819,2.900902,70.112,8,0.000,0.000,0.000,0,0.000,0,13.948,325161.072,280870.858,300993.695,334768.103,318989.403,328806.151,309991.549,317032.045 +3.906787,0.100131,8901168,0.265275,2500563.734,0.000,0.000,13.960,0.000,0.145,5.608,305.422,0.000,366.432,3.007187,70.076,8,0.000,0.000,0.000,0,0.000,0,13.960,305518.485,280761.021,344716.970,309882.749,312758.969,325911.684,291566.820,329447.037 +4.006941,0.100154,9165357,0.273149,2637822.557,0.000,0.000,12.516,0.000,0.129,5.444,305.422,0.000,239.787,3.080490,70.053,8,0.000,0.000,0.000,0,0.000,0,12.516,353614.737,271511.338,343240.733,341862.858,332117.884,339137.061,321404.404,334933.543 +4.107085,0.100144,9373009,0.279337,2073531.212,0.000,0.000,17.576,0.000,56.243,6.886,305.441,62.995,156.571,3.162739,70.037,8,0.000,0.000,0.000,0,0.000,0,17.576,146309.110,338222.486,334997.135,147786.980,342346.542,345082.598,277609.853,141176.508 +4.207249,0.100163,9525079,0.283869,1518224.156,0.000,0.000,78.272,0.000,80.397,8.230,306.535,141.917,131.348,3.267982,70.025,8,0.000,0.000,0.000,0,0.000,0,78.272,124856.390,207671.340,260345.442,246757.600,309904.623,67060.641,69876.050,231752.070 +4.307409,0.100160,9765144,0.291024,2396807.989,0.000,0.000,91.362,0.000,0.143,6.017,306.535,29.874,131.196,3.342487,70.012,8,0.000,0.000,0.000,0,0.000,0,91.362,314126.465,313008.257,315664.000,311320.962,305520.260,257277.592,275378.576,304511.877 +4.407549,0.100140,10005466,0.298186,2399857.080,0.000,0.000,12.606,0.000,0.158,5.077,306.535,0.000,331.957,3.410408,69.979,8,0.000,0.000,0.000,0,0.000,0,12.606,296194.942,303824.251,270461.003,314519.264,290413.044,312581.979,300758.547,311104.050 +4.507685,0.100136,10255056,0.305624,2492514.841,0.000,0.000,16.164,0.000,2.110,5.215,306.535,0.000,482.002,3.488224,69.932,8,0.000,0.000,0.000,0,0.000,0,16.164,327375.382,316530.111,319985.418,271870.764,278601.623,336982.334,316739.826,324429.383 +4.607840,0.100155,10501037,0.312955,2456005.745,0.000,0.000,16.834,0.000,0.037,5.480,306.535,0.000,392.166,3.586845,69.894,8,0.000,0.000,0.000,0,0.000,0,16.834,320972.826,309280.936,319624.914,273915.716,308841.617,316250.141,303220.324,303899.272 +4.707989,0.100149,10755692,0.320545,2542763.799,0.000,0.000,12.608,0.000,0.137,5.308,306.535,0.000,251.735,3.657352,69.869,8,0.000,0.000,0.000,0,0.000,0,12.608,320372.961,336598.801,312115.257,266373.367,312294.990,347262.922,320652.545,327092.955 +4.808139,0.100151,11011970,0.328182,2558926.283,0.000,0.000,11.449,0.000,1.674,5.325,306.535,0.000,214.794,3.731224,69.848,8,0.000,0.000,0.000,0,0.000,0,11.449,316313.635,338809.756,305899.319,333148.282,337851.199,331341.004,260108.280,335454.808 +4.908275,0.100135,11167732,0.332824,1555513.213,0.000,0.000,83.869,0.000,2.393,6.996,306.535,111.529,158.574,3.817741,69.833,8,0.000,0.000,0.000,0,0.000,0,83.869,292573.738,289128.404,44339.946,48034.941,288299.527,47715.374,249262.399,296158.882 +5.008423,0.100149,11326616,0.337559,1586481.489,0.000,0.000,88.916,0.000,0.189,8.156,307.629,178.407,173.141,3.915409,69.816,8,0.000,0.000,0.000,0,0.000,0,88.916,314772.050,156467.391,176218.029,269699.057,163287.252,316249.853,149048.420,40739.436 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.log new file mode 100644 index 000000000..e978539d3 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.log @@ -0,0 +1,14 @@ +version=splinterdb_build_version 7bb5fab3 records=100000000 key=16 value=16 cache_mib=256 disk_gib=16 inserters=8 readers=0 memtable_bg=1 normal_bg=3 memtable_mib=8 queue_scale=100 log=1 checkpoints=disabled direct=1 query_range=inserted seed=42 report_ms=100 barrier_ms=0 target_insert_ops_s=0 id_base=0 mode=create +Splinter task system created 1 background thread of type 'TASK_TYPE_MEMTABLE'. +Splinter task system created 3 background threads of type 'TASK_TYPE_NORMAL'. +worker=5 kind=insert tid=5 os_tid=186 +worker=0 kind=insert tid=6 os_tid=181 +worker=4 kind=insert tid=11 os_tid=185 +worker=1 kind=insert tid=7 os_tid=182 +worker=2 kind=insert tid=9 os_tid=183 +worker=6 kind=insert tid=10 os_tid=187 +worker=3 kind=insert tid=12 os_tid=184 +start_monotonic_ns=1819320977826794 main_address=0x562444fe3a80 +worker=7 kind=insert tid=8 os_tid=188 +load_end reason=time_limit inserted=11326742 elapsed_s=5.009 error=0 barrier_count=0 barrier_avg_ms=0.000 barrier_max_ms=0.000 +close_s=0.218 close_rc=0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.summary.csv new file mode 100644 index 000000000..b791b6e94 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-12-main.summary.csv @@ -0,0 +1,2 @@ +target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted,insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms,insert_service_p999_ms,insert_service_max_ms,insert_response_mean_ms,insert_response_p50_ms,insert_response_p99_ms,insert_response_p999_ms,insert_response_max_ms,query_service_mean_ms,query_service_p99_ms,query_service_max_ms +0,2261423.232769,5.008679,11326742,0.003481,0.001392,0.064000,0.206848,106.089198,0.003481,0.001392,0.064000,0.206848,106.089198,0.000000,0.000000,0.000000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.csv new file mode 100644 index 000000000..16a48ff69 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.csv @@ -0,0 +1,51 @@ +elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s,hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib,read_mib_s,write_mib_s,image_gib,free_gib,inserters_running,barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s,offered_insert_ops_s,insert_backlog,response_max_ms,i0_ops_s,i1_ops_s,i2_ops_s,i3_ops_s,i4_ops_s,i5_ops_s,i6_ops_s,i7_ops_s +0.100062,0.100062,282524,0.008420,2823491.073,0.000,0.000,16.419,0.000,0.176,5.083,76.438,0.000,148.892,0.059223,71.050,8,0.000,0.000,0.000,0,0.000,0,16.419,360156.912,293837.991,359417.370,363744.689,361705.952,369031.415,361635.995,353960.750 +0.200201,0.100139,476187,0.014191,1933942.671,0.000,0.000,27.332,0.000,0.153,5.172,116.984,0.000,102.007,0.104740,71.040,8,0.000,0.000,0.000,0,0.000,0,27.332,251989.845,242143.527,247556.006,245448.934,257831.727,242043.666,252728.818,194200.147 +0.300340,0.100140,702215,0.020928,2257129.656,0.000,0.000,19.353,0.000,0.152,5.402,164.484,0.000,119.131,0.154949,71.028,8,0.000,0.000,0.000,0,0.000,0,19.353,288197.754,294848.471,223168.517,291892.597,293370.534,286609.970,285980.848,293060.966 +0.400473,0.100133,968762,0.028871,2661929.793,0.000,0.000,16.735,0.000,0.114,5.983,216.258,0.000,140.594,0.211666,71.014,8,0.000,0.000,0.000,0,0.000,0,16.735,349555.113,338469.855,337730.838,330340.667,345660.293,277670.715,339598.355,342903.958 +0.500624,0.100150,1234361,0.036787,2652000.993,0.000,0.000,16.094,0.000,0.062,5.790,276.945,0.000,139.985,0.281887,71.001,8,0.000,0.000,0.000,0,0.000,0,16.094,342105.422,352449.863,339888.756,346738.453,345999.565,282724.740,296643.803,345450.391 +0.600767,0.100143,1517958,0.045239,2831920.948,0.000,0.000,9.933,0.000,0.107,5.927,285.695,0.000,149.474,0.345444,70.986,8,0.000,0.000,0.000,0,0.000,0,9.933,357648.638,355431.807,348811.273,365317.672,366056.616,316357.675,367993.846,354303.421 +0.700900,0.100134,1767640,0.052680,2493488.052,0.000,0.000,11.384,0.000,3.268,5.792,285.695,0.000,211.748,0.400570,70.965,8,0.000,0.000,0.000,0,0.000,0,11.384,311274.057,310345.298,309616.272,323208.110,281933.264,321560.312,314070.320,321480.419 +0.801064,0.100164,2017643,0.060130,2495945.560,0.000,0.000,15.467,0.000,0.137,5.721,292.715,0.000,268.428,0.469959,70.939,8,0.000,0.000,0.000,0,0.000,0,15.467,321963.130,321414.028,318768.358,326545.631,328212.903,271755.291,318428.913,288857.305 +0.901225,0.100160,2232865,0.066545,2148773.110,0.000,0.000,30.077,0.000,11.543,5.709,300.090,33.072,226.083,0.540501,70.917,8,0.000,0.000,0.000,0,0.000,0,30.077,261540.458,292251.194,256927.857,299958.830,195097.041,289725.246,259274.093,293998.391 +1.001371,0.100146,2366431,0.070525,1333707.679,0.000,0.000,110.391,0.000,111.441,10.122,304.465,88.308,278.148,0.635777,70.890,8,0.000,0.000,0.000,0,0.000,0,110.391,9.985,12142.226,0.000,286380.787,348350.075,325154.030,23885.036,337785.539 +1.101535,0.100164,2488059,0.074150,1214284.736,0.000,0.000,112.192,0.000,104.165,4.970,304.895,120.778,683.213,0.709534,70.823,8,0.000,0.000,0.000,0,0.000,0,112.192,178037.456,173904.247,166795.928,176150.556,178316.997,0.000,176380.179,164699.373 +1.201679,0.100143,2697805,0.080401,2094456.863,0.000,0.000,158.303,0.000,0.195,6.442,304.895,16.851,573.202,0.794327,70.767,8,0.000,0.000,0.000,0,0.000,0,158.303,288925.724,304153.889,283753.141,281696.090,283753.141,138900.837,285969.962,227304.080 +1.301832,0.100153,2913978,0.086843,2158417.110,0.000,0.000,15.428,0.000,0.101,5.993,304.895,0.039,369.589,0.871387,70.731,8,0.000,0.000,0.000,0,0.000,0,15.428,278552.458,223137.513,277813.592,273080.856,288157.715,286260.627,261348.863,270065.485 +1.401980,0.100148,3155066,0.094028,2407311.978,0.000,0.000,13.846,0.000,7.886,5.006,304.895,0.039,244.793,0.944859,70.706,8,0.000,0.000,0.000,0,0.000,0,13.846,285217.262,317139.948,269240.942,324519.011,318467.979,306056.375,314244.240,272426.221 +1.502131,0.100150,3396933,0.101236,2415038.410,0.000,0.000,13.560,0.000,0.206,5.549,304.895,0.000,210.660,1.046566,70.686,8,0.000,0.000,0.000,0,0.000,0,13.560,300178.609,294666.898,279349.930,310463.144,333159.015,298780.711,319938.895,278501.207 +1.602274,0.100144,3655551,0.108944,2582468.605,0.000,0.000,11.541,0.000,0.110,5.460,304.895,0.000,235.989,1.117271,70.663,8,0.000,0.000,0.000,0,0.000,0,11.541,348589.025,348618.982,279737.975,302095.843,332602.001,312570.789,347360.790,310893.200 +1.702434,0.100160,3893434,0.116033,2375030.284,0.000,0.000,12.282,0.000,0.159,5.368,304.895,0.000,247.456,1.186909,70.639,8,0.000,0.000,0.000,0,0.000,0,12.282,322863.464,313917.776,293390.616,316952.920,289007.628,277056.748,278474.480,283366.653 +1.802574,0.100139,4065577,0.121164,1719032.380,0.000,0.000,19.095,0.000,67.367,6.583,304.895,63.739,166.213,1.269016,70.622,8,0.000,0.000,0.000,0,0.000,0,19.095,83962.893,258988.775,295837.381,101238.797,298413.787,81476.361,292442.116,306672.269 +1.902732,0.100158,4217124,0.125680,1513073.383,0.000,0.000,78.862,0.000,0.061,6.468,304.895,72.268,101.909,1.360325,70.612,8,0.000,0.000,0.000,0,0.000,0,78.862,196808.268,174253.992,181033.254,189130.429,231343.566,191167.203,190488.278,158848.394 +2.002875,0.100143,4448747,0.132583,2312928.526,0.000,0.000,11.936,0.000,0.116,5.578,304.895,20.284,123.535,1.435368,70.600,8,0.000,0.000,0.000,0,0.000,0,11.936,291943.280,252699.297,287090.207,304764.978,299921.891,287249.979,297375.526,291883.366 +2.103015,0.100140,4703507,0.140175,2544034.840,0.000,0.000,9.890,0.000,0.128,5.272,304.895,0.000,208.224,1.504902,70.580,8,0.000,0.000,0.000,0,0.000,0,9.890,327481.075,323906.084,293368.879,321409.583,324405.385,323127.176,311104.025,319232.634 +2.203164,0.100149,4942410,0.147295,2385487.337,0.000,0.000,15.596,0.000,1.934,5.174,304.895,0.000,555.230,1.595455,70.526,8,0.000,0.000,0.000,0,0.000,0,15.596,259904.019,302670.507,334742.877,300463.784,301861.708,268121.815,319944.853,297777.773 +2.303298,0.100135,5192050,0.154735,2493038.561,0.000,0.000,11.502,0.000,0.073,5.322,304.895,0.000,359.905,1.680714,70.490,8,0.000,0.000,0.000,0,0.000,0,11.502,315364.784,298617.365,271194.341,318260.879,326939.178,323683.568,310681.099,328297.347 +2.403443,0.100145,5457241,0.162638,2648083.281,0.000,0.000,12.965,0.000,0.074,5.486,304.895,0.000,266.685,1.752506,70.464,8,0.000,0.000,0.000,0,0.000,0,12.965,343803.173,321435.497,324870.533,339599.249,325180.086,338630.648,352480.634,302083.462 +2.503590,0.100147,5713879,0.170287,2562608.353,0.000,0.000,15.574,0.000,7.866,5.347,304.895,0.000,213.982,1.830315,70.443,8,0.000,0.000,0.000,0,0.000,0,15.574,306658.660,297681.872,339420.441,327158.488,317922.082,340868.310,318161.730,314736.770 +2.603741,0.100151,5850482,0.174358,1363968.525,0.000,0.000,86.654,0.000,9.246,6.960,304.895,92.087,165.102,1.909115,70.427,8,0.000,0.000,0.000,0,0.000,0,86.654,251949.209,269053.358,261624.586,19949.848,17543.485,267885.124,14787.650,261175.265 +2.703900,0.100158,6014359,0.179242,1636178.490,0.000,0.000,26.210,0.000,109.403,5.529,304.895,108.656,98.282,2.002285,70.418,8,0.000,0.000,0.000,0,0.000,0,26.210,242745.520,254157.445,185196.671,259329.254,246030.318,231513.311,217205.972,0.000 +2.804050,0.100151,6276260,0.187047,2615068.099,0.000,0.000,109.503,0.000,0.069,5.516,304.895,10.570,138.931,2.075722,70.404,8,0.000,0.000,0.000,0,0.000,0,109.503,357950.433,294016.805,329832.817,343582.091,325109.936,328544.758,319887.808,316143.452 +2.904188,0.100137,6522507,0.194386,2459093.296,0.000,0.000,12.759,0.000,0.116,5.335,304.895,0.000,426.290,2.181194,70.362,8,0.000,0.000,0.000,0,0.000,0,12.759,313329.751,322996.477,257007.091,326202.076,322696.889,284659.120,316904.842,315297.050 +3.004330,0.100142,6777211,0.201977,2543427.722,0.000,0.000,16.338,0.000,0.063,5.206,304.895,0.000,419.014,2.251942,70.321,8,0.000,0.000,0.000,0,0.000,0,16.338,317079.671,315362.110,274949.506,344041.379,317958.423,319995.530,321733.062,332308.043 +3.104477,0.100147,7014219,0.209040,2366591.431,0.000,0.000,13.256,0.000,0.072,4.938,304.895,0.000,332.908,2.320183,70.289,8,0.000,0.000,0.000,0,0.000,0,13.256,292538.772,292778.418,294825.401,300377.217,319439.118,319029.721,245907.510,301695.274 +3.204611,0.100134,7256939,0.216274,2423953.938,0.000,0.000,12.293,0.000,0.118,5.328,304.895,0.000,220.252,2.389214,70.267,8,0.000,0.000,0.000,0,0.000,0,12.293,313190.588,292328.525,320011.453,305570.792,335310.965,297821.170,311153.316,248567.129 +3.304761,0.100150,7492480,0.223293,2351882.576,0.000,0.000,16.173,0.000,16.645,5.199,304.895,0.819,201.105,2.467999,70.248,8,0.000,0.000,0.000,0,0.000,0,16.173,298662.058,295137.344,311053.473,279301.096,254977.577,290384.473,325112.387,297254.169 +3.404901,0.100140,7608387,0.226748,1157446.924,0.000,0.000,92.261,0.000,4.723,7.122,304.895,102.552,148.191,2.541851,70.233,8,0.000,0.000,0.000,0,0.000,0,92.261,10724.960,8258.419,215947.179,223167.055,231974.704,223167.055,237187.394,7020.156 +3.505049,0.100147,7749224,0.230945,1406298.424,0.000,0.000,45.222,0.000,101.520,6.150,306.348,97.162,93.729,2.629681,70.224,8,0.000,0.000,0.000,0,0.000,0,45.222,0.000,163908.551,184178.692,223890.194,229062.575,235892.514,191398.057,177967.841 +3.605203,0.100155,7984598,0.237960,2350100.869,0.000,0.000,109.699,0.000,0.215,5.978,306.348,26.755,124.729,2.701881,70.212,8,0.000,0.000,0.000,0,0.000,0,109.699,263072.632,310319.470,292726.713,304798.020,300714.344,307923.181,277809.812,292736.697 +3.705370,0.100166,8226046,0.245155,2410475.188,0.000,0.000,10.901,0.000,0.178,5.172,306.348,0.000,292.678,2.769859,70.183,8,0.000,0.000,0.000,0,0.000,0,10.901,272187.782,298893.413,299442.500,306940.044,326537.484,303166.313,307329.397,295978.256 +3.805511,0.100141,8471792,0.252479,2453999.836,0.000,0.000,14.407,0.000,14.103,5.069,306.348,0.000,486.150,2.846622,70.136,8,0.000,0.000,0.000,0,0.000,0,14.407,319269.826,269619.833,312669.133,321446.757,275291.836,316743.389,322425.377,316533.684 +3.905661,0.100151,8720468,0.259890,2483020.075,0.000,0.000,14.443,0.000,0.120,5.487,306.348,0.000,384.655,2.946472,70.098,8,0.000,0.000,0.000,0,0.000,0,14.443,319159.282,318869.718,322803.793,320477.297,270302.870,314027.013,299728.549,317651.553 +4.005798,0.100137,8978559,0.267582,2577378.605,0.000,0.000,11.116,0.000,0.126,5.189,306.348,0.000,253.403,3.017399,70.073,8,0.000,0.000,0.000,0,0.000,0,11.116,309635.753,311852.715,329268.852,317624.806,354444.358,340403.596,303004.838,311143.686 +4.105947,0.100149,9232244,0.275142,2533087.504,0.000,0.000,13.004,0.000,2.827,5.362,306.348,0.000,208.596,3.091755,70.053,8,0.000,0.000,0.000,0,0.000,0,13.004,295570.977,348791.925,304747.347,278905.730,333245.018,304487.732,333215.062,334123.713 +4.206097,0.100150,9323282,0.277855,909016.820,0.000,0.000,30.257,0.000,73.380,3.456,306.660,51.953,102.659,3.146378,70.043,8,0.000,0.000,0.000,0,0.000,0,30.257,113779.374,164004.056,48027.976,162646.093,161817.335,48087.886,159171.304,51482.795 +4.306256,0.100159,9443809,0.281447,1203350.884,0.000,0.000,119.462,0.000,70.987,6.662,307.754,145.861,115.597,3.219002,70.031,8,0.000,0.000,0.000,0,0.000,0,119.462,174361.926,106400.312,73392.952,225720.019,188399.539,102107.158,217712.789,115256.188 +4.406415,0.100159,9688099,0.288728,2439032.378,0.000,0.000,108.623,0.000,0.157,6.513,307.910,42.667,140.519,3.333035,70.018,8,0.000,0.000,0.000,0,0.000,0,108.623,320651.536,171707.720,321879.589,312055.168,339910.996,330086.575,325783.399,316957.394 +4.506567,0.100153,9950674,0.296553,2621751.357,0.000,0.000,11.080,0.000,0.099,5.023,307.910,0.000,334.880,3.403568,69.985,8,0.000,0.000,0.000,0,0.000,0,11.080,324365.285,325773.137,332492.888,301240.554,348698.173,325273.899,322787.691,341119.731 +4.606716,0.100148,10181774,0.303441,2307579.106,0.000,0.000,16.610,0.000,0.114,4.837,307.910,0.000,438.140,3.470173,69.942,8,0.000,0.000,0.000,0,0.000,0,16.610,291128.414,296800.006,302950.887,293564.802,294493.425,286145.800,247632.894,294862.878 +4.706862,0.100147,10427405,0.310761,2452710.304,0.000,0.000,12.689,0.000,13.255,5.085,307.910,0.000,319.453,3.546928,69.911,8,0.000,0.000,0.000,0,0.000,0,12.689,274137.665,303714.257,283014.636,324383.921,325452.353,319481.117,318712.245,303814.110 +4.807021,0.100158,10664444,0.317825,2366646.013,0.000,0.000,14.836,0.000,0.127,5.781,307.910,0.000,337.123,3.646305,69.878,8,0.000,0.000,0.000,0,0.000,0,14.836,297010.136,298467.830,308362.177,273367.538,281494.681,278749.025,303919.206,325275.421 +4.907153,0.100132,10915770,0.325315,2509936.367,0.000,0.000,9.718,0.000,0.090,5.321,307.910,0.000,236.211,3.717476,69.855,8,0.000,0.000,0.000,0,0.000,0,9.718,312386.341,269742.809,308781.115,336154.867,309649.965,323431.715,326887.139,322902.416 +5.007294,0.100141,11104939,0.330953,1889022.040,0.000,0.000,20.576,0.000,54.232,6.123,308.379,55.820,148.657,3.793026,69.840,8,0.000,0.000,0.000,0,0.000,0,20.576,304240.306,145164.976,307925.102,299377.175,142708.446,301524.142,144835.442,243246.451 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.log new file mode 100644 index 000000000..0b9c4561a --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.log @@ -0,0 +1,14 @@ +version=splinterdb_build_version fb878f13-dirty records=100000000 key=16 value=16 cache_mib=256 disk_gib=16 inserters=8 readers=0 memtable_bg=1 normal_bg=3 memtable_mib=8 queue_scale=100 log=1 checkpoints=disabled direct=1 query_range=inserted seed=42 report_ms=100 barrier_ms=0 target_insert_ops_s=0 id_base=0 mode=create +Splinter task system created 1 background thread of type 'TASK_TYPE_MEMTABLE'. +Splinter task system created 3 background threads of type 'TASK_TYPE_NORMAL'. +worker=4 kind=insert tid=5 os_tid=339 +worker=2 kind=insert tid=7 os_tid=337 +worker=5 kind=insert tid=6 os_tid=340 +worker=7 kind=insert tid=8 os_tid=342 +worker=0 kind=insert tid=10 os_tid=335 +worker=6 kind=insert tid=9 os_tid=341 +worker=1 kind=insert tid=11 os_tid=336 +start_monotonic_ns=1819379870134847 main_address=0x55660b081aa0 +worker=3 kind=insert tid=12 os_tid=338 +load_end reason=time_limit inserted=11105082 elapsed_s=5.030 error=0 barrier_count=0 barrier_avg_ms=0.000 barrier_max_ms=0.000 +close_s=0.336 close_rc=0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.summary.csv new file mode 100644 index 000000000..0dc5d2534 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-23-disk-format.summary.csv @@ -0,0 +1,2 @@ +target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted,insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms,insert_service_p999_ms,insert_service_max_ms,insert_response_mean_ms,insert_response_p50_ms,insert_response_p99_ms,insert_response_p999_ms,insert_response_max_ms,query_service_mean_ms,query_service_p99_ms,query_service_max_ms +0,2207885.214204,5.029737,11105082,0.003560,0.001360,0.065024,0.210944,158.302842,0.003560,0.001360,0.065024,0.210944,158.302842,0.000000,0.000000,0.000000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.csv new file mode 100644 index 000000000..4ca591c0a --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.csv @@ -0,0 +1,51 @@ +elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s,hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib,read_mib_s,write_mib_s,image_gib,free_gib,inserters_running,barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s,offered_insert_ops_s,insert_backlog,response_max_ms,i0_ops_s,i1_ops_s,i2_ops_s,i3_ops_s,i4_ops_s,i5_ops_s,i6_ops_s,i7_ops_s +0.100064,0.100064,293185,0.008738,2929976.339,0.000,0.000,15.820,0.000,0.059,5.426,79.574,0.000,154.472,0.062214,71.049,8,0.000,0.000,0.000,0,0.000,0,15.820,315778.066,375230.048,377788.412,376948.949,377898.342,361628.746,375679.760,369024.017 +0.200209,0.100145,566490,0.016883,2729084.013,0.000,0.000,17.110,0.000,0.082,5.757,130.820,0.000,144.087,0.118664,71.035,8,0.000,0.000,0.000,0,0.000,0,17.110,351728.857,345527.869,345927.288,348333.791,352937.101,280552.293,353086.884,350989.931 +0.300351,0.100142,823208,0.024534,2563539.159,0.000,0.000,15.777,0.000,0.007,5.620,188.172,0.000,135.355,0.184269,71.022,8,0.000,0.000,0.000,0,0.000,0,15.777,284366.132,270086.412,338419.363,333236.724,335882.966,329252.383,338289.548,334005.632 +0.400496,0.100145,1086500,0.032380,2629116.168,0.000,0.000,16.150,0.000,0.125,6.016,247.234,0.000,138.784,0.246544,71.008,8,0.000,0.000,0.000,0,0.000,0,16.150,335604.444,265475.907,342883.912,337032.378,335344.820,340467.408,339049.460,333257.839 +0.500632,0.100136,1347768,0.040167,2609139.242,0.000,0.000,14.207,0.000,0.054,5.928,284.348,0.000,137.743,0.303226,70.995,8,0.000,0.000,0.000,0,0.000,0,14.207,280509.332,333557.343,326946.314,336063.941,338460.689,333287.709,327765.203,332548.711 +0.600771,0.100140,1610115,0.047985,2619810.805,0.000,0.000,11.590,0.000,0.035,5.579,286.066,0.000,138.361,0.372295,70.981,8,0.000,0.000,0.000,0,0.000,0,11.590,339495.813,334732.466,340194.836,307021.171,334013.470,291532.804,336649.788,336170.457 +0.700926,0.100155,1858554,0.055389,2480550.430,0.000,0.000,12.433,0.000,0.110,5.819,292.133,0.000,285.691,0.427090,70.953,8,0.000,0.000,0.000,0,0.000,0,12.433,316799.635,312466.343,317129.125,319615.277,280026.555,308083.127,313834.225,312596.142 +0.801067,0.100141,2102143,0.062649,2432460.158,0.000,0.000,15.861,0.000,0.047,6.152,293.070,2.496,250.740,0.490196,70.929,8,0.000,0.000,0.000,0,0.000,0,15.861,299797.277,265226.023,313647.747,309493.605,311101.338,317791.904,307905.844,307496.421 +0.901249,0.100181,2262089,0.067416,1596564.566,0.000,0.000,34.278,0.000,43.703,7.451,302.504,102.548,179.206,0.577896,70.911,8,0.000,0.000,0.000,0,0.000,0,34.278,218813.174,174583.386,175751.268,173335.649,177557.991,330520.588,173505.341,172497.169 +1.001414,0.100166,2357880,0.070270,956325.588,0.000,0.000,117.314,0.000,2.219,9.106,307.660,62.163,584.500,0.660961,70.854,8,0.000,0.000,0.000,0,0.000,0,117.314,274854.629,66140.421,69564.747,58003.901,71381.737,282212.439,51904.007,82263.708 +1.101555,0.100141,2569822,0.076587,2116429.654,0.000,0.000,21.754,0.000,9.150,5.738,307.660,20.869,660.512,0.739876,70.789,8,0.000,0.000,0.000,0,0.000,0,21.754,227049.198,283159.918,277447.988,268240.997,272674.732,275850.246,241548.711,270457.865 +1.201687,0.100131,2790437,0.083162,2203257.409,0.000,0.000,18.702,0.000,2.990,6.220,307.660,46.345,495.989,0.843422,70.741,8,0.000,0.000,0.000,0,0.000,0,18.702,280691.489,287482.573,284606.349,248313.996,289370.095,273960.326,261266.990,277565.592 +1.301840,0.100153,3032517,0.090376,2417102.631,0.000,0.000,13.568,0.000,0.065,5.861,307.660,13.222,278.831,0.924255,70.714,8,0.000,0.000,0.000,0,0.000,0,13.568,313550.371,314618.737,310245.427,292592.430,305053.369,295368.184,258045.276,327628.836 +1.401989,0.100149,3283306,0.097850,2504152.977,0.000,0.000,14.920,0.000,0.704,5.365,307.660,0.000,215.148,0.993320,70.692,8,0.000,0.000,0.000,0,0.000,0,14.920,313901.555,328210.202,328809.308,304425.696,330716.462,318644.477,262458.326,316986.951 +1.502118,0.100129,3535314,0.105361,2516828.182,0.000,0.000,9.695,0.000,2.945,5.319,307.660,0.000,192.603,1.065781,70.674,8,0.000,0.000,0.000,0,0.000,0,9.695,315901.845,313365.123,304876.091,324440.813,325918.903,303258.181,321015.239,308051.988 +1.602269,0.100151,3774655,0.112493,2389802.712,0.000,0.000,12.851,0.000,0.087,6.030,307.660,0.000,206.875,1.171494,70.653,8,0.000,0.000,0.000,0,0.000,0,12.851,337340.801,271220.606,276692.347,291859.453,279727.765,310820.831,320546.152,301594.758 +1.702417,0.100148,3993571,0.119018,2185929.000,0.000,0.000,18.607,0.000,22.978,5.364,307.660,24.027,150.676,1.231133,70.639,8,0.000,0.000,0.000,0,0.000,0,18.607,297779.855,299377.493,243859.554,307435.583,252626.595,238936.830,244438.698,301474.394 +1.802573,0.100156,4150892,0.123706,1570752.228,0.000,0.000,82.439,0.000,42.077,9.111,307.660,102.184,151.170,1.330788,70.624,8,0.000,0.000,0.000,0,0.000,0,82.439,268200.344,197301.281,102369.821,248900.543,307718.510,109438.760,33986.820,302836.149 +1.902727,0.100153,4388151,0.130777,2368961.343,0.000,0.000,83.700,0.000,0.104,6.559,307.660,24.845,125.199,1.427853,70.612,8,0.000,0.000,0.000,0,0.000,0,83.700,314757.870,188940.590,323933.814,290554.942,319221.033,314967.548,313280.133,303305.412 +2.002873,0.100146,4654048,0.138701,2655081.023,0.000,0.000,10.684,0.000,0.199,5.116,307.660,0.000,144.164,1.498943,70.598,8,0.000,0.000,0.000,0,0.000,0,10.684,335139.112,334939.404,339163.217,307259.947,336207.547,335638.380,332512.958,334220.457 +2.103017,0.100144,4892177,0.145798,2377866.680,0.000,0.000,13.958,0.000,0.095,5.083,307.660,0.000,455.633,1.566139,70.553,8,0.000,0.000,0.000,0,0.000,0,13.958,294426.126,279108.179,263061.281,314966.555,313239.042,302644.295,297621.526,312799.675 +2.203163,0.100146,5136732,0.153087,2441982.630,0.000,0.000,14.884,0.000,1.842,5.248,307.660,0.000,416.189,1.655655,70.512,8,0.000,0.000,0.000,0,0.000,0,14.884,318474.756,277095.205,267099.807,304984.463,327561.482,320721.474,311135.478,314909.964 +2.303309,0.100146,5377896,0.160274,2408135.152,0.000,0.000,15.272,0.000,0.070,5.746,307.660,0.039,363.299,1.742668,70.477,8,0.000,0.000,0.000,0,0.000,0,15.272,240759.594,328082.502,304826.350,282269.180,320872.995,307073.080,318466.497,305784.954 +2.403456,0.100147,5632413,0.167859,2541434.270,0.000,0.000,13.784,0.000,0.108,5.320,307.660,0.039,211.018,1.814381,70.456,8,0.000,0.000,0.000,0,0.000,0,13.784,334118.869,330733.844,304432.506,261575.502,330703.888,313009.897,336565.273,330294.490 +2.503587,0.100132,5818600,0.173408,1859419.490,0.000,0.000,24.378,0.000,56.529,5.953,307.660,65.461,173.131,1.888203,70.439,8,0.000,0.000,0.000,0,0.000,0,24.378,295920.010,139975.528,300873.482,139675.922,141893.001,303160.468,242310.661,295610.418 +2.603752,0.100165,5983099,0.178310,1642288.370,0.000,0.000,89.955,0.000,71.561,8.048,307.660,129.514,130.059,1.992992,70.426,8,0.000,0.000,0.000,0,0.000,0,89.955,284861.392,175830.751,288036.169,188849.335,149973.288,95133.502,229103.116,230500.817 +2.703896,0.100145,6234741,0.185810,2512788.593,0.000,0.000,89.557,0.000,0.077,6.021,307.660,21.414,132.894,2.070530,70.413,8,0.000,0.000,0.000,0,0.000,0,89.557,325689.324,317740.811,322843.436,331321.185,330302.657,272835.706,294953.742,317101.734 +2.804028,0.100131,6486823,0.193322,2517511.713,0.000,0.000,10.058,0.000,1.788,5.190,307.660,0.000,283.417,2.141766,70.386,8,0.000,0.000,0.000,0,0.000,0,10.058,293574.211,322955.601,316374.249,318341.664,305957.938,313687.780,325951.664,320668.606 +2.904175,0.100148,6726352,0.200461,2391761.387,0.000,0.000,15.043,0.000,3.413,5.224,307.660,0.000,522.432,2.243080,70.335,8,0.000,0.000,0.000,0,0.000,0,15.043,338260.957,288025.071,296222.976,274315.297,308295.166,294445.599,278928.491,313267.830 +3.004312,0.100137,6992435,0.208391,2657200.397,0.000,0.000,12.168,0.000,0.102,5.293,307.660,0.000,367.116,2.317726,70.299,8,0.000,0.000,0.000,0,0.000,0,12.168,338957.002,321900.300,348074.548,299291.183,348633.784,325585.267,342152.637,332605.677 +3.104452,0.100140,7260956,0.216393,2681446.242,0.000,0.000,13.086,0.000,0.151,5.383,307.660,0.000,251.678,2.389786,70.274,8,0.000,0.000,0.000,0,0.000,0,13.086,329976.835,347022.908,361203.005,339183.911,305930.587,323276.240,336228.060,338624.696 +3.204591,0.100139,7505132,0.223670,2438378.700,0.000,0.000,13.780,0.000,19.449,5.238,307.660,1.521,206.354,2.470005,70.254,8,0.000,0.000,0.000,0,0.000,0,13.780,330321.943,304457.808,270694.628,332958.287,295789.828,304457.808,281549.575,318148.823 +3.304739,0.100148,7659347,0.228266,1539871.883,0.000,0.000,81.953,0.000,20.134,7.974,307.660,121.734,155.044,2.560226,70.239,8,0.000,0.000,0.000,0,0.000,0,81.953,313186.665,311309.443,257858.519,40220.497,2106.883,40639.877,310340.876,264209.124 +3.404887,0.100148,7858784,0.234210,1991427.765,0.000,0.000,91.109,0.000,0.047,7.429,307.660,105.664,148.336,2.648403,70.224,8,0.000,0.000,0.000,0,0.000,0,91.109,299367.699,293945.710,83506.623,248672.599,234144.064,286127.261,290311.080,255352.729 +3.505048,0.100162,8112775,0.241780,2535811.014,0.000,0.000,11.206,0.000,0.318,5.673,307.660,0.000,242.772,2.755836,70.200,8,0.000,0.000,0.000,0,0.000,0,11.206,273358.133,327201.099,319164.090,328369.211,333201.400,338273.202,304467.846,311776.033 +3.605181,0.100133,8360200,0.249153,2470967.739,0.000,0.000,16.433,0.000,0.177,4.964,307.660,0.000,411.914,2.824760,70.160,8,0.000,0.000,0.000,0,0.000,0,16.433,316189.995,313923.007,327684.727,308909.666,310857.079,320664.052,262351.511,310387.703 +3.705328,0.100147,8605660,0.256469,2450997.524,0.000,0.000,15.410,0.000,0.209,5.226,307.660,0.000,346.288,2.893204,70.126,8,0.000,0.000,0.000,0,0.000,0,15.410,311961.479,314937.105,331472.801,293917.999,278101.247,301476.889,303773.514,315356.489 +3.805482,0.100154,8855947,0.263928,2499031.587,0.000,0.000,14.474,0.000,2.275,5.155,307.660,0.000,349.775,2.965168,70.092,8,0.000,0.000,0.000,0,0.000,0,14.474,326548.435,311801.086,274997.615,315665.151,319928.602,306629.030,336103.758,307357.911 +3.905639,0.100157,9091296,0.270942,2349806.115,0.000,0.000,11.663,0.000,0.058,5.741,307.660,0.000,270.864,3.071556,70.066,8,0.000,0.000,0.000,0,0.000,0,11.663,288357.930,291842.467,313568.406,318271.034,259972.431,288477.742,312080.739,277235.367 +4.005790,0.100151,9311177,0.277495,2195485.245,0.000,0.000,17.778,0.000,24.773,4.508,307.660,22.778,132.222,3.129276,70.053,8,0.000,0.000,0.000,0,0.000,0,17.778,252487.643,303470.437,299546.379,253605.949,292696.752,243501.251,249452.240,300724.595 +4.105930,0.100140,9465957,0.282108,1545629.966,0.000,0.000,84.209,0.000,43.888,8.839,307.660,165.042,173.545,3.215137,70.036,8,0.000,0.000,0.000,0,0.000,0,84.209,51727.376,206040.723,205521.452,113540.591,214998.147,325902.439,107908.499,319990.739 +4.206083,0.100153,9674281,0.288316,2080057.200,0.000,0.000,90.346,0.000,0.696,7.165,307.660,49.690,144.700,3.319000,70.022,8,0.000,0.000,0.000,0,0.000,0,90.346,302197.592,167863.144,180933.145,193663.666,336964.394,338152.576,226992.667,333290.016 +4.306228,0.100145,9919182,0.295615,2445466.910,0.000,0.000,17.314,0.000,0.118,5.268,307.660,0.000,292.662,3.399254,69.993,8,0.000,0.000,0.000,0,0.000,0,17.314,320206.073,318478.576,316261.787,311269.021,266014.587,303929.655,294653.095,314654.117 +4.406355,0.100126,10164081,0.302913,2445901.878,0.000,0.000,14.353,0.000,0.082,4.901,307.660,0.000,452.085,3.467793,69.949,8,0.000,0.000,0.000,0,0.000,0,14.353,307941.203,257395.021,304735.250,310208.340,312265.743,320455.403,320944.785,311956.134 +4.506499,0.100145,10422997,0.310630,2585419.854,0.000,0.000,12.850,0.000,0.083,5.192,307.660,0.000,314.623,3.537777,69.918,8,0.000,0.000,0.000,0,0.000,0,12.850,329563.244,317690.420,341296.271,320027.039,295192.965,320995.638,316452.211,344202.067 +4.606658,0.100159,10674831,0.318135,2514352.112,0.000,0.000,13.291,0.000,0.148,5.630,307.660,0.000,368.517,3.644642,69.882,8,0.000,0.000,0.000,0,0.000,0,13.291,326861.581,330955.089,328628.778,241866.388,326562.056,325823.228,343914.534,289740.457 +4.706801,0.100144,10937858,0.325974,2626497.955,0.000,0.000,9.828,0.000,0.089,5.491,307.660,0.000,256.741,3.719723,69.857,8,0.000,0.000,0.000,0,0.000,0,9.828,333700.756,311981.947,322556.760,339033.098,294037.718,351734.856,339292.725,334160.096 +4.806949,0.100147,11143742,0.332109,2055814.971,0.000,0.000,15.820,0.000,56.376,6.684,307.660,60.302,186.717,3.798325,69.838,8,0.000,0.000,0.000,0,0.000,0,15.820,320698.109,149420.136,160104.414,330833.195,330353.901,330293.989,103038.384,331072.843 +4.907116,0.100168,11279463,0.336154,1354938.865,0.000,0.000,84.918,0.000,38.887,8.386,308.582,173.225,150.529,3.899437,69.824,8,0.000,0.000,0.000,0,0.000,0,84.918,138877.215,121725.964,195632.083,205305.868,132238.343,216267.495,197249.373,147642.523 +5.007264,0.100147,11499655,0.342716,2198678.862,0.000,0.000,93.918,0.000,0.202,6.461,308.738,65.450,190.656,3.977310,69.805,8,0.000,0.000,0.000,0,0.000,0,93.918,304431.229,276302.694,318820.018,121450.965,305170.140,261025.215,304920.508,306558.094 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.log new file mode 100644 index 000000000..5d2960ec1 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.log @@ -0,0 +1,14 @@ +version=splinterdb_build_version 7bb5fab3 records=100000000 key=16 value=16 cache_mib=256 disk_gib=16 inserters=8 readers=0 memtable_bg=1 normal_bg=3 memtable_mib=8 queue_scale=100 log=1 checkpoints=disabled direct=1 query_range=inserted seed=42 report_ms=100 barrier_ms=0 target_insert_ops_s=0 id_base=0 mode=create +Splinter task system created 1 background thread of type 'TASK_TYPE_MEMTABLE'. +Splinter task system created 3 background threads of type 'TASK_TYPE_NORMAL'. +worker=4 kind=insert tid=5 os_tid=353 +worker=5 kind=insert tid=6 os_tid=354 +worker=0 kind=insert tid=7 os_tid=349 +worker=7 kind=insert tid=8 os_tid=356 +worker=6 kind=insert tid=9 os_tid=355 +worker=2 kind=insert tid=12 os_tid=351 +start_monotonic_ns=1819385337434145 main_address=0x558b032b8a80 +worker=3 kind=insert tid=11 os_tid=352 +worker=1 kind=insert tid=10 os_tid=350 +load_end reason=time_limit inserted=11499824 elapsed_s=5.008 error=0 barrier_count=0 barrier_avg_ms=0.000 barrier_max_ms=0.000 +close_s=0.179 close_rc=0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.summary.csv new file mode 100644 index 000000000..de9393521 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert/wal1-24-main.summary.csv @@ -0,0 +1,2 @@ +target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted,insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms,insert_service_p999_ms,insert_service_max_ms,insert_response_mean_ms,insert_response_p50_ms,insert_response_p99_ms,insert_response_p999_ms,insert_response_max_ms,query_service_mean_ms,query_service_p99_ms,query_service_max_ms +0,2296488.128709,5.007570,11499824,0.003426,0.001392,0.064000,0.202752,117.313900,0.003426,0.001392,0.064000,0.202752,117.313900,0.000000,0.000000,0.000000 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert_benchmark.c b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert_benchmark.c new file mode 100644 index 000000000..7da615cbe --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/insert_benchmark.c @@ -0,0 +1,1002 @@ +// Copyright 2026 VMware, Inc. +// SPDX-License-Identifier: Apache-2.0 + +/* Concurrent random insertion and point queries, sampled independently. */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "splinterdb/default_data_config.h" +#include "splinterdb/splinterdb.h" +#ifndef KVBENCH_NO_TRACE +# include "platform_trace.h" +#endif +#include "platform_threads.h" + +#define GIB (UINT64_C(1) << 30) +#define MAX_WORKERS 64 +#define MAX_RECORD_PART 1024 +#define HIST_SUB_BITS 5 +#define HIST_SUB (1u << HIST_SUB_BITS) +#define HIST_BUCKETS (64 * HIST_SUB) + +typedef struct latency_histogram { + uint64_t buckets[HIST_BUCKETS]; + uint64_t count; + uint64_t sum_ns; + uint64_t max_ns; +} latency_histogram; + +typedef struct options { + const char *filename; + const char *csv; + const char *summary; + const char *trace; + uint64_t records, data_gib, cache_mib, disk_gib, seed, id_base; + uint64_t inserts, queries, memtable_bg, normal_bg, memtable_mib; + uint64_t key_size, value_size, report_ms, seconds, min_free_gib; + uint64_t queue_scale, barrier_ms, target_insert_ops_s; + uint64_t log_group_size_pages; + bool full_query_range, direct, use_log, open_existing; +} options; + +typedef struct benchmark benchmark; +typedef struct worker { + _Alignas(64) atomic_uint_fast64_t completed; + atomic_uint_fast64_t hits, max_ns, response_max_ns, active_since_ns; + atomic_bool done; + pthread_t thread; + benchmark *bench; + uint64_t index; + bool query; + latency_histogram service_latency; + latency_histogram response_latency; +} worker; + +struct benchmark { + options opt; + splinterdb *db; + atomic_bool start, stop; + uint64_t start_ns; + atomic_uint_fast64_t arrival_sequence; + atomic_int error; + worker workers[MAX_WORKERS]; + pthread_t barrier_thread; + atomic_bool barrier_started, barrier_done; + atomic_uint_fast64_t barrier_count, barrier_sum_ns, barrier_max_ns; +}; + +static volatile sig_atomic_t interrupted; + +static void +handle_signal(int signo) +{ + interrupted = signo; +} + +static uint64_t +now_ns(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000 + ts.tv_nsec; +} + +/* A bijection on 64-bit integers, so the random insertion keys are unique. */ +static uint64_t +mix(uint64_t x) +{ + x = (x ^ (x >> 30)) * UINT64_C(0xbf58476d1ce4e5b9); + x = (x ^ (x >> 27)) * UINT64_C(0x94d049bb133111eb); + return x ^ (x >> 31); +} + +static uint64_t +next_random(uint64_t *state) +{ + *state += UINT64_C(0x9e3779b97f4a7c15); + return mix(*state); +} + +static void +make_key(const options *opt, uint64_t id, char *key) +{ + uint64_t x = mix(id + opt->seed); + for (uint64_t i = 0; i < opt->key_size; i++) { + key[i] = (char)(x >> (56 - 8 * (i % 8))); + } +} + +static void +record_max(atomic_uint_fast64_t *slot, uint64_t ns) +{ + uint64_t old = atomic_load_explicit(slot, memory_order_relaxed); + while (old < ns + && !atomic_compare_exchange_weak_explicit( + slot, &old, ns, memory_order_relaxed, memory_order_relaxed)) + { + } +} + +static void +record_latency(worker *w, uint64_t ns) +{ + record_max(&w->max_ns, ns); +} + +static unsigned +histogram_index(uint64_t value) +{ + if (value < HIST_SUB) { + return (unsigned)value; + } + unsigned exponent = 63 - __builtin_clzll(value); + unsigned sub = + (unsigned)((value >> (exponent - HIST_SUB_BITS)) & (HIST_SUB - 1)); + return (exponent - HIST_SUB_BITS + 1) * HIST_SUB + sub; +} + +static uint64_t +histogram_value(unsigned index) +{ + if (index < HIST_SUB) { + return index; + } + unsigned exponent = index / HIST_SUB + HIST_SUB_BITS - 1; + unsigned sub = index % HIST_SUB; + uint64_t low = + (UINT64_C(1) << exponent) | ((uint64_t)sub << (exponent - HIST_SUB_BITS)); + uint64_t width = UINT64_C(1) << (exponent - HIST_SUB_BITS); + return low + width / 2; +} + +static void +histogram_add(latency_histogram *histogram, uint64_t value) +{ + histogram->buckets[histogram_index(value)]++; + histogram->count++; + histogram->sum_ns += value; + histogram->max_ns = MAX(histogram->max_ns, value); +} + +static void +histogram_merge(latency_histogram *result, const latency_histogram *source) +{ + for (unsigned i = 0; i < HIST_BUCKETS; i++) { + result->buckets[i] += source->buckets[i]; + } + result->count += source->count; + result->sum_ns += source->sum_ns; + result->max_ns = MAX(result->max_ns, source->max_ns); +} + +static double +histogram_percentile_ms(const latency_histogram *histogram, double percentile) +{ + if (histogram->count == 0) { + return 0; + } + uint64_t target = (uint64_t)(percentile * histogram->count); + if (target == 0) { + target = 1; + } + uint64_t cumulative = 0; + for (unsigned i = 0; i < HIST_BUCKETS; i++) { + cumulative += histogram->buckets[i]; + if (cumulative >= target) { + return histogram_value(i) / 1e6; + } + } + return histogram->max_ns / 1e6; +} + +static void +wait_until(benchmark *b, uint64_t deadline) +{ + while (!atomic_load_explicit(&b->stop, memory_order_relaxed)) { + uint64_t now = now_ns(); + if (now >= deadline) { + return; + } + uint64_t remaining = deadline - now; + if (remaining > UINT64_C(10000000)) { + remaining = UINT64_C(10000000); + } + struct timespec delay = {.tv_sec = remaining / UINT64_C(1000000000), + .tv_nsec = remaining % UINT64_C(1000000000)}; + nanosleep(&delay, NULL); + } +} + +static void * +run_worker(void *arg) +{ + worker *w = arg; + benchmark *b = w->bench; + const options *o = &b->opt; + char key[MAX_RECORD_PART], value[MAX_RECORD_PART]; + memset(value, 'v', o->value_size); + int rc = platform_register_thread(); + if (rc) { + atomic_store(&b->error, rc); + atomic_store(&b->stop, true); + atomic_store(&w->done, true); + return NULL; + } + splinterdb_lookup_result result; + fprintf(stderr, + "worker=%lu kind=%s tid=%lu os_tid=%d\n", + w->index, + w->query ? "query" : "insert", + platform_get_tid(), + gettid()); + if (w->query) { + splinterdb_lookup_result_init( + b->db, &result, SPLINTERDB_LOOKUP_VALUE, sizeof(value), value); + } + while (!atomic_load_explicit(&b->start, memory_order_acquire)) { + struct timespec delay = {.tv_nsec = 1000000}; + nanosleep(&delay, NULL); + } + uint64_t completed = 0, hits = 0; + uint64_t random = o->seed + w->index + 1; + while (!atomic_load_explicit(&b->stop, memory_order_relaxed)) { + uint64_t scheduled = 0; + uint64_t arrival = UINT64_MAX; + if (!w->query && o->target_insert_ops_s != 0) { + arrival = atomic_fetch_add_explicit( + &b->arrival_sequence, 1, memory_order_relaxed); + if (arrival >= o->records) { + break; + } + scheduled = b->start_ns + + (uint64_t)((__uint128_t)arrival * UINT64_C(1000000000) + / o->target_insert_ops_s); + wait_until(b, scheduled); + if (atomic_load_explicit(&b->stop, memory_order_relaxed)) { + break; + } + } + uint64_t id; + if (!w->query) { + uint64_t offset = o->target_insert_ops_s != 0 + ? arrival + : completed * o->inserts + w->index; + if (offset >= o->records) { + break; + } + id = o->id_base + offset; + } else if (o->full_query_range) { + id = o->id_base + next_random(&random) % o->records; + } else { + uint64_t source = next_random(&random) % o->inserts; + uint64_t available = atomic_load_explicit( + &b->workers[source].completed, memory_order_acquire); + if (!available) { + struct timespec delay = {.tv_nsec = 100000}; + nanosleep(&delay, NULL); + continue; + } + id = o->id_base + (next_random(&random) % available) * o->inserts + + source; + } + make_key(o, id, key); + uint64_t start = now_ns(); + atomic_store_explicit(&w->active_since_ns, start, memory_order_relaxed); + if (w->query) { + rc = splinterdb_lookup(b->db, slice_create(o->key_size, key), &result); + if (!rc && splinterdb_lookup_found(&result)) { + hits++; + } + } else { + rc = splinterdb_insert(b->db, + slice_create(o->key_size, key), + slice_create(o->value_size, value), + NULL); + } + uint64_t end = now_ns(); + record_latency(w, end - start); + histogram_add(&w->service_latency, end - start); + if (!w->query) { + uint64_t response = scheduled == 0 ? end - start : end - scheduled; + histogram_add(&w->response_latency, response); + record_max(&w->response_max_ns, response); + } + atomic_store_explicit(&w->active_since_ns, 0, memory_order_relaxed); + if (rc) { + fprintf(stderr, "worker %lu: operation failed: %d\n", w->index, rc); + atomic_store(&b->error, rc); + atomic_store(&b->stop, true); + break; + } + completed++; + atomic_store_explicit(&w->hits, hits, memory_order_relaxed); + atomic_store_explicit(&w->completed, completed, memory_order_release); + } + if (w->query) { + splinterdb_lookup_result_deinit(&result); + } + platform_deregister_thread(); + atomic_store(&w->done, true); + return NULL; +} + +/* + * Dedicated thread that calls splinterdb_durable_barrier() on a fixed period, + * independent of insert/query progress. If a barrier call takes longer than + * the period, the next call follows immediately (no attempt to catch up on + * skipped slots). + */ +static void * +run_barrier(void *arg) +{ + benchmark *b = arg; + const options *o = &b->opt; + int rc = platform_register_thread(); + if (rc) { + atomic_store(&b->error, rc); + atomic_store(&b->stop, true); + atomic_store(&b->barrier_done, true); + return NULL; + } + while (!atomic_load_explicit(&b->start, memory_order_acquire)) { + struct timespec delay = {.tv_nsec = 1000000}; + nanosleep(&delay, NULL); + } + uint64_t period_ns = o->barrier_ms * UINT64_C(1000000); + uint64_t next = now_ns() + period_ns; + while (!atomic_load_explicit(&b->stop, memory_order_relaxed)) { + uint64_t now = now_ns(); + if (now < next) { + uint64_t remaining = next - now; + struct timespec delay = {.tv_sec = remaining / 1000000000, + .tv_nsec = remaining % 1000000000}; + nanosleep(&delay, NULL); + continue; + } + uint64_t start = now_ns(); +#ifdef KVBENCH_NO_DURABLE_BARRIER + rc = ENOTSUP; +#else + #ifdef HAVE_SENTINEL_NOTIFICATION + rc = splinterdb_durable_barrier(b->db, NULL); +#else + rc = splinterdb_durable_barrier(b->db); +#endif +#endif + uint64_t elapsed = now_ns() - start; + if (rc) { + fprintf(stderr, "barrier: durable_barrier failed: %d\n", rc); + atomic_store(&b->error, rc); + atomic_store(&b->stop, true); + break; + } + atomic_fetch_add_explicit(&b->barrier_count, 1, memory_order_relaxed); + atomic_fetch_add_explicit( + &b->barrier_sum_ns, elapsed, memory_order_relaxed); + record_max(&b->barrier_max_ns, elapsed); + next += period_ns; + } + platform_deregister_thread(); + atomic_store(&b->barrier_done, true); + return NULL; +} + +static void +usage(const char *name) +{ + fprintf( + stderr, + "Usage: %s --db NEW_IMAGE --csv NEW_CSV [options]\n" + " --open open an existing image instead of creating one\n" + " --id-base N first logical id inserted (default 0)\n" + " --data-gib N logical key+value GiB (default 60)\n" + " --records N override logical data target\n" + " --cache-mib N default 3072\n" + " --disk-gib N image capacity (default 512; grows lazily)\n" + " --inserters N default 8\n" + " --readers N default 8\n" + " --memtable-bg N default 1\n" + " --normal-bg N default 4\n" + " --memtable-mib N default 24\n" + " --key-size N default 16, minimum 8\n" + " --value-size N default 16\n" + " --query-range inserted|all (default inserted)\n" + " --queue-scale N foreground task threshold (default 100)\n" + " --report-ms N default 1000\n" + " --seconds N optional time limit (0 = no limit)\n" + " --target-insert-ops-s N aggregate open-loop arrival rate; " + "requires --seconds (default 0 = closed loop)\n" + " --summary NEW_CSV write aggregate latency percentiles\n" + " --min-free-gib N stop before filesystem fills (default 8)\n" + " --seed N default 1\n" + " --barrier-ms N call splinterdb_durable_barrier from a " + "dedicated thread every N ms (default 0 = disabled)\n" + " --log-group-size-pages N automatic durability threshold " + "(0 = default, UINT64_MAX = disabled)\n" + " --trace NEW_CSV trace internal tasks and waits (optional)\n" + " --direct use O_DIRECT (default buffered I/O)\n" + " --no-log disable WAL (default enabled)\n" + "Automatic checkpoints are disabled. Existing files are refused.\n" + "Load measurements exclude create, final drain and close.\n", + name); +} + +static bool +parse_number(const char *s, uint64_t *out) +{ + char *end; + errno = 0; + unsigned long long value = strtoull(s, &end, 10); + if (errno || s == end || *end || *s == '-') { + return false; + } + *out = value; + return true; +} + +static bool +parse_options(int argc, char **argv, options *o) +{ + *o = (options){.data_gib = 60, + .cache_mib = 3072, + .disk_gib = 512, + .seed = 1, + .inserts = 8, + .queries = 8, + .memtable_bg = 1, + .normal_bg = 4, + .memtable_mib = 24, + .key_size = 16, + .value_size = 16, + .report_ms = 1000, + .min_free_gib = 8, + .queue_scale = 100, + .use_log = true}; + for (int i = 1; i < argc; i++) { + if (!strcmp(argv[i], "--direct")) { + o->direct = true; + continue; + } + if (!strcmp(argv[i], "--no-log")) { + o->use_log = false; + continue; + } + if (!strcmp(argv[i], "--open")) { + o->open_existing = true; + continue; + } + if (i + 1 >= argc) { + return false; + } + const char *name = argv[i++], *value = argv[i]; + if (!strcmp(name, "--db")) { + o->filename = value; + continue; + } + if (!strcmp(name, "--csv")) { + o->csv = value; + continue; + } + if (!strcmp(name, "--summary")) { + o->summary = value; + continue; + } + if (!strcmp(name, "--trace")) { + o->trace = value; + continue; + } + if (!strcmp(name, "--query-range")) { + if (strcmp(value, "inserted") && strcmp(value, "all")) { + return false; + } + o->full_query_range = !strcmp(value, "all"); + continue; + } + struct { + const char *name; + uint64_t *value; + } numbers[] = { + {"--data-gib", &o->data_gib}, + {"--records", &o->records}, + {"--cache-mib", &o->cache_mib}, + {"--disk-gib", &o->disk_gib}, + {"--inserters", &o->inserts}, + {"--readers", &o->queries}, + {"--memtable-bg", &o->memtable_bg}, + {"--normal-bg", &o->normal_bg}, + {"--memtable-mib", &o->memtable_mib}, + {"--key-size", &o->key_size}, + {"--value-size", &o->value_size}, + {"--report-ms", &o->report_ms}, + {"--seconds", &o->seconds}, + {"--min-free-gib", &o->min_free_gib}, + {"--seed", &o->seed}, + {"--id-base", &o->id_base}, + {"--queue-scale", &o->queue_scale}, + {"--barrier-ms", &o->barrier_ms}, + {"--log-group-size-pages", &o->log_group_size_pages}, + {"--target-insert-ops-s", &o->target_insert_ops_s}, + }; + bool found = false; + for (size_t j = 0; j < sizeof(numbers) / sizeof(numbers[0]); j++) { + if (!strcmp(name, numbers[j].name)) { + if (!parse_number(value, numbers[j].value)) { + return false; + } + found = true; + break; + } + } + if (!found) { + return false; + } + } + if (!o->filename || !o->csv || !o->inserts || o->inserts > MAX_WORKERS + || o->queries > MAX_WORKERS - o->inserts || o->key_size < 8 + || o->key_size > MAX_RECORD_PART || o->value_size > MAX_RECORD_PART + || !o->report_ms || o->report_ms > 60000 || !o->cache_mib + || o->cache_mib > UINT64_MAX / (1 << 20) || !o->disk_gib + || o->disk_gib > INT64_MAX / GIB || o->data_gib > UINT64_MAX / GIB + || o->min_free_gib > UINT64_MAX / GIB || !o->memtable_mib + || o->memtable_mib > UINT64_MAX / (1 << 20) || o->memtable_bg > 32 + || o->normal_bg > 32 + || o->inserts + o->queries + o->memtable_bg + o->normal_bg + 1 + > MAX_WORKERS + || o->seconds > UINT64_MAX / 1000000000 + || o->barrier_ms > UINT64_MAX / 1000000 + || (o->target_insert_ops_s != 0 && (o->seconds == 0 || o->queries != 0))) + { + return false; + } +#ifdef KVBENCH_NO_DURABLE_BARRIER + if (o->barrier_ms != 0) { + return false; + } +#endif +#ifdef KVBENCH_NO_TRACE + if (o->trace != NULL) { + return false; + } +#endif + if (!o->records) { + o->records = o->data_gib * GIB / (o->key_size + o->value_size); + } + return o->records > 0 && o->records <= UINT64_MAX / MAX_WORKERS + && o->records <= UINT64_MAX / (o->key_size + o->value_size) + && o->id_base <= UINT64_MAX - o->records; +} + +static void +process_io(uint64_t *reads, uint64_t *writes) +{ + *reads = *writes = 0; + FILE *file = fopen("/proc/self/io", "r"); + if (!file) { + return; + } + char name[64]; + uint64_t value; + while (fscanf(file, "%63s %" SCNu64, name, &value) == 2) { + if (!strcmp(name, "read_bytes:")) { + *reads = value; + } + if (!strcmp(name, "write_bytes:")) { + *writes = value; + } + } + fclose(file); +} + +static int +write_summary(const benchmark *b, uint64_t inserted, double elapsed_s) +{ + const options *o = &b->opt; + if (o->summary == NULL) { + return 0; + } + + latency_histogram insert_service = {0}, insert_response = {0}; + latency_histogram query_service = {0}; + for (uint64_t i = 0; i < o->inserts + o->queries; i++) { + if (b->workers[i].query) { + histogram_merge(&query_service, &b->workers[i].service_latency); + } else { + histogram_merge(&insert_service, &b->workers[i].service_latency); + histogram_merge(&insert_response, &b->workers[i].response_latency); + } + } + + FILE *summary = fopen(o->summary, "wx"); + if (summary == NULL) { + return errno; + } + fprintf(summary, + "target_insert_ops_s,achieved_insert_ops_s,elapsed_s,inserted," + "insert_service_mean_ms,insert_service_p50_ms,insert_service_p99_ms," + "insert_service_p999_ms,insert_service_max_ms," + "insert_response_mean_ms,insert_response_p50_ms," + "insert_response_p99_ms,insert_response_p999_ms," + "insert_response_max_ms,query_service_mean_ms,query_service_p99_ms," + "query_service_max_ms\n"); + fprintf( + summary, + "%lu,%.6f,%.6f,%lu,%.6f,%.6f,%.6f,%.6f,%.6f," + "%.6f,%.6f,%.6f,%.6f,%.6f,%.6f,%.6f,%.6f\n", + o->target_insert_ops_s, + elapsed_s == 0 ? 0 : inserted / elapsed_s, + elapsed_s, + inserted, + insert_service.count ? insert_service.sum_ns / 1e6 / insert_service.count + : 0, + histogram_percentile_ms(&insert_service, 0.50), + histogram_percentile_ms(&insert_service, 0.99), + histogram_percentile_ms(&insert_service, 0.999), + insert_service.max_ns / 1e6, + insert_response.count + ? insert_response.sum_ns / 1e6 / insert_response.count + : 0, + histogram_percentile_ms(&insert_response, 0.50), + histogram_percentile_ms(&insert_response, 0.99), + histogram_percentile_ms(&insert_response, 0.999), + insert_response.max_ns / 1e6, + query_service.count ? query_service.sum_ns / 1e6 / query_service.count + : 0, + histogram_percentile_ms(&query_service, 0.99), + query_service.max_ns / 1e6); + int rc = ferror(summary) ? EIO : 0; + if (fclose(summary) != 0 && rc == 0) { + rc = errno; + } + return rc; +} + +int +main(int argc, char **argv) +{ + benchmark b = {0}; + if (argc == 2 && !strcmp(argv[1], "--help")) { + usage(argv[0]); + return 0; + } + if (!parse_options(argc, argv, &b.opt)) { + usage(argv[0]); + return 2; + } + const options *o = &b.opt; + if (o->trace) { +#ifdef KVBENCH_NO_TRACE + return 2; +#else + int trace_rc = platform_trace_open(o->trace); + if (trace_rc) { + fprintf(stderr, "trace open failed: %s\n", strerror(trace_rc)); + return 1; + } +#endif + } + FILE *csv = fopen(o->csv, "wx"); + if (!csv) { + perror(o->csv); + return 1; + } + int fd = open(o->filename, + O_RDWR | O_CLOEXEC | (o->open_existing ? 0 : O_CREAT | O_EXCL), + 0600); + if (fd < 0) { + perror(o->filename); + fclose(csv); + return 1; + } + struct statvfs space; + if (fstatvfs(fd, &space) != 0 + || (uint64_t)space.f_bavail * space.f_frsize < o->min_free_gib * GIB) + { + fprintf(stderr, "Insufficient free space\n"); + close(fd); + fclose(csv); + return 1; + } + + data_config data_cfg; + default_data_config_init(&data_cfg); + splinterdb_config cfg = {.filename = o->filename, + .disk_size = o->disk_gib * GIB, + .cache_size = o->cache_mib * (1 << 20), + .data_cfg = &data_cfg, + .io_flags = O_RDWR | (o->direct ? O_DIRECT : 0), + .num_memtable_bg_threads = o->memtable_bg, + .num_normal_bg_threads = o->normal_bg, + .memtable_capacity = o->memtable_mib * (1 << 20), + .queue_scale_percent = o->queue_scale, + .use_log = o->use_log, + .checkpoint_log_size_bytes = UINT64_MAX, + .use_stats = true}; +#ifndef KVBENCH_NO_LOG_GROUP_LIMIT + cfg.log_group_size_pages = o->log_group_size_pages; +#else + if (o->log_group_size_pages != 0) { + fprintf(stderr, "This build has no automatic log-group limit\n"); + return 1; + } +#endif + platform_set_log_streams(stderr, stderr); + fprintf( + stderr, + "version=%s records=%lu key=%lu value=%lu cache_mib=%lu " + "disk_gib=%lu inserters=%lu readers=%lu memtable_bg=%lu normal_bg=%lu " + "memtable_mib=%lu queue_scale=%lu log=%d checkpoints=disabled " + "direct=%d query_range=%s seed=%lu report_ms=%lu barrier_ms=%lu " + "target_insert_ops_s=%lu id_base=%lu mode=%s\n", + splinterdb_get_version(), + o->records, + o->key_size, + o->value_size, + o->cache_mib, + o->disk_gib, + o->inserts, + o->queries, + o->memtable_bg, + o->normal_bg, + o->memtable_mib, + o->queue_scale, + o->use_log, + o->direct, + o->full_query_range ? "all" : "inserted", + o->seed, + o->report_ms, + o->barrier_ms, + o->target_insert_ops_s, + o->id_base, + o->open_existing ? "open" : "create"); + int rc = o->open_existing ? splinterdb_open(&cfg, &b.db) + : splinterdb_create(&cfg, &b.db); + if (rc) { + fprintf( + stderr, "%s failed: %d\n", o->open_existing ? "open" : "create", rc); + close(fd); + fclose(csv); + return 1; + } + + signal(SIGINT, handle_signal); + signal(SIGTERM, handle_signal); + uint64_t started = 0; + for (uint64_t i = 0; i < o->inserts + o->queries; i++) { + worker *w = &b.workers[i]; + w->bench = &b; + w->index = i; + w->query = i >= o->inserts; + rc = pthread_create(&w->thread, NULL, run_worker, w); + if (rc) { + atomic_store(&b.error, rc); + atomic_store(&b.stop, true); + break; + } + started++; + } + if (o->barrier_ms && !atomic_load(&b.stop)) { + rc = pthread_create(&b.barrier_thread, NULL, run_barrier, &b); + if (rc) { + atomic_store(&b.error, rc); + atomic_store(&b.stop, true); + } else { + atomic_store(&b.barrier_started, true); + } + } + fprintf( + csv, + "elapsed_s,interval_s,inserted,logical_gib,insert_ops_s,query_ops_s," + "hit_pct,insert_max_ms,query_max_ms,active_max_ms,cpu_cores,max_rss_mib," + "read_mib_s,write_mib_s,image_gib,free_gib,inserters_running," + "barrier_ops_s,barrier_avg_ms,barrier_max_ms,target_insert_ops_s," + "offered_insert_ops_s,insert_backlog,response_max_ms"); + for (uint64_t i = 0; i < started; i++) { + fprintf(csv, ",%s%lu_ops_s", b.workers[i].query ? "q" : "i", i); + } + fputc('\n', csv); + fflush(csv); + uint64_t prev_completed[MAX_WORKERS] = {0}, prev_hits = 0; + uint64_t prev_barrier_count = 0, prev_barrier_sum_ns = 0; + uint64_t prev_read, prev_write; + process_io(&prev_read, &prev_write); + struct rusage usage_now; + getrusage(RUSAGE_SELF, &usage_now); + double prev_cpu = + usage_now.ru_utime.tv_sec + usage_now.ru_utime.tv_usec / 1e6 + + usage_now.ru_stime.tv_sec + usage_now.ru_stime.tv_usec / 1e6; + uint64_t start = now_ns(), previous = start; + b.start_ns = start; + fprintf( + stderr, "start_monotonic_ns=%lu main_address=%p\n", start, (void *)main); + atomic_store_explicit(&b.start, true, memory_order_release); + const char *reason = "complete"; + uint64_t inserted = 0; + while (true) { + struct timespec delay = {.tv_sec = o->report_ms / 1000, + .tv_nsec = (o->report_ms % 1000) * 1000000}; + nanosleep(&delay, NULL); + uint64_t now = now_ns(), delta_insert = 0, delta_query = 0, hits = 0; + uint64_t insert_max = 0, query_max = 0, response_max = 0, active_max = 0; + uint64_t inserters_running = 0; + double rates[MAX_WORKERS] = {0}; + double elapsed = (now - start) / 1e9, interval = (now - previous) / 1e9; + bool all_inserts_done = true; + inserted = 0; + for (uint64_t i = 0; i < started; i++) { + worker *w = &b.workers[i]; + uint64_t completed = atomic_load(&w->completed); + uint64_t count = completed - prev_completed[i]; + prev_completed[i] = completed; + rates[i] = count / interval; + uint64_t latency = atomic_exchange(&w->max_ns, 0); + uint64_t active = atomic_load(&w->active_since_ns); + if (active && active < now && active_max < now - active) { + active_max = now - active; + } + if (w->query) { + delta_query += count; + hits += atomic_load(&w->hits); + if (query_max < latency) { + query_max = latency; + } + } else { + inserted += completed; + delta_insert += count; + if (insert_max < latency) { + insert_max = latency; + } + uint64_t response = atomic_exchange(&w->response_max_ns, 0); + if (response_max < response) { + response_max = response; + } + if (!atomic_load(&w->done)) { + inserters_running++; + all_inserts_done = false; + } + } + } + uint64_t reads, writes; + process_io(&reads, &writes); + getrusage(RUSAGE_SELF, &usage_now); + double cpu = usage_now.ru_utime.tv_sec + usage_now.ru_utime.tv_usec / 1e6 + + usage_now.ru_stime.tv_sec + + usage_now.ru_stime.tv_usec / 1e6; + struct stat st = {0}; + if (fstat(fd, &st) != 0 || fstatvfs(fd, &space) != 0) { + atomic_store(&b.error, errno); + atomic_store(&b.stop, true); + } + double free_gib = (double)space.f_bavail * space.f_frsize / GIB; + uint64_t barrier_count = atomic_load(&b.barrier_count); + uint64_t barrier_sum_ns = atomic_load(&b.barrier_sum_ns); + uint64_t barrier_max_ns = atomic_exchange(&b.barrier_max_ns, 0); + uint64_t delta_barrier_count = barrier_count - prev_barrier_count; + uint64_t delta_barrier_sum = barrier_sum_ns - prev_barrier_sum_ns; + uint64_t offered = inserted; + if (o->target_insert_ops_s != 0) { + __uint128_t possible = (__uint128_t)(now - start) + * o->target_insert_ops_s / UINT64_C(1000000000); + offered = possible > o->records ? o->records : (uint64_t)possible; + } + fprintf(csv, + "%.6f,%.6f,%lu,%.6f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.3f," + "%.3f,%.6f,%.3f,%lu,%.3f,%.3f,%.3f,%lu,%.3f,%lu,%.3f", + elapsed, + interval, + inserted, + (double)inserted * (o->key_size + o->value_size) / GIB, + delta_insert / interval, + delta_query / interval, + delta_query ? (hits - prev_hits) * 100.0 / delta_query : 0, + insert_max / 1e6, + query_max / 1e6, + active_max / 1e6, + (cpu - prev_cpu) / interval, + usage_now.ru_maxrss / 1024.0, + (reads - prev_read) / (1048576.0 * interval), + (writes - prev_write) / (1048576.0 * interval), + (double)st.st_size / GIB, + free_gib, + inserters_running, + delta_barrier_count / interval, + delta_barrier_count + ? delta_barrier_sum / 1e6 / delta_barrier_count + : 0, + barrier_max_ns / 1e6, + o->target_insert_ops_s, + o->target_insert_ops_s ? (double)o->target_insert_ops_s : 0, + offered > inserted ? offered - inserted : 0, + response_max / 1e6); + for (uint64_t i = 0; i < started; i++) { + fprintf(csv, ",%.3f", rates[i]); + } + fputc('\n', csv); + if (fflush(csv) != 0) { + atomic_store(&b.error, errno); + atomic_store(&b.stop, true); + } + prev_hits = hits; + prev_read = reads; + prev_write = writes; + prev_cpu = cpu; + prev_barrier_count = barrier_count; + prev_barrier_sum_ns = barrier_sum_ns; + previous = now; + if (all_inserts_done || atomic_load(&b.stop)) { + break; + } + if (interrupted) { + reason = "signal"; + break; + } + if (o->seconds && elapsed >= o->seconds) { + reason = "time_limit"; + break; + } + if (free_gib < o->min_free_gib) { + reason = "free_space_limit"; + break; + } + } + atomic_store(&b.stop, true); + for (uint64_t i = 0; i < started; i++) { + pthread_join(b.workers[i].thread, NULL); + } + if (atomic_load(&b.barrier_started)) { + pthread_join(b.barrier_thread, NULL); + } + inserted = 0; + for (uint64_t i = 0; i < o->inserts; i++) { + inserted += atomic_load(&b.workers[i].completed); + } + double elapsed_s = (now_ns() - start) / 1e9; + fprintf(stderr, + "load_end reason=%s inserted=%lu elapsed_s=%.3f error=%d " + "barrier_count=%lu barrier_avg_ms=%.3f barrier_max_ms=%.3f\n", + reason, + inserted, + elapsed_s, + atomic_load(&b.error), + atomic_load(&b.barrier_count), + atomic_load(&b.barrier_count) ? atomic_load(&b.barrier_sum_ns) / 1e6 + / atomic_load(&b.barrier_count) + : 0, + atomic_load(&b.barrier_max_ns) / 1e6); + int summary_rc = write_summary(&b, inserted, elapsed_s); + if (summary_rc != 0) { + fprintf(stderr, "summary write failed: %s\n", strerror(summary_rc)); + } + uint64_t close_start = now_ns(); +#ifdef KVBENCH_LEGACY_CLOSE + splinterdb_close(&b.db); + rc = 0; +#else + rc = splinterdb_close(&b.db, false); +#endif + fprintf( + stderr, "close_s=%.3f close_rc=%d\n", (now_ns() - close_start) / 1e9, rc); + close(fd); + fclose(csv); + int trace_rc = 0; +#ifndef KVBENCH_NO_TRACE + trace_rc = platform_trace_close(); +#endif + bool incomplete = !o->seconds && inserted < o->records; + return rc || trace_rc || summary_rc || atomic_load(&b.error) || incomplete + ? 1 + : 0; +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/inventory.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/inventory.json new file mode 100644 index 000000000..7bc59d997 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/inventory.json @@ -0,0 +1,69 @@ +{ + "audit.py": "0a35a54b0932296bd60df29d54b4a1e28a1c3a4ff680ab096ce6d94c0f696b6c", + "status.json": "7a6ad6ae0ee7558291502adf577fe5dc95ceeffdb1f46435905683bcfd4c6094", + "build_all.py": "eb087dd5567dd000d7e0bf27d76839ac45e0ced00037f6f618e2494e5e518a20", + "run_matrix-fixtures.py": "776d2ee23705f3a60a54f4e78b4f305a24d4a124eeecfe75dfacf5a90e4329c5", + "feature-summaries.json": "be843d623a1047071f5ecf28698812b3a1a5570fb81ace0d2280c0aadaa0b3dc", + "disk-format-build.json": "77749416e4dcd7f3437f0ad83c9116b6f744ebded01b4855259360503149c91f", + "host.json": "3f6265a71d00bb4c31f35d6774a1ac19bdc4b10ffd758947d96b0d0b81538eb3", + "kvbench.c": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "control-drift.csv": "0c5f1c505305edd8f6c55943719aaceb9ff8ded781a746819df64bb305f0e941", + "preparation.json": "1de4fc5b02c3b6b5134859e33a99b881132ca60c193aa611370795131bb80c09", + "package.py": "8591b68385dafd2e808ed6b623a7f4e645c890b3aa1c023a49906061580fb79c", + "summary.csv": "1897072f20183daf0b47297d260ec894847bf7218cce2939fc40ea504cb9ce75", + "report.py": "5c1a9ef7925653b45ee3c211863706ba6cbf2924783a77c1efd7e0b83c3622a6", + "run_matrix.py": "536f10a1a7fd4b1e8e0250ea01cfe36793c7d6e2345f141092616ec06f64ee50", + "report.md": "9649eb4828bef7e39b3b28b1bdcd2ab3ddb72ceb0f71bb5d07c12605c8b417bc", + "run_job.py": "908d86de805e6efd41b423c919187f2df14715104f047e10fa8a6c57129cc773", + "repeat.py": "81fc83731f123067d5183231468513ad7243578ffc051a0d63073b684362fd53", + "audit.json": "ad2fc696155945dd8d7c31db1295ae1f824e7a4324f1523d290f106f95f87d03", + "benchmark-jobs.json": "a208b7be9c97f8c7d1d92fda93941d24a29cdf4d47f255946c33aab465f645c7", + "insert_benchmark.c": "75acbda52b2a8949f1d61710a8d89250d4dac63b12ec9803b89170f9c2cc0b26", + "main-build.json": "5eae9c1ff95058e999f98a2431c93553859fdbb35e531edbe6426a8c86339a99", + "insert/wal0-05-main.summary.csv": "2bd97e5bc7c20389cd6db2ba6410f9b6d8f1eda86b190dc0d62ace42199472d5", + "insert/wal0-00-main.csv": "8dcf4686167c43dd7df121b5a917f4dd88dfa8c06c1d7b496a7a3e68b339d1e0", + "insert/wal0-05-main.log": "e561f2425c9264f1abdedba855ac90a73514a661943399e896c390652cd8749c", + "insert/wal1-23-disk-format.summary.csv": "a525c1909b5de311ca55b0e8650936c206c967efcc56b3cb62e31c26863ae81b", + "insert/wal1-23-disk-format.csv": "4924b3de7fe729fa443e8f2d2a87c94d98902f6c78acdc3e094a4df752c3da89", + "insert/wal1-00-main.log": "d2d2e8354ebcb144c19981057836ad942f500f144aac30bc60c6975892496363", + "insert/wal1-24-main.csv": "618babe5f6164ea27cc3c33511d3cad0a35eb6c0329f8058c4810d0341a3dcb6", + "insert/wal0-00-main.log": "84c9d5144c6b034941a876f264e74f49cdb3449c4b6b75df09d3f774ae93270a", + "insert/manifest.json": "c90265cb7edaeb4ed6c076dc9079e37e2aaa3550808e12b875ff7959e91a5a0e", + "insert/wal1-00-main.summary.csv": "af53990e4fe8caf53c7caed4175c6b818cfb509bc0a6bb237e3b9aab387ec2d1", + "insert/wal1-01-disk-format.summary.csv": "fd0f95b28b0a142356b70b4b91717cc6344df37cb0d6522846e6f71db4341ba0", + "insert/wal1-01-disk-format.csv": "fddcb05a1832678136e070994c287fb532f0fb90af8eeef109f957242dbe6fdb", + "insert/wal1-01-disk-format.log": "55633a0fd575838aff77f7c9cca060d127da54a96ec9e6452898dc3aaf1bc007", + "insert/wal1-24-main.log": "0e3644505976ec7c8a8e4877dab5b0b9493a5f78984892223bd3e7f074c4557b", + "insert/wal1-00-main.csv": "78326f512429dd93c03975ee82baa0289cc22fb24919e9ed076825c5717bf574", + "insert/wal1-12-main.summary.csv": "cf0692fd4487f51302a7a67f0f0ba4b4668f6af4b7b0cb9b26e79c157121d9b7", + "insert/wal1-12-main.csv": "807b627316d359b79b43bea0bb408fa343d936e1095878ba19277dca36de5fe2", + "insert/wal1-23-disk-format.log": "d91b330e4a66204ecdd0794c0f82f7f92441f91f8746df2a59ee9e054f2a5b39", + "insert/wal1-12-main.log": "dfdc27d0b401b30c49872cd01233f3282c693df04abe9a7e47d84ad9c6bf553d", + "insert/wal1-24-main.summary.csv": "1aa6095d93e086f52d65679d4c2989e7a521a24991c85823fcc7873214b5c4da", + "insert/wal0-05-main.csv": "d9b770ba882910062d7e71cd7268f1db66b68aeae2459e2c47defee49ecdd709", + "insert/results.csv": "0f1db16b097ec601aed20258da6e51ba3c976d21f592e1dc2401ae79fd219bdf", + "insert/wal0-00-main.summary.csv": "cce097428cf6196a066ee953db6cba485e672210956aed375a8fcf6746d67891", + "fixtures/disk-main-ordinary.csv": "a3a321becf1795295cdecb25858a7ba1db4999b2631d026e0c45ccc1f37db0aa", + "fixtures/resident-disk-format-ordinary.csv": "508d0538a091c7ef27eca0fb87fa7f7a5a620cc1d0161ecac4dc79849c9944b8", + "fixtures/manifest.json": "6fcf525ed542823a87f9fb309d5326d0f317ea6ca6d2b6d577ee6947b04f53b9", + "fixtures/disk-main-ordinary.log": "df659510c3e45ad4982e8a4d87cf79bfc17eee6e29d823cfb16a86fa83d91419", + "fixtures/disk-disk-format-ordinary.csv": "04c055117a6066b222818f57ed86102eb94f73cd4ed3c648105743b99062b81b", + "fixtures/disk-disk-format-ordinary.log": "5e21dfcd258b47ef7d95ab24a76f428a25495b9aa00cca100f45cf9661d4a646", + "fixtures/resident-main-ordinary.log": "e55092c3e8d86fa11a0c8308f2951ca7d933e1437c34285fc696792625d8eea3", + "fixtures/resident-main-ordinary.csv": "c84ff2d1855eba7db823dc88621fad5a8230d9f68a0c4bb21d73fb69d8604e34", + "fixtures/resident-disk-format-ordinary.log": "80b8a6d6db1250cf8b95ab86ef8c7105f10b8560ed9e0203b3919dbff00d90c7", + "disk/22-main.log": "5086e61f0aaac72a1061e180c0877301383d9148d4822a6439cb0a2db936c1bd", + "disk/manifest.json": "674943c06a03c675c82292c2fb1c565e1b5ec43d963acc6c914206c179669c23", + "disk/21-disk-format.log": "e9a3eed5b11581ea5e2fc9a837325a51ebb80ab369c33222d4e17a7bb562b266", + "disk/00-main.log": "31d7f4877605590f45d53a31f1da36cfd3464ed056f9aea89f6083b962487883", + "disk/11-main.log": "7c21a91723c434998d5c9b8a8dd15c86bbea62e8b787cc23f4ed16acea5a5edf", + "disk/01-disk-format.log": "33d0fa616187da01c016c5cbd39e028762a63f877ace9c9e3c84a1297eaf6269", + "disk/results.csv": "407d883ad8dd53dd317a07e6ac5eb04fed390e1922ab78acb39e90f8da3c9f7e", + "resident/22-main.log": "12947e19e2731c0041d5caf9492a0f102ac3edab312e9eb80d03bcbe7275fea0", + "resident/manifest.json": "43b83a2513ed20b4f8a2bbeae0a04ae15a3bea6ad9ad5477195697d8ffa85945", + "resident/21-disk-format.log": "b4bff98b28bcae54d590f70bc4098eb4e952d4fc69aa0a383c0da4f9a8c2154a", + "resident/00-main.log": "0861c0ade82e24816323caa4da330c4cac0c53104f1160285c1cf8a8df22251a", + "resident/11-main.log": "034a238757bbc021b03c57327181adea2db55fca23bb013540bf3f4ab4091d19", + "resident/01-disk-format.log": "e94a6af4a231157eeac29ed3e093bf33df2acfa168489bc4518d37315075178e", + "resident/results.csv": "c856de7b448e7f8628d47792518b644042d11c82d32f338c26d722d00dd8bc1e" +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/kvbench.c b/benchmarks/results-20261013/main-7bb5fab/benchmarks/kvbench.c new file mode 100644 index 000000000..9b91f4ef9 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/kvbench.c @@ -0,0 +1,1707 @@ +// kvbench.c -- SplinterDB public-API benchmark used to compare two versions. +// +// Uses only splinterdb/splinterdb.h + splinterdb/default_data_config.h + libc, +// so the same source builds against versions whose internals differ. The one +// public-API difference handled here is splinterdb_close(): +// -DKVBENCH_CLOSE_HAS_FORCE => int splinterdb_close(splinterdb **, bool32) +// otherwise => void splinterdb_close(splinterdb **) +// +// Key scheme: logical id in [0,N). Present key = 8-byte big-endian(2*id) +// followed by 16 bytes of fixed padding (24 bytes total), so memcmp order == +// id order. Absent ("negative") keys use 2*id+1, interleaved with present +// keys. Value = 100 bytes: id (LE u64), generation (LE u64), filler. +// +// Each invocation: create/open the DB, run the listed phases in order, close. +// Phase syntax (optional "@T" overrides the thread count): +// load:random | load:seq fixed-count insert of all N ids, then close +// (close timed separately as "drain") +// warm parallel full scan; verifies every id once +// point:uniform|zipf|seq|neg fixed-duration lookups +// range:uniform|zipf:L fixed-duration forward range queries +// ow:uniform|zipf fixed-duration overwrites (insert existing) +// One CSV row per phase is appended to --csv. + +#ifndef _GNU_SOURCE +# define _GNU_SOURCE +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "splinterdb/default_data_config.h" +#include "splinterdb/splinterdb.h" + +#define KEY_SIZE 24 +#define VALUE_SIZE 100 +#define MAX_PHASES 64 +#define MAX_WORKERS 32 + +static const char KEY_PAD[16] = "kvbench-padding!"; + +/* ------------------------------------------------------------------------ */ +/* Time, RNG, permutations, zipf */ +/* ------------------------------------------------------------------------ */ + +static inline uint64_t +now_ns(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +/* Bijective 64-bit mixer (splitmix64 finalizer). */ +static inline uint64_t +mix(uint64_t x) +{ + x = (x ^ (x >> 30)) * UINT64_C(0xbf58476d1ce4e5b9); + x = (x ^ (x >> 27)) * UINT64_C(0x94d049bb133111eb); + return x ^ (x >> 31); +} + +static inline uint64_t +rng_next(uint64_t *s) +{ + *s += UINT64_C(0x9e3779b97f4a7c15); + return mix(*s); +} + +static inline double +rng_unit(uint64_t *s) +{ + return (double)(rng_next(s) >> 11) * 0x1.0p-53; +} + +/* + * Pseudo-random bijection on [0,n): a 4-round balanced Feistel network over + * 2*half bits (2^(2*half) >= n), with cycle-walking to stay inside [0,n). + */ +typedef struct { + uint64_t n; + unsigned half; + uint64_t mask; + uint64_t keys[4]; +} perm_t; + +static void +perm_init(perm_t *p, uint64_t n, uint64_t seed) +{ + unsigned bits = 1; + while (bits < 64 && (UINT64_C(1) << bits) < n) { + bits++; + } + p->n = n; + p->half = (bits + 1) / 2; + if (p->half < 1) { + p->half = 1; + } + p->mask = (UINT64_C(1) << p->half) - 1; + for (int r = 0; r < 4; r++) { + p->keys[r] = mix(seed * UINT64_C(0x632be59bd9b4e019) + r + 1); + } +} + +static inline uint64_t +feistel(const perm_t *p, uint64_t x) +{ + uint64_t l = x >> p->half, r = x & p->mask; + for (int i = 0; i < 4; i++) { + uint64_t t = l ^ (mix(r ^ p->keys[i]) & p->mask); + l = r; + r = t; + } + return (l << p->half) | r; +} + +static inline uint64_t +perm_apply(const perm_t *p, uint64_t i) +{ + uint64_t x = i; + do { + x = feistel(p, x); + } while (x >= p->n); + return x; +} + +/* YCSB-style zipfian over ranks [0,n). */ +typedef struct { + uint64_t n; + double theta, zetan, zeta2, alpha, eta, half_pow_theta; +} zipf_t; + +static double +zeta_sum(uint64_t n, double theta) +{ + uint64_t m = n < 2000000 ? n : 2000000; + double s = 0; + for (uint64_t i = 1; i <= m; i++) { + s += pow((double)i, -theta); + } + if (n > m) { + /* Euler-Maclaurin tail: sum_{k=m+1}^{n} k^-t */ + s += (pow((double)n, 1 - theta) - pow((double)m, 1 - theta)) / (1 - theta) + + 0.5 * (pow((double)n, -theta) - pow((double)m, -theta)); + } + return s; +} + +static void +zipf_init(zipf_t *z, uint64_t n, double theta) +{ + z->n = n; + z->theta = theta; + z->zetan = zeta_sum(n, theta); + z->zeta2 = 1.0 + pow(0.5, theta); + z->alpha = 1.0 / (1.0 - theta); + z->eta = + (1.0 - pow(2.0 / (double)n, 1.0 - theta)) / (1.0 - z->zeta2 / z->zetan); + z->half_pow_theta = pow(0.5, theta); +} + +static inline uint64_t +zipf_next(const zipf_t *z, uint64_t *rng) +{ + double u = rng_unit(rng); + double uz = u * z->zetan; + if (uz < 1.0) { + return 0; + } + if (uz < 1.0 + z->half_pow_theta) { + return 1; + } + uint64_t r = + (uint64_t)((double)z->n * pow(z->eta * u - z->eta + 1.0, z->alpha)); + return r >= z->n ? z->n - 1 : r; +} + +/* ------------------------------------------------------------------------ */ +/* Histogram: log2 buckets with 32 linear sub-buckets (~3% resolution) */ +/* ------------------------------------------------------------------------ */ + +#define HSUB_BITS 5 +#define HSUB (1u << HSUB_BITS) +#define HBUCKETS (64 * HSUB) + +typedef struct { + uint64_t b[HBUCKETS]; + uint64_t count, sum, max; +} hist_t; + +static inline unsigned +hidx(uint64_t v) +{ + if (v < HSUB) { + return (unsigned)v; + } + unsigned e = 63 - __builtin_clzll(v); + unsigned sub = (unsigned)((v >> (e - HSUB_BITS)) & (HSUB - 1)); + return (e - HSUB_BITS + 1) * HSUB + sub; +} + +static inline uint64_t +hval(unsigned idx) +{ + if (idx < HSUB) { + return idx; + } + unsigned e = idx / HSUB + HSUB_BITS - 1; + unsigned sub = idx % HSUB; + uint64_t lo = (UINT64_C(1) << e) | ((uint64_t)sub << (e - HSUB_BITS)); + uint64_t width = UINT64_C(1) << (e - HSUB_BITS); + return lo + width / 2; +} + +static inline void +hist_add(hist_t *h, uint64_t v) +{ + h->b[hidx(v)]++; + h->count++; + h->sum += v; + if (v > h->max) { + h->max = v; + } +} + +static void +hist_merge(hist_t *dst, const hist_t *src) +{ + for (unsigned i = 0; i < HBUCKETS; i++) { + dst->b[i] += src->b[i]; + } + dst->count += src->count; + dst->sum += src->sum; + if (src->max > dst->max) { + dst->max = src->max; + } +} + +static double +hist_pct_us(const hist_t *h, double p) +{ + if (!h->count) { + return NAN; + } + uint64_t rank = (uint64_t)ceil(p * (double)h->count); + if (rank < 1) { + rank = 1; + } + uint64_t acc = 0; + for (unsigned i = 0; i < HBUCKETS; i++) { + acc += h->b[i]; + if (acc >= rank) { + uint64_t v = hval(i); + return (double)(v > h->max ? h->max : v) / 1e3; + } + } + return (double)h->max / 1e3; +} + +static double +hist_mean_us(const hist_t *h) +{ + return h->count ? (double)h->sum / (double)h->count / 1e3 : NAN; +} + +/* ------------------------------------------------------------------------ */ +/* Options / global state */ +/* ------------------------------------------------------------------------ */ + +enum phase_kind { PH_LOAD, PH_WARM, PH_POINT, PH_RANGE, PH_OW }; +enum dist { D_UNIFORM, D_ZIPF, D_SEQ, D_NEG, D_RANDOM }; + +typedef struct { + char name[64]; + enum phase_kind kind; + enum dist dist; + uint64_t L; + int threads; +} phase_t; + +typedef struct { + const char *db, *csv, *hang_dir; + const char *vtag, *stag, *otag; + int rep; + bool create; + uint64_t n, cache_mib, disk_gib, memtable_mib; + int use_log, threads, mt_bg, normal_bg; + double secs, warmup; + uint64_t sabotage; + uint64_t hang_secs; + uint64_t seed; + int nphases; + phase_t phases[MAX_PHASES]; +} options; + +static options opt; +static splinterdb *kvs; +static data_config data_cfg; +static perm_t load_perm, zipf_perm; +static zipf_t zipf; +static uint64_t generation; /* value generation for overwrites */ + +enum { ST_WAIT = 0, ST_WARMUP = 1, ST_MEASURE = 2, ST_STOP = 3 }; +static _Atomic int run_state; +static _Atomic uint64_t seq_counter; + +/* Watchdog */ +enum { WD_IDLE, WD_OPS, WD_OPENCLOSE }; +static _Atomic int wd_mode; +static _Atomic uint64_t wd_mode_since_ns; +static const char *wd_what = ""; + +typedef struct worker { + pthread_t th; + int idx; + const phase_t *ph; + hist_t lat; + hist_t startup; + uint64_t ops, tuples, pt_ns, pt_div, deinit_ns, deinit_n; + uint64_t violations, errors, notfound, found; + uint64_t rng; + uint64_t measure_rng; + bool measure_rng_captured; + _Atomic uint64_t progress; + char first_violation[256]; +} worker_t; + +static worker_t *workers; +static int nworkers; + +/* ------------------------------------------------------------------------ */ +/* Keys / values */ +/* ------------------------------------------------------------------------ */ + +static inline void +make_key(uint64_t k, char *key) +{ + for (int i = 0; i < 8; i++) { + key[i] = (char)(k >> (56 - 8 * i)); + } + memcpy(key + 8, KEY_PAD, 16); +} + +static inline bool +decode_key(slice s, uint64_t *k) +{ + if (slice_length(s) != KEY_SIZE) { + return false; + } + const unsigned char *p = slice_data(s); + uint64_t x = 0; + for (int i = 0; i < 8; i++) { + x = (x << 8) | p[i]; + } + *k = x; + return memcmp(p + 8, KEY_PAD, 16) == 0; +} + +static inline void +make_value(uint64_t id, uint64_t gen, char *val) +{ + memcpy(val, &id, 8); + memcpy(val + 8, &gen, 8); + memset(val + 16, 'v', VALUE_SIZE - 16); +} + +static inline bool +value_id_ok(slice v, uint64_t id) +{ + if (slice_length(v) != VALUE_SIZE) { + return false; + } + uint64_t got; + memcpy(&got, slice_data(v), 8); + return got == id; +} + +static inline bool +value_generation_ok(slice v, uint64_t id, uint64_t expected_generation) +{ + if (!value_id_ok(v, id)) { + return false; + } + uint64_t got_generation; + memcpy(&got_generation, (const char *)slice_data(v) + 8, 8); + return got_generation == expected_generation; +} + +static void +violation(worker_t *w, const char *fmt, ...) + __attribute__((format(printf, 2, 3))); + +static void +violation(worker_t *w, const char *fmt, ...) +{ + w->violations++; + if (w->violations <= 3) { + char buf[256]; + va_list ap; + va_start(ap, fmt); + vsnprintf(buf, sizeof(buf), fmt, ap); + va_end(ap); + if (w->violations == 1) { + snprintf(w->first_violation, sizeof(w->first_violation), "%s", buf); + } + fprintf(stderr, "VIOLATION [thread %d]: %s\n", w->idx, buf); + } +} + +/* ------------------------------------------------------------------------ */ +/* Process metrics */ +/* ------------------------------------------------------------------------ */ + +typedef struct { + uint64_t t_ns; + double cpu_s; + uint64_t rd, wr; + uint64_t swpin, swpout; + uint64_t majflt; +} snap_t; + +static void +proc_io(uint64_t *rd, uint64_t *wr) +{ + *rd = *wr = 0; + FILE *f = fopen("/proc/self/io", "r"); + if (!f) { + return; + } + char name[64]; + uint64_t v; + while (fscanf(f, "%63s %" SCNu64, name, &v) == 2) { + if (!strcmp(name, "read_bytes:")) { + *rd = v; + } else if (!strcmp(name, "write_bytes:")) { + *wr = v; + } + } + fclose(f); +} + +static void +vm_swap(uint64_t *in, uint64_t *out) +{ + *in = *out = 0; + FILE *f = fopen("/proc/vmstat", "r"); + if (!f) { + return; + } + char name[64]; + uint64_t v; + while (fscanf(f, "%63s %" SCNu64, name, &v) == 2) { + if (!strcmp(name, "pswpin")) { + *in = v; + } else if (!strcmp(name, "pswpout")) { + *out = v; + } + } + fclose(f); +} + +static void +take_snap(snap_t *s) +{ + struct rusage ru; + getrusage(RUSAGE_SELF, &ru); + s->cpu_s = ru.ru_utime.tv_sec + ru.ru_utime.tv_usec / 1e6 + + ru.ru_stime.tv_sec + ru.ru_stime.tv_usec / 1e6; + s->majflt = ru.ru_majflt; + proc_io(&s->rd, &s->wr); + vm_swap(&s->swpin, &s->swpout); + s->t_ns = now_ns(); +} + +/* ------------------------------------------------------------------------ */ +/* DB lifecycle */ +/* ------------------------------------------------------------------------ */ + +static void +fill_config(splinterdb_config *cfg) +{ + *cfg = (splinterdb_config){ + .filename = opt.db, + .cache_size = opt.cache_mib << 20, + .disk_size = opt.disk_gib << 30, + .data_cfg = &data_cfg, + .io_flags = O_RDWR | O_CREAT | O_DIRECT, + .io_perms = 0644, + .use_log = opt.use_log ? 1 : 0, + .memtable_capacity = opt.memtable_mib << 20, + .fanout = 8, + .num_memtable_bg_threads = (uint64)opt.mt_bg, + .num_normal_bg_threads = (uint64)opt.normal_bg, + .queue_scale_percent = 100, + }; +#ifdef KVBENCH_SPACE_MODE + cfg->use_stats = 1; +#endif +} + +static double open_secs; + +static void +db_open(bool create) +{ + splinterdb_config cfg; + fill_config(&cfg); + wd_what = create ? "create" : "open"; + atomic_store(&wd_mode_since_ns, now_ns()); + atomic_store(&wd_mode, WD_OPENCLOSE); + uint64_t t0 = now_ns(); + int rc = + create ? splinterdb_create(&cfg, &kvs) : splinterdb_open(&cfg, &kvs); + open_secs = (now_ns() - t0) / 1e9; + atomic_store(&wd_mode, WD_IDLE); + if (rc) { + fprintf(stderr, "FATAL: splinterdb_%s failed rc=%d\n", wd_what, rc); + exit(3); + } + fprintf(stderr, "%s took %.3f s\n", wd_what, open_secs); +} + +static double +db_close(void) +{ +#ifdef KVBENCH_SPACE_MODE + /* Space/stats build only: dump insertion + allocator stats for the session. + */ + if (getenv("KVBENCH_STATS_BEFORE_CLOSE")) { + platform_set_log_streams(stdout, stderr); + splinterdb_stats_print_insertion(kvs); + fflush(stdout); + } +#endif + wd_what = "close"; + atomic_store(&wd_mode_since_ns, now_ns()); + atomic_store(&wd_mode, WD_OPENCLOSE); + uint64_t t0 = now_ns(); +#ifdef KVBENCH_CLOSE_HAS_FORCE + int rc = splinterdb_close(&kvs, 0); + if (rc) { + fprintf(stderr, + "FINDING: splinterdb_close(force=0) refused rc=%d; forcing\n", + rc); + rc = splinterdb_close(&kvs, 1); + if (rc) { + fprintf(stderr, "FINDING: splinterdb_close(force=1) rc=%d\n", rc); + } + } +#else + splinterdb_close(&kvs); +#endif + double s = (now_ns() - t0) / 1e9; + atomic_store(&wd_mode, WD_IDLE); + kvs = NULL; + fprintf(stderr, "close took %.3f s\n", s); + return s; +} + +/* ------------------------------------------------------------------------ */ +/* Watchdog */ +/* ------------------------------------------------------------------------ */ + +static void +hang_dump(const char *why) +{ + char path[512], cmd[1024]; + snprintf(path, + sizeof(path), + "%s/hang-%s-%s-%d-%ld.txt", + opt.hang_dir ? opt.hang_dir : ".", + opt.vtag, + wd_what, + (int)getpid(), + (long)time(NULL)); + fprintf(stderr, + "HANG DETECTED (%s) during %s; dumping stacks to %s\n", + why, + wd_what, + path); + snprintf(cmd, + sizeof(cmd), + "gdb -batch -ex 'info threads' -ex 'thread apply all bt' -p %d > " + "'%s' 2>&1", + (int)getpid(), + path); + int rc = system(cmd); + fprintf(stderr, "gdb rc=%d; exiting with status 99\n", rc); + fflush(stderr); + _exit(99); +} + +static void * +watchdog_main(void *arg) +{ + (void)arg; + uint64_t last_prog = 0, last_change = now_ns(), last_report = now_ns(); + uint64_t last_report_prog = 0; + double last_cpu = -1; + uint64_t last_io = 0; + int last_mode = WD_IDLE; + for (;;) { + sleep(1); + int mode = atomic_load(&wd_mode); + uint64_t now = now_ns(); + if (mode != last_mode) { + last_mode = mode; + last_change = now; + last_prog = 0; + last_report_prog = 0; + last_report = now; + last_cpu = -1; + } + if (mode == WD_OPS) { + uint64_t prog = 0; + for (int i = 0; i < nworkers; i++) { + prog += + atomic_load_explicit(&workers[i].progress, memory_order_relaxed); + } + if (prog != last_prog) { + last_prog = prog; + last_change = now; + } else if (now - last_change > opt.hang_secs * 1000000000ull) { + hang_dump("no op progress for hang_secs"); + } + if (now - last_report >= 10ull * 1000000000ull) { + fprintf(stderr, + " [%s] progress %" PRIu64 " ops (+%.0f ops/s)\n", + wd_what, + prog, + (prog - last_report_prog) / ((now - last_report) / 1e9)); + last_report = now; + last_report_prog = prog; + } + } else if (mode == WD_OPENCLOSE) { + snap_t s; + take_snap(&s); + uint64_t io = s.rd + s.wr; + if (last_cpu < 0 || s.cpu_s - last_cpu > 0.05 || io != last_io) { + last_cpu = s.cpu_s; + last_io = io; + last_change = now; + } else if (now - last_change > opt.hang_secs * 1000000000ull) { + hang_dump("no CPU or IO progress for hang_secs"); + } + uint64_t since = atomic_load(&wd_mode_since_ns); + if (now - since > 2400ull * 1000000000ull) { + hang_dump("open/close exceeded 40 minutes"); + } + } + } + return NULL; +} + +/* ------------------------------------------------------------------------ */ +/* Worker bodies */ +/* ------------------------------------------------------------------------ */ + +static inline uint64_t +pick_id(worker_t *w, enum dist d, uint64_t *seqpos, uint64_t lo, uint64_t hi) +{ + switch (d) { + case D_ZIPF: + return perm_apply(&zipf_perm, zipf_next(&zipf, &w->rng)); + case D_SEQ: + { + uint64_t id = *seqpos; + *seqpos = (id + 1 >= hi) ? lo : id + 1; + return id; + } + case D_UNIFORM: + case D_NEG: + default: + return rng_next(&w->rng) % opt.n; + } +} + +static inline void +wait_start(void) +{ + while (atomic_load_explicit(&run_state, memory_order_acquire) == ST_WAIT) { + struct timespec d = {.tv_nsec = 100000}; + nanosleep(&d, NULL); + } +} + +static void * +load_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE], val[VALUE_SIZE]; + uint64_t T = (uint64_t)nworkers; + uint64_t lo = opt.n * (uint64_t)w->idx / T, + hi = opt.n * (uint64_t)(w->idx + 1) / T; + wait_start(); + uint64_t pos = lo, chunk_end = 0, cur = 0; + for (;;) { + uint64_t id; + if (w->ph->dist == D_RANDOM) { + if (pos >= hi) { + break; + } + id = perm_apply(&load_perm, pos++); + } else { + if (cur >= chunk_end) { + cur = atomic_fetch_add(&seq_counter, 64); + if (cur >= opt.n) { + break; + } + chunk_end = cur + 64 > opt.n ? opt.n : cur + 64; + } + id = cur++; + } + make_key(2 * id, key); + make_value(id, 0, val); + uint64_t t0 = now_ns(); + int rc = splinterdb_insert( + kvs, slice_create(KEY_SIZE, key), slice_create(VALUE_SIZE, val), NULL); + uint64_t t1 = now_ns(); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: insert id=%" PRIu64 " rc=%d\n", id, rc); + } + } + hist_add(&w->lat, t1 - t0); + w->ops++; + atomic_store_explicit(&w->progress, w->ops, memory_order_relaxed); + } + return NULL; +} + +static void * +warm_worker(void *arg) +{ + worker_t *w = arg; + uint64_t T = (uint64_t)nworkers; + uint64_t lo = opt.n * (uint64_t)w->idx / T, + hi = opt.n * (uint64_t)(w->idx + 1) / T; + if ((opt.sabotage & 8) && w->idx == nworkers - 1) { + hi++; /* sabotage: expect one more id than exists */ + } + char key[KEY_SIZE]; + make_key(2 * lo, key); + wait_start(); + splinterdb_iterator *it; + int rc = splinterdb_iterator_init( + kvs, &it, greater_than_or_equal, slice_create(KEY_SIZE, key)); + if (rc) { + w->errors++; + fprintf(stderr, "ERROR: warm iterator_init rc=%d\n", rc); + return NULL; + } + uint64_t expect = lo; + while (expect < hi && splinterdb_iterator_valid(it)) { + slice k, v; + uint64_t kk; + splinterdb_iterator_get_current(it, &k, &v); + if (!decode_key(k, &kk) || kk != 2 * expect || !value_id_ok(v, expect)) { + violation(w, + "warm: expected id %" PRIu64 " got key %" PRIu64 + " (len %" PRIu64 ")", + expect, + kk, + slice_length(k)); + if (kk > 2 * expect && (kk & 1) == 0) { + expect = kk / 2; /* resync after a gap */ + } + } + expect++; + w->ops++; + if ((w->ops & 1023) == 0) { + atomic_store_explicit(&w->progress, w->ops, memory_order_relaxed); + } + if (expect < hi) { + splinterdb_iterator_next(it); + } + } + rc = splinterdb_iterator_status(it); + if (rc) { + w->errors++; + fprintf(stderr, "ERROR: warm iterator status rc=%d\n", rc); + } + if (expect != hi) { + violation(w, + "warm: range [%" PRIu64 ",%" PRIu64 ") ended at %" PRIu64, + lo, + hi, + expect); + } + splinterdb_iterator_deinit(it); + atomic_store_explicit(&w->progress, w->ops, memory_order_relaxed); + return NULL; +} + +static void * +point_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE]; + char buf[256]; + splinterdb_lookup_result res; + splinterdb_lookup_result_init( + kvs, &res, SPLINTERDB_LOOKUP_VALUE, sizeof(buf), buf); + uint64_t T = (uint64_t)nworkers; + uint64_t lo = opt.n * (uint64_t)w->idx / T, + hi = opt.n * (uint64_t)(w->idx + 1) / T; + uint64_t seqpos = lo; + bool neg = w->ph->dist == D_NEG; + wait_start(); + uint64_t n = 0; + for (;;) { + int st = atomic_load_explicit(&run_state, memory_order_relaxed); + if (st == ST_STOP) { + break; + } + uint64_t id = pick_id(w, w->ph->dist, &seqpos, lo, hi); + bool q_neg = neg; + bool e_neg = neg; + if (opt.sabotage & 1) { + q_neg = true; /* sabotage: query absent key, expect present */ + } + if ((opt.sabotage & 16) && n > 1000) { + sleep(100000); /* test hook: simulate a hang for the watchdog */ + } + if (opt.sabotage & 2) { + q_neg = false; /* sabotage: query present key, expect absent */ + } + make_key(q_neg ? 2 * id + 1 : 2 * id, key); + uint64_t t0 = now_ns(); + int rc = splinterdb_lookup(kvs, slice_create(KEY_SIZE, key), &res); + uint64_t t1 = now_ns(); + n++; + atomic_store_explicit(&w->progress, n, memory_order_relaxed); + if (st != ST_MEASURE) { + continue; + } + w->ops++; + hist_add(&w->lat, t1 - t0); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: lookup id=%" PRIu64 " rc=%d\n", id, rc); + } + continue; + } + bool found = splinterdb_lookup_found(&res); + if (found) { + w->found++; + } else { + w->notfound++; + } + if (e_neg) { + if (found) { + violation(w, "negative key 2*%" PRIu64 "+1 found", id); + } + } else { + slice v; + if (!found) { + violation(w, "present id %" PRIu64 " not found", id); + } else if (splinterdb_lookup_result_value(&res, &v) != 0 + || !value_id_ok(v, id)) + { + violation(w, "present id %" PRIu64 " wrong value", id); + } + } + } + splinterdb_lookup_result_deinit(&res); + return NULL; +} + +static void * +range_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE]; + uint64_t L = w->ph->L; + uint64_t seqpos = 0; + uint64_t n = 0; + wait_start(); + for (;;) { + int st = atomic_load_explicit(&run_state, memory_order_relaxed); + if (st == ST_STOP) { + break; + } + uint64_t start = pick_id(w, w->ph->dist, &seqpos, 0, opt.n); + uint64_t expect = opt.n - start < L ? opt.n - start : L; + if (opt.sabotage & 4) { + expect++; /* sabotage: expect one extra tuple */ + } + make_key(2 * start, key); + uint64_t t0 = now_ns(), t1 = 0, tl = 0; + splinterdb_iterator *it; + int rc = splinterdb_iterator_init( + kvs, &it, greater_than_or_equal, slice_create(KEY_SIZE, key)); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: iterator_init rc=%d\n", rc); + } + continue; + } + uint64_t got = 0; + bool bad = false; + while (got < L && splinterdb_iterator_valid(it)) { + slice k, v; + uint64_t kk = 0; + splinterdb_iterator_get_current(it, &k, &v); + if (!bad + && (!decode_key(k, &kk) || kk != 2 * (start + got) + || !value_id_ok(v, start + got))) + { + bad = true; + if (st == ST_MEASURE) { + violation(w, + "range start %" PRIu64 " tuple %" PRIu64 + ": expected key %" PRIu64 " got %" PRIu64, + start, + got, + 2 * (start + got), + kk); + } + } + got++; + if (got == 1) { + t1 = now_ns(); + } + if (got == L) { + tl = now_ns(); + break; + } + splinterdb_iterator_next(it); + } + int status = splinterdb_iterator_status(it); + uint64_t t2 = now_ns(); + splinterdb_iterator_deinit(it); + uint64_t t3 = now_ns(); + n++; + atomic_store_explicit(&w->progress, n, memory_order_relaxed); + if (st != ST_MEASURE) { + continue; + } + if (status) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: iterator status rc=%d\n", status); + } + } + if (got != expect) { + violation(w, + "range start %" PRIu64 " L=%" PRIu64 ": got %" PRIu64 + " tuples, expected %" PRIu64, + start, + L, + got, + expect); + } + w->ops++; + w->tuples += got; + hist_add(&w->lat, t3 - t0); + if (got >= 1) { + hist_add(&w->startup, t1 - t0); + } + if (L > 1 && got == L) { + w->pt_ns += tl - t1; + w->pt_div += L - 1; + } + w->deinit_ns += t3 - t2; + w->deinit_n++; + } + return NULL; +} + +static void * +ow_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE], val[VALUE_SIZE]; + uint64_t seqpos = 0, n = 0; + wait_start(); + for (;;) { + int st = atomic_load_explicit(&run_state, memory_order_relaxed); + if (st == ST_STOP) { + break; + } + if (st == ST_MEASURE && !w->measure_rng_captured) { + w->measure_rng = w->rng; + w->measure_rng_captured = true; + } + uint64_t id = pick_id(w, w->ph->dist, &seqpos, 0, opt.n); + make_key(2 * id, key); + make_value(id, generation, val); + uint64_t t0 = now_ns(); + int rc = splinterdb_insert( + kvs, slice_create(KEY_SIZE, key), slice_create(VALUE_SIZE, val), NULL); + uint64_t t1 = now_ns(); + n++; + atomic_store_explicit(&w->progress, n, memory_order_relaxed); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: overwrite id=%" PRIu64 " rc=%d\n", id, rc); + } + } + if (st != ST_MEASURE) { + continue; + } + w->ops++; + hist_add(&w->lat, t1 - t0); + } + return NULL; +} + +/* + * Validate an untimed deterministic sample from the beginning of each + * overwrite worker's measured stream. Replaying the saved RNG state keeps all + * validation work out of the measured hot path. + */ +static void +verify_overwrite_sample(const phase_t *ph, int threads) +{ + enum { SAMPLES_PER_WORKER = 4096 }; + char key[KEY_SIZE], buffer[VALUE_SIZE]; + splinterdb_lookup_result result; + splinterdb_lookup_result_init( + kvs, &result, SPLINTERDB_LOOKUP_VALUE, sizeof(buffer), buffer); + + for (int i = 0; i < threads; i++) { + worker_t *original = &workers[i]; + if (original->ops == 0) { + continue; + } + if (!original->measure_rng_captured) { + original->errors++; + fprintf( + stderr, + "ERROR: overwrite verifier has no measurement RNG for thread %d\n", + i); + continue; + } + worker_t replay = {.rng = original->measure_rng}; + uint64_t seqpos = 0; + uint64_t samples = original->ops < SAMPLES_PER_WORKER + ? original->ops + : SAMPLES_PER_WORKER; + for (uint64_t sample = 0; sample < samples; sample++) { + uint64_t id = pick_id(&replay, ph->dist, &seqpos, 0, opt.n); + make_key(2 * id, key); + int rc = splinterdb_lookup(kvs, slice_create(KEY_SIZE, key), &result); + if (rc != 0) { + original->errors++; + if (original->errors <= 3) { + fprintf(stderr, + "ERROR: overwrite verification id=%" PRIu64 " rc=%d\n", + id, + rc); + } + continue; + } + slice value; + if (!splinterdb_lookup_found(&result) + || splinterdb_lookup_result_value(&result, &value) != 0 + || !value_generation_ok(value, id, generation)) + { + violation(original, + "overwrite verification id=%" PRIu64 + " did not return generation %" PRIu64, + id, + generation); + } + } + } + splinterdb_lookup_result_deinit(&result); +} + +/* ------------------------------------------------------------------------ */ +/* Phase driver */ +/* ------------------------------------------------------------------------ */ + +static const char * +dist_name(enum dist d) +{ + switch (d) { + case D_UNIFORM: + return "uniform"; + case D_ZIPF: + return "zipf"; + case D_SEQ: + return "seq"; + case D_NEG: + return "neg"; + case D_RANDOM: + return "random"; + } + return "?"; +} + +static void +sleep_secs(double s) +{ + uint64_t end = now_ns() + (uint64_t)(s * 1e9); + for (;;) { + uint64_t now = now_ns(); + if (now >= end) { + break; + } + uint64_t rem = end - now; + struct timespec d = {.tv_sec = rem / 1000000000ull, + .tv_nsec = rem % 1000000000ull}; + nanosleep(&d, NULL); + } +} + +static void +file_sizes(uint64_t *apparent, uint64_t *alloc) +{ + struct stat st; + if (stat(opt.db, &st) == 0) { + *apparent = (uint64_t)st.st_size; + *alloc = (uint64_t)st.st_blocks * 512; + } else { + *apparent = *alloc = 0; + } +} + +static int csv_header_written; + +static void +csv_row(const phase_t *ph, + int threads, + hist_t *lat, + hist_t *startup, + uint64_t ops, + uint64_t tuples, + uint64_t pt_ns, + uint64_t pt_div, + uint64_t deinit_ns, + uint64_t deinit_n, + uint64_t viol, + uint64_t errs, + const snap_t *a, + const snap_t *b, + double insert_s, + double drain_s, + const snap_t *da, + const snap_t *db_, + uint64_t app, + uint64_t alc) +{ + double secs = (b->t_ns - a->t_ns) / 1e9; + double cpu = b->cpu_s - a->cpu_s; + uint64_t rd = b->rd - a->rd, wr = b->wr - a->wr; + double d_cpu = da ? db_->cpu_s - da->cpu_s : 0; + uint64_t d_rd = da ? db_->rd - da->rd : 0, d_wr = da ? db_->wr - da->wr : 0; + uint64_t swin = b->swpin - a->swpin + (da ? db_->swpin - da->swpin : 0); + uint64_t swout = b->swpout - a->swpout + (da ? db_->swpout - da->swpout : 0); + uint64_t majflt = b->majflt - a->majflt; + double logical = (double)ops * (KEY_SIZE + VALUE_SIZE); + double wamp = (ph->kind == PH_LOAD || ph->kind == PH_OW) && ops + ? (double)(wr + d_wr) / logical + : NAN; + double pt = pt_div ? (double)pt_ns / (double)pt_div : NAN; + double dmean = deinit_n ? (double)deinit_ns / (double)deinit_n / 1e3 : NAN; + + printf("== %-22s T=%-2d ops=%-10" PRIu64 " secs=%7.2f ops/s=%11.0f " + "lat us p50=%.2f p99=%.2f p99.9=%.2f max=%.1f mean=%.2f\n", + ph->name, + threads, + ops, + secs, + ops / secs, + hist_pct_us(lat, 0.5), + hist_pct_us(lat, 0.99), + hist_pct_us(lat, 0.999), + (double)lat->max / 1e3, + hist_mean_us(lat)); + printf(" cpu=%.2fs ops/cpu-s=%.0f rd=%.1f B/op wr=%.1f B/op " + "violations=%" PRIu64 " errors=%" PRIu64 " swapin=%" PRIu64 + " swapout=%" PRIu64 " majflt=%" PRIu64 "\n", + cpu, + ops / cpu, + ops ? (double)rd / ops : 0, + ops ? (double)wr / ops : 0, + viol, + errs, + swin, + swout, + majflt); + if (ph->kind == PH_RANGE) { + printf(" startup us p50=%.2f p99=%.2f mean=%.2f per-tuple=%.1f ns " + "deinit mean=%.2f us tuples/s=%.0f\n", + hist_pct_us(startup, 0.5), + hist_pct_us(startup, 0.99), + hist_mean_us(startup), + pt, + dmean, + tuples / secs); + } + if (da) { + printf( + " insert/run=%.2fs drain(close)=%.2fs total=%.2fs drain_cpu=%.2fs " + "drain_rd=%" PRIu64 " drain_wr=%" PRIu64 " wamp=%.3f " + "db_apparent=%" PRIu64 " db_alloc=%" PRIu64 "\n", + insert_s, + drain_s, + insert_s + drain_s, + d_cpu, + d_rd, + d_wr, + wamp, + app, + alc); + } + fflush(stdout); + + if (!opt.csv) { + return; + } + FILE *f = fopen(opt.csv, "a"); + if (!f) { + perror(opt.csv); + return; + } + if (!csv_header_written) { + fseek(f, 0, SEEK_END); + if (ftell(f) == 0) { + fprintf(f, + "version,size,order,log,phase,threads,rep,n,ops,secs,ops_s," + "p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s," + "rd_bytes,wr_bytes,rd_B_op,wr_B_op," + "startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns," + "deinit_mean_us,tuples,tuples_s," + "insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp," + "db_apparent,db_alloc,open_s,violations,errors,swapin,swapout," + "majflt\n"); + } + csv_header_written = 1; + } + fprintf(f, + "%s,%s,%s,%d,%s,%d,%d,%" PRIu64 ",%" PRIu64 ",%.3f,%.1f," + "%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.1f," + "%" PRIu64 ",%" PRIu64 ",%.2f,%.2f," + "%.3f,%.3f,%.3f,%.2f," + "%.3f,%" PRIu64 ",%.1f," + "%.3f,%.3f,%.3f,%.3f,%" PRIu64 ",%" PRIu64 ",%.4f," + "%" PRIu64 ",%" PRIu64 ",%.3f,%" PRIu64 ",%" PRIu64 ",%" PRIu64 + ",%" PRIu64 ",%" PRIu64 "\n", + opt.vtag, + opt.stag, + opt.otag, + opt.use_log, + ph->name, + threads, + opt.rep, + opt.n, + ops, + secs, + ops / secs, + hist_pct_us(lat, 0.5), + hist_pct_us(lat, 0.99), + hist_pct_us(lat, 0.999), + (double)lat->max / 1e3, + hist_mean_us(lat), + cpu, + ops / cpu, + rd, + wr, + ops ? (double)rd / ops : 0, + ops ? (double)wr / ops : 0, + hist_pct_us(startup, 0.5), + hist_pct_us(startup, 0.99), + hist_mean_us(startup), + pt, + dmean, + tuples, + tuples / secs, + insert_s, + drain_s, + insert_s + drain_s, + d_cpu, + d_rd, + d_wr, + wamp, + app, + alc, + open_secs, + viol, + errs, + swin, + swout, + majflt); + fclose(f); +} + +static void +run_phase(const phase_t *ph, bool last) +{ + int T = ph->threads > 0 ? ph->threads : opt.threads; + nworkers = T; + workers = calloc((size_t)T, sizeof(worker_t)); + if (!workers) { + fprintf(stderr, "FATAL: out of memory\n"); + exit(3); + } + void *(*fn)(void *) = NULL; + switch (ph->kind) { + case PH_LOAD: + fn = load_worker; + break; + case PH_WARM: + fn = warm_worker; + break; + case PH_POINT: + fn = point_worker; + break; + case PH_RANGE: + fn = range_worker; + break; + case PH_OW: + fn = ow_worker; + break; + } + bool fixed_count = ph->kind == PH_LOAD || ph->kind == PH_WARM; + atomic_store(&run_state, ST_WAIT); + atomic_store(&seq_counter, 0); + for (int i = 0; i < T; i++) { + workers[i].idx = i; + workers[i].ph = ph; + workers[i].rng = mix(opt.seed * 1000003 + (uint64_t)i * 7919 + 17 + + (uint64_t)ph->kind * 131 + (uint64_t)ph->dist); + if (pthread_create(&workers[i].th, NULL, fn, &workers[i])) { + fprintf(stderr, "FATAL: pthread_create\n"); + exit(3); + } + } + fprintf(stderr, + "phase %s: T=%d dist=%s %s\n", + ph->name, + T, + dist_name(ph->dist), + fixed_count ? "(fixed count)" : "(fixed duration)"); + wd_what = ph->name; + atomic_store(&wd_mode, WD_OPS); + snap_t a, b; + if (fixed_count) { + take_snap(&a); + atomic_store_explicit(&run_state, ST_MEASURE, memory_order_release); + for (int i = 0; i < T; i++) { + pthread_join(workers[i].th, NULL); + } + take_snap(&b); + } else { + atomic_store_explicit(&run_state, ST_WARMUP, memory_order_release); + sleep_secs(opt.warmup); + take_snap(&a); + atomic_store_explicit(&run_state, ST_MEASURE, memory_order_release); + sleep_secs(opt.secs); + atomic_store_explicit(&run_state, ST_STOP, memory_order_release); + take_snap(&b); + for (int i = 0; i < T; i++) { + pthread_join(workers[i].th, NULL); + } + } + atomic_store(&wd_mode, WD_IDLE); + + if (ph->kind == PH_OW) { + verify_overwrite_sample(ph, T); + } + + hist_t *lat = calloc(1, sizeof(hist_t)), + *startup = calloc(1, sizeof(hist_t)); + uint64_t ops = 0, tuples = 0, pt_ns = 0, pt_div = 0, dns = 0, dn = 0, + viol = 0, errs = 0, found = 0, notfound = 0; + for (int i = 0; i < T; i++) { + worker_t *w = &workers[i]; + hist_merge(lat, &w->lat); + hist_merge(startup, &w->startup); + ops += w->ops; + tuples += w->tuples; + pt_ns += w->pt_ns; + pt_div += w->pt_div; + dns += w->deinit_ns; + dn += w->deinit_n; + viol += w->violations; + errs += w->errors; + found += w->found; + notfound += w->notfound; + } + if (ph->kind == PH_WARM) { + tuples = ops; + if (ops != opt.n && !(opt.sabotage & 8)) { + fprintf(stderr, + "VIOLATION: warm scanned %" PRIu64 " tuples, expected %" PRIu64 + "\n", + ops, + opt.n); + viol++; + } + } + if (ph->kind == PH_POINT) { + fprintf(stderr, + " found=%" PRIu64 " notfound=%" PRIu64 "\n", + found, + notfound); + } + + double insert_s = (b.t_ns - a.t_ns) / 1e9, drain_s = 0; + snap_t da, dbs; + uint64_t app = 0, alc = 0; + bool closed = false; + if (ph->kind == PH_LOAD || last) { + take_snap(&da); + drain_s = db_close(); + take_snap(&dbs); + file_sizes(&app, &alc); + closed = true; + } + csv_row(ph, + T, + lat, + startup, + ops, + tuples, + pt_ns, + pt_div, + dns, + dn, + viol, + errs, + &a, + &b, + insert_s, + drain_s, + closed ? &da : NULL, + closed ? &dbs : NULL, + app, + alc); + if (viol) { + printf(" FINDING: %" PRIu64 + " correctness violations in %s (first: %s)\n", + viol, + ph->name, + workers[0].first_violation[0] ? workers[0].first_violation + : "see stderr"); + } + free(lat); + free(startup); + free(workers); + workers = NULL; + nworkers = 0; + if (closed && !last) { + db_open(false); + } +} + +/* ------------------------------------------------------------------------ */ +/* CLI */ +/* ------------------------------------------------------------------------ */ + +static void +usage(void) +{ + fprintf( + stderr, + "usage: kvbench --db PATH (--create|--open) --n N --cache-mib M " + "--disk-gib G --log 0|1\n" + " [--threads 6] [--mt-bg 1] [--normal-bg 4] [--memtable-mib 24]\n" + " [--secs 20] [--warmup 3] [--csv FILE] [--hang-dir DIR]\n" + " [--vtag S] [--stag S] [--otag S] [--rep R] [--seed S]\n" + " [--sabotage MASK] (1: point present->query absent; 2: point " + "neg->query present;\n" + " 4: range expects +1; 8: warm expects +1; 16: " + "point threads hang)\n" + " [--hang-secs 120] (no-progress timeout before gdb dump + exit 99\n" + " phase... load:random|load:seq warm point:uniform|zipf|seq|neg\n" + " range:uniform|zipf:L ow:uniform|zipf (suffix @T = " + "threads)\n"); + exit(2); +} + +static bool +parse_phase(const char *s, phase_t *ph) +{ + memset(ph, 0, sizeof(*ph)); + snprintf(ph->name, sizeof(ph->name), "%s", s); + char buf[64]; + snprintf(buf, sizeof(buf), "%s", s); + char *at = strchr(buf, '@'); + if (at) { + *at = 0; + ph->threads = atoi(at + 1); + } + char *a = strtok(buf, ":"), *b = strtok(NULL, ":"), *c = strtok(NULL, ":"); + if (!a) { + return false; + } + enum dist d = D_UNIFORM; + if (b) { + if (!strcmp(b, "uniform")) + d = D_UNIFORM; + else if (!strcmp(b, "zipf")) + d = D_ZIPF; + else if (!strcmp(b, "seq")) + d = D_SEQ; + else if (!strcmp(b, "neg")) + d = D_NEG; + else if (!strcmp(b, "random")) + d = D_RANDOM; + else + return false; + } + ph->dist = d; + if (!strcmp(a, "load")) { + ph->kind = PH_LOAD; + return b && (d == D_RANDOM || d == D_SEQ); + } else if (!strcmp(a, "warm")) { + ph->kind = PH_WARM; + return true; + } else if (!strcmp(a, "point")) { + ph->kind = PH_POINT; + return b != NULL; + } else if (!strcmp(a, "range")) { + ph->kind = PH_RANGE; + if (!b || !c || (d != D_UNIFORM && d != D_ZIPF)) { + return false; + } + ph->L = strtoull(c, NULL, 10); + return ph->L >= 1; + } else if (!strcmp(a, "ow")) { + ph->kind = PH_OW; + return b && (d == D_UNIFORM || d == D_ZIPF); + } + return false; +} + +int +main(int argc, char **argv) +{ + opt = (options){.threads = 6, + .mt_bg = 1, + .normal_bg = 4, + .memtable_mib = 24, + .secs = 20, + .warmup = 3, + .vtag = "?", + .stag = "?", + .otag = "?", + .rep = 1, + .seed = 42, + .hang_secs = 120, + .use_log = -1}; + int mode = -1; +#ifdef KVBENCH_SPACE_MODE + /* + * Space-check build only: open with use_stats, print insertion stats + * (which include allocator "Currently Allocated" extents, recomputed from + * the persisted allocator map at open), close. No phases, no writes. + */ + bool space_mode = false; +#endif + for (int i = 1; i < argc; i++) { + const char *s = argv[i]; +#define ARG(name) (!strcmp(s, name) && i + 1 < argc) + if (!strcmp(s, "--create")) + mode = 1; + else if (!strcmp(s, "--open")) + mode = 0; + else if (ARG("--db")) + opt.db = argv[++i]; + else if (ARG("--csv")) + opt.csv = argv[++i]; + else if (ARG("--hang-dir")) + opt.hang_dir = argv[++i]; + else if (ARG("--vtag")) + opt.vtag = argv[++i]; + else if (ARG("--stag")) + opt.stag = argv[++i]; + else if (ARG("--otag")) + opt.otag = argv[++i]; + else if (ARG("--rep")) + opt.rep = atoi(argv[++i]); + else if (ARG("--n")) + opt.n = strtoull(argv[++i], NULL, 0); + else if (ARG("--cache-mib")) + opt.cache_mib = strtoull(argv[++i], NULL, 0); + else if (ARG("--disk-gib")) + opt.disk_gib = strtoull(argv[++i], NULL, 0); + else if (ARG("--memtable-mib")) + opt.memtable_mib = strtoull(argv[++i], NULL, 0); + else if (ARG("--log")) + opt.use_log = atoi(argv[++i]); + else if (ARG("--threads")) + opt.threads = atoi(argv[++i]); + else if (ARG("--mt-bg")) + opt.mt_bg = atoi(argv[++i]); + else if (ARG("--normal-bg")) + opt.normal_bg = atoi(argv[++i]); + else if (ARG("--secs")) + opt.secs = atof(argv[++i]); + else if (ARG("--warmup")) + opt.warmup = atof(argv[++i]); + else if (ARG("--sabotage")) + opt.sabotage = strtoull(argv[++i], NULL, 0); + else if (ARG("--hang-secs")) + opt.hang_secs = strtoull(argv[++i], NULL, 0); + else if (ARG("--seed")) + opt.seed = strtoull(argv[++i], NULL, 0); +#ifdef KVBENCH_SPACE_MODE + else if (!strcmp(s, "space")) + space_mode = true; +#endif + else if (s[0] == '-') + usage(); + else { + if (opt.nphases >= MAX_PHASES + || !parse_phase(s, &opt.phases[opt.nphases])) + { + fprintf(stderr, "bad phase '%s'\n", s); + usage(); + } + opt.nphases++; + } +#undef ARG + } +#ifdef KVBENCH_SPACE_MODE + if (space_mode && opt.db && opt.cache_mib && opt.disk_gib + && opt.use_log >= 0) + { + default_data_config_init(&data_cfg); + platform_set_log_streams(stdout, stderr); + db_open(false); + splinterdb_stats_print_insertion(kvs); + fflush(stdout); + db_close(); + uint64_t app, alc; + file_sizes(&app, &alc); + printf("space: db_apparent=%" PRIu64 " db_alloc=%" PRIu64 "\n", app, alc); + return 0; + } +#endif + if (!opt.db || mode < 0 || !opt.n || !opt.cache_mib || !opt.disk_gib + || opt.use_log < 0 || !opt.nphases || opt.threads < 1 + || opt.threads > MAX_WORKERS) + { + usage(); + } + for (int i = 0; i < opt.nphases; i++) { + if (opt.phases[i].threads > MAX_WORKERS) { + usage(); + } + } + prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY, 0, 0, 0); + setvbuf(stdout, NULL, _IOLBF, 0); + + default_data_config_init(&data_cfg); + perm_init(&load_perm, opt.n, opt.seed * 2 + 1); + perm_init(&zipf_perm, opt.n, opt.seed * 2 + 2); + zipf_init(&zipf, opt.n, 0.99); + generation = (uint64_t)time(NULL); + + printf("kvbench version=%s splinter=%s db=%s mode=%s n=%" PRIu64 + " cache_mib=%" PRIu64 " disk_gib=%" PRIu64 + " log=%d threads=%d mt_bg=%d normal_bg=%d memtable_mib=%" PRIu64 + " secs=%.1f warmup=%.1f rep=%d sabotage=%" PRIu64 "\n", + opt.vtag, + splinterdb_get_version(), + opt.db, + mode ? "create" : "open", + opt.n, + opt.cache_mib, + opt.disk_gib, + opt.use_log, + opt.threads, + opt.mt_bg, + opt.normal_bg, + opt.memtable_mib, + opt.secs, + opt.warmup, + opt.rep, + opt.sabotage); + + pthread_t wd; + pthread_create(&wd, NULL, watchdog_main, NULL); + + db_open(mode == 1); + for (int i = 0; i < opt.nphases; i++) { + run_phase(&opt.phases[i], i == opt.nphases - 1); + } + if (kvs) { + db_close(); + } + return 0; +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/main-build.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/main-build.json new file mode 100644 index 000000000..dc269eb85 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/main-build.json @@ -0,0 +1,193 @@ +{ + "tree": "/home/rob/projects/splinterdb", + "head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "library": "/home/rob/projects/splinterdb/build-main-7bb5fab/release/lib/libsplinterdb.a", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "binaries": { + "kvbench": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/include", + "-I", + "/home/rob/projects/splinterdb/src", + "-I", + "/home/rob/projects/splinterdb/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/kvbench.c", + "/home/rob/projects/splinterdb/build-main-7bb5fab/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/kvbench.c", + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9" + }, + "insert_benchmark": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_NO_LOG_GROUP_LIMIT", + "-I", + "/home/rob/projects/splinterdb/include", + "-I", + "/home/rob/projects/splinterdb/src", + "-I", + "/home/rob/projects/splinterdb/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/insert_benchmark.c", + "/home/rob/projects/splinterdb/build-main-7bb5fab/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/insert_benchmark" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/insert_benchmark", + "binary_sha256": "9c39506eca1e6dc3088f26df34ce0c3af5920baa767533552e4d723579947c6c", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/sources/insert_benchmark.c", + "source_sha256": "75acbda52b2a8949f1d61710a8d89250d4dac63b12ec9803b89170f9c2cc0b26" + } + }, + "source_files_sha256": { + "include/splinterdb/data.h": "6d8984e702da446c2fe145eb2f6a576ca040cfd5740733c405d0d7387b243ec2", + "include/splinterdb/default_data_config.h": "179386f3e224d322a9d7ac97afd929292994a1d890a3254705f9ebea525bce32", + "include/splinterdb/platform_linux/public_platform.h": "c5ded82c5b5bb5bbcedf06a66951eb98dc8892f7b3c4f8ce07c474db98ac68a0", + "include/splinterdb/public_platform.h": "4ce9da3fea9a54fb7bb613ec1458f11f043525efb7d0bc55182f7183dd4a4f6e", + "include/splinterdb/public_util.h": "4e8f6b30408fb4174894d4ba809acff4e96f559370c7749b65bb3724a853603f", + "include/splinterdb/splinterdb.h": "364073691c390663f6b732f57c3ee4bfecc9f68aca33ef6750d6106e04462dc8", + "src/PackedArray.c": "97b64eb0a4f550e55f3385d39d729a878fbc581909ea16ec3b8987a7f46604cc", + "src/PackedArray.h": "e46f854b487a8b130f0613efcab11836012d27c8aacf9ca9e76a2de1a7eb6874", + "src/allocator.c": "96c7f0a5e0808719cf114d267c84da1f057b5eed748b0672d60686d7f130b96f", + "src/allocator.h": "bc0621be7903d64e25059140005dafeea3f2a4a4ae0e658686501e52e4be934f", + "src/batch_rwlock.c": "c043d9e5ead75cd9e1266a6f51fb555012269f3822d7f0233dc2abbd962c3aae", + "src/batch_rwlock.h": "0ff5f0bd7fdf5f9a63ca2f77c387d574d8009ad726b47283c93566cf06bc0927", + "src/blob.c": "7d46871423a15992f697cabb363316183cb842a11c26017f8566267174e5ae45", + "src/blob.h": "8f68d2f67de8a9d81a422b28d47aa636cc7234fccc3dbabe59f2355bc361e93c", + "src/blob_build.c": "e484e9c14fd8a9b3c1d211485c35931c944fc66ac4ee78ea20ffc5b3f074b10b", + "src/blob_build.h": "38176881cf9159247e6f578bbc7abb725cfa969c493715df7d1388cb7cbfd119", + "src/btree.c": "f637081a194302c584cccfd8d717253817794b34bbca754bd2c9e597acc534b9", + "src/btree.h": "6ed244b6731c490f224c7b5fea7b5a9dc27eca1aa808e6dab6b44a0c7b511638", + "src/btree_private.h": "43693dd3048ae050b3358ccfc9b91371a3d9dff4f3d6d5c8b07445897dee09f6", + "src/cache.h": "d4c1e133d6423fff324d0ebcf5c95e13647c0022a3458935156746b90d816d6d", + "src/clockcache.c": "e0ef2e19a8280849df4ee80d9b8a54a0faaf56a13faefe7a91d54f5b7269f933", + "src/clockcache.h": "68f4e7b019d0440f85929bfc0da1891ce941318ee7dfd14a3d4d923e3d4e72d8", + "src/core.c": "70caf9f8d7794dea8d2aeec339883ccfcfdfc29ce6c2e1edd2d0353c4a3975ea", + "src/core.h": "7cf4d086f6d4dcaa4cbd161fef786dbe3cc502ced4512b7fe547c2aa2020462e", + "src/data_blob_build.c": "13475faa56bba94ee26ef7fec4e4da8a5045e0efe510a63291fd5e2697cb5d22", + "src/data_blob_build.h": "dd7f263a0b1138325b211c561ba7289183a2cc8472978cba3ca430295e11e5f3", + "src/data_internal.c": "0a42dd477aaa6ad6c4910841045c4c768ad8d248c0126c64c51035f53e769cde", + "src/data_internal.h": "98b4ebe77576c4cbb7dd9984ca5e297851175494062073f1a4c137c5afb8b587", + "src/default_data_config.c": "161352f5247978f3d57033818d05a79d0a1a0d0ffac2bc9f25f4f33eabca2227", + "src/histogram.c": "24e09dfabd6085021d8a06ef9b10f6e85b24914d95831be359df09f8a9729aff", + "src/histogram.h": "7ccfde6d3ada7ead18fe477c06a6b021db83ca8c041ab0998ca023b945a3c1c7", + "src/iterator.h": "2fcddeb9fa6bd49f4e3da933224219bdb830f9e2f11638ea1ed13eba534fa27f", + "src/log.h": "49a2b0aec91a1c40784b8b3b54f016ef5504c79492b1393674123e27656a09bb", + "src/log_data.h": "06ab1d39b77abb01ec66a901fa52354908be26db8daef52899f5c03a6db2462e", + "src/lookup_result.h": "7528b988a28ef91b6bd7a4fa10dfa51b9ce2475e801b3b6a8bf30adf88f1ef16", + "src/memtable.c": "7db3b4754497c73b4071c7e7e591cc76fc226197fb70d98fb46caf139b3953f4", + "src/memtable.h": "244cac7cf304df85a25c8bede2df106d6f840d65bbd1a3551b3d9f2ed9129fc1", + "src/merge.c": "df98421e597d1edeac871755f0473055b590791af8efad495575d20260e46a12", + "src/merge.h": "c17e2a0b6272ec52e6d1e80acd9df7dc85221f186e6c60e323c37b467dc5c41a", + "src/mini_allocator.c": "d9444da13fca2000f4128500b84be2b778300a4befc93f72e52b071b83c6491e", + "src/mini_allocator.h": "8d5242f5e4df5443baeab7a494a3c3b2f1491216abfd914c6dbc273370e45683", + "src/notification.c": "2f3bcda49d8ce35ad532fef8ec8f4aeca48bec280cdf6210c21d55cf2aa4a661", + "src/notification.h": "47eb73f5813d6f54658f339ce44d6a475f8f64f2b2eb6368261e839a090f083f", + "src/ondisk_ref.c": "2ace8cbbce3f80a54ff7762ad03a79a564665fa660394f1d3446235a3e40ed7b", + "src/ondisk_ref.h": "643325c8b4f16fab41d8c884c97bdfcf6c86e5178398508fa0f81b6b0606926d", + "src/pcq.h": "862cd346b0097014ccae797dc1463e7b3fc387f409319d5adcef7613b6f82f84", + "src/platform_linux/async.h": "41383b57e21f0f51d737749a4ce7c2c5330ac9363e742330625c9aedd4f13a52", + "src/platform_linux/laio.c": "0ce628806e6896070359a9e39d069f326f460ab697c6c17ff17d171614290881", + "src/platform_linux/laio.h": "f3987e22f1fd7a9d6b46c79f84e75559aec5410239a13d52a7a5343ba792b28e", + "src/platform_linux/platform.h": "f1c3d7c5bcd089ed7a7df129d3b8d00d7617de9a7aa6f9d1f7deeb739281e768", + "src/platform_linux/platform_assert.c": "112e803b6db03dd8c14a3bfc3f1139bbcc4b81f1d5e24a232b7b99ce89146c2d", + "src/platform_linux/platform_assert.h": "751dccceac9ce4a97d25d441b5216ced5ee5f079a13c9ccd43551b0994aa7cc2", + "src/platform_linux/platform_backtrace.h": "ef13ccbf4ae4747b0de244a37c026ad8769018d84f9fc4d6493db43d9714358c", + "src/platform_linux/platform_buffer.c": "b790d645b3dc5f638da871bf9611865dc700927f3b594a1d645b0350bd1bd027", + "src/platform_linux/platform_buffer.h": "72e016dbb19b8250378b3a5c79fbebf9c3d72a31d07a232f36431e04f0e4afa6", + "src/platform_linux/platform_condvar.c": "e595f69194accb3f3460d6fc9a4a9144bd7f495dbeb77f11bf3616dc0e9b03ec", + "src/platform_linux/platform_condvar.h": "a4ee02ae5069637bce4cae07f3ed9869925f17a9e6e1b04d69baedfc148fd493", + "src/platform_linux/platform_hash.h": "f69f5ed6218bfe62c62718b387e1c110e9fca46c07603d4d51ccc4fc782b09ab", + "src/platform_linux/platform_heap.c": "72bb2753f525e56f8c398988a787feb38f02efaa749e5d57cb476f967269c479", + "src/platform_linux/platform_heap.h": "921f3113d20b44c36817c284702e965e01769596ff012edd6ceaa70cbdbbd10a", + "src/platform_linux/platform_io.c": "875a7331f8bfa46e9a6961f50acb40b107d40883839c94def52a289a521c66be", + "src/platform_linux/platform_io.h": "69ae9556daf1c7a69e007abd494bf9c86ec81c08bca1e032663009cef95b2fac", + "src/platform_linux/platform_log.c": "318bd17c858475a3a91c5e9641b52808f7c4194e1c6fc9a7d801392b04c49538", + "src/platform_linux/platform_log.h": "569484386150fcfbf5dde722c5b8e34d67d65be3454e3e009cb9df41f11efea6", + "src/platform_linux/platform_machine.h": "c02745313c7268dbe99efa2b93dd4d37640b5a7aa2e88c78fd5b73742f252c7d", + "src/platform_linux/platform_mutex.c": "7ec1dd6a5397217d28fd85c10648d29a70bf66649f59f76d7387236031b15bf3", + "src/platform_linux/platform_mutex.h": "688991d6a7dc57a7a88f565541561344cdfbb1733ca6073f1655d6c436ed6544", + "src/platform_linux/platform_random.h": "184ac9da9c38a6c10275891be61ccb91beaf2234ebb9ce046d059cb16824d68f", + "src/platform_linux/platform_semaphore.h": "9fdb7cfeaf36c8bf9aa346a16466ed6bc50bea76fdb5722c86001563d55bedfe", + "src/platform_linux/platform_sleep.h": "34aeaea35133bf33e47eb1d9ace429297644e13bc100d90ebbd205bceb58d3ab", + "src/platform_linux/platform_sort.h": "d374788510f94a8c3b86ff6bfa1f6a3ff8ac33168eb26f6f7774984252f6d01e", + "src/platform_linux/platform_spinlock.h": "7b3f0409109f3341f2287fe7cd8fb69faac4a74b009a44c1eb83220f71e94de0", + "src/platform_linux/platform_status.h": "2749285ea1322e5c1639019b728fde4b87b0cae6f71a7fc16e420bd592d38c00", + "src/platform_linux/platform_string.h": "fa41faf9c6661c62ece783740250232709656b0a8677f542e390a5f3b0d834a6", + "src/platform_linux/platform_threads.c": "789465a2c6ac7ac501e5878f429cc0dd9b163145a522e247a4833e3de7af8f2a", + "src/platform_linux/platform_threads.h": "653e070115131756e6aaffc3c2764a6437fb39ff96df4afc8a01e6e751c9d622", + "src/platform_linux/platform_time.h": "f212a08cf8432693711cecfd389f9ed3d0a70e0c056dc062321bd11c4a8cbec7", + "src/platform_linux/platform_trace.c": "4a68e89f214920dd2762d072a0ad8e7c80d9b1733a64e4a7e3dd4bb04b91c352", + "src/platform_linux/platform_trace.h": "345f7283b01e5a9b4d5da7bea66ad44ff61f53155417c97990e23bb8fedadf33", + "src/platform_linux/platform_typed_alloc.h": "42c1d2bbd56c183d11768a5f43d440cf3c1bfe329c4352afa6acd3bc2b191a8e", + "src/platform_linux/platform_units.h": "1cb22a07b33ec8b58f9381438f480f561d12b838f315a117cc1f979e95b1ef65", + "src/platform_linux/platform_util.h": "4fa2702b3919eb1a7efcf29457f5ee9a3d7a7b4c38fed4606264562022fa2638", + "src/platform_linux/poison.h": "a32dfa445439eb08913e519117e7b11f72b2c3550577a74769ed935a792c42c5", + "src/platform_linux/shmalloc.c": "e4631d63a9db1ad58472ad3d034d081f2bb9a97f34c5898f7c848dda923727f8", + "src/platform_linux/shmalloc.h": "081a82473f6021f8ed628e79f8ae5dc7104988981f8c3b25749ea42c8158113d", + "src/prefetch.h": "9d55d9dad64e3795c81856bc38688936e4e74e488dfe2d29a1f5438ad000df42", + "src/rc_allocator.c": "a145d39af39caf9dec6f1e1e24b6606ed615587d8dee52a23e0df410b2d07545", + "src/rc_allocator.h": "65f01decb7244b8f6a6c9e7acb47d4ccb4f2f3f369461e70aff33211f03ee39b", + "src/routing_filter.c": "3e722a880217cc4f5831f162dd6b3853a5d0d3f67ca3d3421b2144650657f73a", + "src/routing_filter.h": "579a3a4cc00b9cb125479b23eaa93e4a8309a195fe4082d988b629d4b95a2c91", + "src/shard_log.c": "ed1fcdb429d4fc26382546ecbce04154910275494e9fef522cf34b647cf5e734", + "src/shard_log.h": "07c5867b959eefd35e3e9f08e6c015a594f6331850844590e9c2b1126cd2716b", + "src/splinterdb.c": "e054c2dc9224c7de8bdb63206bb5150947f853d903e3b8012ba46fe1a66dba5f", + "src/splinterdb_tests_private.h": "579031767b8869f49cf1e3fc94caeec13a1362dfd047782538df40b281c93414", + "src/superblock.c": "34dd4c724e53c98f170c024c70dd22dae0a466806c167134200f92128420dfba", + "src/superblock.h": "72cd00ce0a7c43d1c25905d6bd39a400fa3245c133a3f559c58ae34a962157f2", + "src/task.c": "35886e598b4464af16e76497917955d3d55c6c077fbcf80fb3fd2284c8ddc08d", + "src/task.h": "35405e04fbdcef9bc5c5d6ef43debc4b0889d6bc4d8d530649b0c45a34012825", + "src/trunk.c": "892373ec383717289a6220b4351b670637e7c41877a97eb0edcb27a9e4e270f2", + "src/trunk.h": "05c5c5313f5a42b68482d03b58d2f17081c6da468a0ce888b0b0b38cc069baf2", + "src/util.c": "969df408c4c393f73feb05dfa37ebe1ed8be627cef630061830e4992c4c1fd9c", + "src/util.h": "9fa455f5a6268fa4a6864d444849b0872391af00512932c1587363cfd38304ef", + "src/vector.h": "6855dd0430f1a868d098797cbd4f3453ab542b5f6fbe716c79871124bee200a9", + "src/writeback_set.c": "cb485716f57a3183ce82543848edaf61f6568c9ff0445ea5ff989f95c00848ff", + "src/writeback_set.h": "68d6f589d24719a09e2680b05a1eb7e36c22b1fab978768f5b54f958f667aec9" + }, + "library_build_configuration": { + "build-config.fb6bc8eeb8390429253883c09491f5bd": { + "sha256": "5aa1a945982ec32bed4c489a7d08bea5e92d78220a31d095780e5978b5b537ee", + "text": "BUILD_ROOT=build-main-7bb5fab\nMAKEFLAGS= -j6 --jobserver-auth=fifo:/tmp/splinter-main-7bb5fab-benchmark-baseline-libs-icr_rin0/GMfifo4 -- BUILD_MODE=release BUILD_ROOT=build-main-7bb5fab\nBUILD_MODE=release\nQT_ACCESSIBILITY=1\nCC = cc\nDEPFLAGS = -MMD -MP\nCFLAGS = -D_GNU_SOURCE -ggdb -Wall -pthread -Wfatal-errors -Werror -Wvla -DXXH_STATIC_LINKING_ONLY -fPIC -DSPLINTERDB_PLATFORM_DIR=platform_linux -O3 -ffast-math -flto -DPLATFORM_MEMORY_FAULT_INJECTION=0\nTEST_MEMORY_FAULT_DISABLED_CFLAGS = -DPLATFORM_MEMORY_FAULT_INJECTION_DISABLED_IN_THIS_FILE\nINCLUDE = -I include -I src -I src/platform_linux -I tests\nTARGET_ARCH =\nLD = cc\nLDFLAGS = -ggdb -pthread -O3 -ffast-math -flto\nLIBS = -lm -lpthread -laio -lxxhash\nAR = ar\n" + } + } +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/package.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/package.py new file mode 100644 index 000000000..d55883554 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/package.py @@ -0,0 +1,67 @@ +"""Produce compact per-project archives without fixtures or executables.""" +import csv,hashlib,json,shutil,subprocess +from pathlib import Path +COORD=Path(__file__).resolve().parent +OUT=Path('/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab') +DEST=COORD/'project-archives' +PROJECTS=['disk-format','filtered-ranges','log-group-limit','branch-prefix','lazy-merge','cache-directory','async-durable-barrier','cache-invalid-addresses'] +def sha(p):return hashlib.sha256(p.read_bytes()).hexdigest() +def write(p,v):p.write_text(json.dumps(v,indent=2)+'\n') +def readcsv(p): + with p.open() as f:return list(csv.DictReader(f)) +def writecsv(p,rs): + if not rs:return + with p.open('w') as f: + w=csv.DictWriter(f,fieldnames=list(rs[0]),lineterminator='\n');w.writeheader();w.writerows(rs) +subprocess.run(['python3',str(COORD/'report.py')],check=True) +summary=readcsv(COORD/'summary.csv') +expected={'fixtures':12,'resident':23,'disk':23,'prefix-resident-common':4,'prefix-resident-random':4,'prefix-disk-common':4,'prefix-disk-random':4,'insert':31,'hash':4,'barrier':24,'repeat-warmup':1,'repeat-cache-invalid-addresses':4,'repeat-directory-bucket-atomic':4,'repeat-async-durable-barrier':4} +for mode,count in expected.items(): + manifest=json.loads((OUT/mode/'manifest.json').read_text()) + assert manifest['complete'] and len(manifest['runs'])==count and all(r['verified'] and r['returncode']==0 for r in manifest['runs']),mode + +for project in PROJECTS: + profiles=[project] if project!='cache-directory' else ['directory-'+x for x in ['direct','bucket','bucket-simd','bucket-atomic']] + versions=profiles+(['log-disabled','log-default'] if project=='log-group-limit' else []) + path=DEST/project;path.mkdir(parents=True,exist_ok=False) + own=[r for r in summary if r['version'] in versions] + for r in own:r.update(scope='new main versus updated feature; merge cost not isolated') + writecsv(path/'summary.csv',own) + source_names={'kvbench.c','insert_benchmark.c'} + fixtures=json.loads((OUT/'fixtures/manifest.json').read_text());assert fixtures['complete'] and all(r['verified'] for r in fixtures['runs']) + fixtures['runs']=[r for r in fixtures['runs'] if r['profile'] in ['main',*profiles] and (('ordinary' not in r['label']) if project=='branch-prefix' else ('ordinary' in r['label']))] + fixture_out=path/'fixtures';fixture_out.mkdir();write(fixture_out/'manifest.json',fixtures) + for record in fixtures['runs']: + for suffix in ['.csv','.log']: + file=OUT/'fixtures'/(record['label']+suffix);shutil.copyfile(file,fixture_out/file.name) + if project=='branch-prefix':source_names.add('prefix_kvbench.c') + if project=='async-durable-barrier':source_names.add('barrier_bench.c') + if project=='filtered-ranges':source_names.add('hash_range_benchmark.c') + for name in source_names:shutil.copyfile(COORD/'sources'/name,path/name) + for name in ['run_matrix.py','build_all.py','report.py','package.py','preparation.json','feature-summaries.json','host.json','run_matrix-fixtures.py','control-drift.csv','repeat.py','audit.json','audit.py','report.md','benchmark-jobs.json','run_job.py']:shutil.copyfile(COORD/name,path/name) + for profile in ['main',*profiles]: + build=json.loads((COORD/(profile+'-build.json')).read_text());build['binaries']={name:value for name,value in build['binaries'].items() if name+'.c' in source_names};write(path/(profile+'-build.json'),build) + tests=[] + for folder in OUT.iterdir(): + if not folder.is_dir() or folder.name=='fixtures' or not (folder/'manifest.json').exists():continue + m=json.loads((folder/'manifest.json').read_text()) + assert m.get('complete') and m['runs'] and all(r['verified'] for r in m['runs']),folder + selected=[r for r in m['runs'] if r['profile'] in profiles] + if folder.name=='repeat-warmup' and project in ['async-durable-barrier','cache-invalid-addresses','cache-directory']:selected=m['runs'] + if not selected:continue + target=path/folder.name;target.mkdir() + rr=[r for r in m['runs'] if r['profile'] in profiles+['main']] + m['runs']=rr;write(target/'manifest.json',m) + for r in rr: + for file in folder.glob(r['label']+'.*'): + if file.suffix in ['.csv','.log']:shutil.copyfile(file,target/file.name) + if (folder/'results.csv').exists(): + rows=readcsv(folder/'results.csv') + if 'version' in rows[0]:rows=[r for r in rows if r['version'] in versions+['main']] + elif 'label' in rows[0]:rows=[r for r in rows if r['label'] in [x['label'] for x in rr]] + writecsv(target/'results.csv',rows) + tests.append(dict(workload=folder.name,runs=len(rr),feature_runs=sum(r['profile'] in profiles for r in rr),warmup_runs=len(rr) if folder.name=='repeat-warmup' else 0,verified=sum(r['verified'] for r in rr),complete=m['complete'])) + assert own and tests,project + write(path/'status.json',dict(project=project,scope='new main versus updated feature; merge cost not isolated',workloads=tests,reported_errors=sum(int(r['errors']) for r in own),reported_violations=sum(int(r['violations']) for r in own),all_runs_verified=bool(tests) and all(x['verified']==x['runs'] and x['complete'] for x in tests))) + inventory={str(p.relative_to(path)):sha(p) for p in path.rglob('*') if p.is_file()};write(path/'inventory.json',inventory) +print(DEST) diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/preparation.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/preparation.json new file mode 100644 index 000000000..2a6d4b750 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/preparation.json @@ -0,0 +1,20 @@ +{ + "main": "7bb5fab305b9c6179e31f75998dabea833537a08", + "sources": { + "hash_range_benchmark.c": "bed236eceff4c7c1ce1570f55ddf35aac1858f53520c9447f24f5a32c055b6fe", + "kvbench.c": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "prefix_kvbench.c": "127357ddacf129b1a15d2c8e5d60f928d28a101b2ea414ff6d7fff4c2e386d51", + "barrier_bench.c": "1784a5d6d1b627858a0c8a9a6f718cd7d002a562e680d5cbdde23829167c015c", + "insert_benchmark.c": "75acbda52b2a8949f1d61710a8d89250d4dac63b12ec9803b89170f9c2cc0b26" + }, + "source_notes": { + "kvbench.c": "Byte-identical copy of campaign baseline/benchmarks/kvbench.c", + "prefix_kvbench.c": "Frozen campaign prefix workload generated from the same baseline kvbench, 64-byte common/random keys", + "insert_benchmark.c": "Frozen log-group-limit branch harness, with a compile-time two-argument blocking durable-barrier adapter for async branch", + "barrier_bench.c": "Frozen async-durable-barrier branch common harness with compile-time main/candidate API adapters", + "hash_range_benchmark.c": "Frozen filtered-ranges branch public-API probe; candidate routing hash versus main application filtering" + }, + "query_order": "Main control, all variants forward, main control, all variants reverse, main control", + "scope": "Screen feature branches freshly based on main; historical measurements remain historical. This does not isolate merge overhead.", + "expected_minutes": 35 +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/repeat.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/repeat.py new file mode 100644 index 000000000..a6045a183 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/repeat.py @@ -0,0 +1,19 @@ +"""Nearby warmed ABBA controls; invoke under run_job.py benchmark.""" +import argparse +import run_matrix as r +p=argparse.ArgumentParser();p.add_argument('profile',choices=['warmup','cache-invalid-addresses','directory-bucket-atomic','async-durable-barrier']);a=p.parse_args() +def interrupted(signum,frame):raise InterruptedError(f'benchmark interrupted by signal {signum}') +for sig in [r.signal.SIGTERM,r.signal.SIGHUP]:r.signal.signal(sig,interrupted) +original=r.kvargs +def longer(*args,**kwargs): + kwargs['secs']=10 + return original(*args,**kwargs) +r.kvargs=longer +folder,m=r.newmode('repeat-'+a.profile) +m['runner_sha256']=r.sha(__file__) +m['shared_runner_sha256']=r.sha(r.__file__) +m['scope']='Nearby main ABBA controls after broad screening; all earlier samples retained' +if a.profile=='warmup':m['scope']='Recorded workload warmup before nearby ABBA comparisons; retained separately from their samples' +phases=['warm']+(['point:uniform@1'] if a.profile=='cache-invalid-addresses' else [])+['point:uniform@8','range:uniform:1@8'] +for i,name in enumerate(['main'] if a.profile=='warmup' else ['main',a.profile,a.profile,'main']):r.query_one(folder,m,i,name,'resident',phases) +m['complete']=True;r.save(folder/'manifest.json',m) diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/report.md b/benchmarks/results-20261013/main-7bb5fab/benchmarks/report.md new file mode 100644 index 000000000..c6d09c230 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/report.md @@ -0,0 +1,237 @@ +# Fresh main integration benchmark screening + +These compare current feature branches with main 7bb5fab. They do not isolate the cost of merging main. Short runs are screening evidence, not proof of performance neutrality. All databases are on verified real disk with O_DIRECT; copied fixtures are fsynced before opening. Query samples use serial-built immutable fixtures, with separate equivalent fixtures for incompatible formats. Main controls bracket forward and reverse variant sweeps. + +| Workload | Phase | Branch | Change | Sample range overlap | +| --- | --- | --- | ---: | --- | +| resident | point:uniform@1 | disk-format | -1.21% | True | +| resident | point:uniform@1 | filtered-ranges | -1.11% | True | +| resident | point:uniform@1 | log-group-limit | -3.24% | True | +| resident | point:uniform@1 | lazy-merge | -2.99% | True | +| resident | point:uniform@1 | async-durable-barrier | -3.58% | True | +| resident | point:uniform@1 | cache-invalid-addresses | -4.60% | False | +| resident | point:uniform@1 | directory-direct | -3.66% | True | +| resident | point:uniform@1 | directory-bucket | -13.61% | False | +| resident | point:uniform@1 | directory-bucket-simd | -14.75% | False | +| resident | point:uniform@1 | directory-bucket-atomic | -11.59% | False | +| resident | point:uniform@8 | disk-format | -1.92% | True | +| resident | point:uniform@8 | filtered-ranges | -1.37% | True | +| resident | point:uniform@8 | log-group-limit | -3.94% | True | +| resident | point:uniform@8 | lazy-merge | -4.99% | True | +| resident | point:uniform@8 | async-durable-barrier | -5.29% | True | +| resident | point:uniform@8 | cache-invalid-addresses | -7.08% | True | +| resident | point:uniform@8 | directory-direct | -6.43% | True | +| resident | point:uniform@8 | directory-bucket | -25.62% | False | +| resident | point:uniform@8 | directory-bucket-simd | -25.20% | False | +| resident | point:uniform@8 | directory-bucket-atomic | -11.45% | False | +| resident | range:uniform:1@1 | disk-format | +1.54% | True | +| resident | range:uniform:1@1 | filtered-ranges | -0.07% | True | +| resident | range:uniform:1@1 | log-group-limit | -2.20% | True | +| resident | range:uniform:1@1 | lazy-merge | +4.48% | True | +| resident | range:uniform:1@1 | async-durable-barrier | -0.38% | True | +| resident | range:uniform:1@1 | cache-invalid-addresses | -2.25% | True | +| resident | range:uniform:1@1 | directory-direct | -0.18% | True | +| resident | range:uniform:1@1 | directory-bucket | -7.11% | False | +| resident | range:uniform:1@1 | directory-bucket-simd | -6.10% | False | +| resident | range:uniform:1@1 | directory-bucket-atomic | -5.32% | False | +| resident | range:uniform:1@8 | disk-format | -4.07% | True | +| resident | range:uniform:1@8 | filtered-ranges | -3.38% | True | +| resident | range:uniform:1@8 | log-group-limit | -4.75% | True | +| resident | range:uniform:1@8 | lazy-merge | +1.35% | True | +| resident | range:uniform:1@8 | async-durable-barrier | -6.06% | False | +| resident | range:uniform:1@8 | cache-invalid-addresses | -4.95% | True | +| resident | range:uniform:1@8 | directory-direct | -4.39% | True | +| resident | range:uniform:1@8 | directory-bucket | -18.35% | False | +| resident | range:uniform:1@8 | directory-bucket-simd | -18.65% | False | +| resident | range:uniform:1@8 | directory-bucket-atomic | -7.65% | False | +| resident | range:uniform:10000@8 | disk-format | -2.41% | True | +| resident | range:uniform:10000@8 | filtered-ranges | +1.43% | True | +| resident | range:uniform:10000@8 | log-group-limit | -1.49% | True | +| resident | range:uniform:10000@8 | lazy-merge | -5.38% | True | +| resident | range:uniform:10000@8 | async-durable-barrier | -4.49% | True | +| resident | range:uniform:10000@8 | cache-invalid-addresses | -2.56% | True | +| resident | range:uniform:10000@8 | directory-direct | -8.56% | False | +| resident | range:uniform:10000@8 | directory-bucket | -6.41% | True | +| resident | range:uniform:10000@8 | directory-bucket-simd | -6.44% | False | +| resident | range:uniform:10000@8 | directory-bucket-atomic | -6.85% | True | +| disk | point:uniform@8 | disk-format | -0.81% | True | +| disk | point:uniform@8 | filtered-ranges | -3.82% | True | +| disk | point:uniform@8 | log-group-limit | -0.25% | True | +| disk | point:uniform@8 | lazy-merge | -0.22% | True | +| disk | point:uniform@8 | async-durable-barrier | -2.05% | True | +| disk | point:uniform@8 | cache-invalid-addresses | -2.24% | True | +| disk | point:uniform@8 | directory-direct | -1.93% | True | +| disk | point:uniform@8 | directory-bucket | -2.54% | True | +| disk | point:uniform@8 | directory-bucket-simd | -2.53% | True | +| disk | point:uniform@8 | directory-bucket-atomic | -1.32% | True | +| disk | range:uniform:1@8 | disk-format | -0.78% | True | +| disk | range:uniform:1@8 | filtered-ranges | -1.67% | True | +| disk | range:uniform:1@8 | log-group-limit | -0.59% | True | +| disk | range:uniform:1@8 | lazy-merge | -0.98% | True | +| disk | range:uniform:1@8 | async-durable-barrier | -2.40% | True | +| disk | range:uniform:1@8 | cache-invalid-addresses | -2.34% | True | +| disk | range:uniform:1@8 | directory-direct | -3.11% | True | +| disk | range:uniform:1@8 | directory-bucket | -2.79% | True | +| disk | range:uniform:1@8 | directory-bucket-simd | -3.56% | True | +| disk | range:uniform:1@8 | directory-bucket-atomic | -2.25% | True | +| disk | range:uniform:10000@8 | disk-format | -0.41% | True | +| disk | range:uniform:10000@8 | filtered-ranges | -1.80% | True | +| disk | range:uniform:10000@8 | log-group-limit | +0.11% | True | +| disk | range:uniform:10000@8 | lazy-merge | -1.24% | True | +| disk | range:uniform:10000@8 | async-durable-barrier | -2.13% | True | +| disk | range:uniform:10000@8 | cache-invalid-addresses | -2.08% | True | +| disk | range:uniform:10000@8 | directory-direct | -2.14% | True | +| disk | range:uniform:10000@8 | directory-bucket | -2.08% | True | +| disk | range:uniform:10000@8 | directory-bucket-simd | -3.48% | True | +| disk | range:uniform:10000@8 | directory-bucket-atomic | -1.09% | True | +| prefix-resident-common | point:uniform@8 | branch-prefix | +10.61% | False | +| prefix-resident-common | range:uniform:1@8 | branch-prefix | +2.87% | False | +| prefix-resident-common | range:uniform:10000@8 | branch-prefix | +0.34% | True | +| prefix-resident-common | ow:uniform@8 | branch-prefix | +9.38% | False | +| prefix-resident-random | point:uniform@8 | branch-prefix | -0.30% | True | +| prefix-resident-random | range:uniform:1@8 | branch-prefix | +2.67% | True | +| prefix-resident-random | range:uniform:10000@8 | branch-prefix | -10.88% | False | +| prefix-resident-random | ow:uniform@8 | branch-prefix | -1.52% | True | +| prefix-disk-common | point:uniform@8 | branch-prefix | -2.69% | True | +| prefix-disk-common | range:uniform:1@8 | branch-prefix | +0.11% | True | +| prefix-disk-common | range:uniform:10000@8 | branch-prefix | +14.37% | False | +| prefix-disk-random | point:uniform@8 | branch-prefix | -5.57% | True | +| prefix-disk-random | range:uniform:1@8 | branch-prefix | -6.30% | True | +| prefix-disk-random | range:uniform:10000@8 | branch-prefix | -4.11% | True | +| repeat-directory-bucket-atomic | point:uniform@8 | directory-bucket-atomic | -6.18% | True | +| repeat-directory-bucket-atomic | range:uniform:1@8 | directory-bucket-atomic | -3.89% | True | +| repeat-async-durable-barrier | point:uniform@8 | async-durable-barrier | -0.52% | True | +| repeat-async-durable-barrier | range:uniform:1@8 | async-durable-barrier | -2.28% | True | +| repeat-cache-invalid-addresses | point:uniform@1 | cache-invalid-addresses | -4.54% | False | +| repeat-cache-invalid-addresses | point:uniform@8 | cache-invalid-addresses | -3.64% | True | +| repeat-cache-invalid-addresses | range:uniform:1@8 | cache-invalid-addresses | +0.03% | True | +| insert | WAL1 | disk-format | -0.88% | True | +| insert | WAL1 | filtered-ranges | -1.02% | False | +| insert | WAL1 | lazy-merge | -0.49% | True | +| insert | WAL1 | async-durable-barrier | -1.08% | True | +| insert | WAL1 | cache-invalid-addresses | -3.00% | False | +| insert | WAL1 | directory-direct | -1.91% | False | +| insert | WAL1 | directory-bucket | -5.46% | False | +| insert | WAL1 | directory-bucket-simd | -3.79% | False | +| insert | WAL1 | directory-bucket-atomic | -3.32% | False | +| insert | WAL1 | log-disabled | -1.34% | True | +| insert | WAL1 | log-default | -42.70% | False | +| insert | WAL0 | cache-invalid-addresses | +6.27% | True | +| insert | WAL0 | directory-bucket-atomic | +6.39% | False | + +Detailed throughput ranges, CPU/op, I/O/op and p99 are in summary.csv. Raw commands, source/library/executable hashes, verification and logs remain in the per-workload manifest and adjacent artifacts. + +{ + "insert": { + "runs": 31, + "complete": true, + "verified": 31 + }, + "repeat-warmup": { + "runs": 1, + "complete": true, + "verified": 1 + }, + "barrier": { + "runs": 24, + "complete": true, + "verified": 24 + }, + "repeat-directory-bucket-atomic": { + "runs": 4, + "complete": true, + "verified": 4 + }, + "fixtures": { + "runs": 12, + "complete": true, + "verified": 12 + }, + "disk": { + "runs": 23, + "complete": true, + "verified": 23 + }, + "repeat-async-durable-barrier": { + "runs": 4, + "complete": true, + "verified": 4 + }, + "prefix-resident-common": { + "runs": 4, + "complete": true, + "verified": 4 + }, + "hash": { + "runs": 4, + "complete": true, + "verified": 4 + }, + "resident": { + "runs": 23, + "complete": true, + "verified": 23 + }, + "repeat-cache-invalid-addresses": { + "runs": 4, + "complete": true, + "verified": 4 + }, + "prefix-disk-random": { + "runs": 4, + "complete": true, + "verified": 4 + }, + "prefix-disk-common": { + "runs": 4, + "complete": true, + "verified": 4 + }, + "prefix-resident-random": { + "runs": 4, + "complete": true, + "verified": 4 + } +} + +## Main control drift + +All controls are retained. Large variation in these unchanged-main samples prevents attributing small feature differences to code. The measured cause of this variation is unknown. Nearby ABBA repeats are separate evidence; they do not replace these results. + +| Regime | Phase | First ops/s | Middle ops/s | Last ops/s | Middle vs first | Last vs first | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| resident | point:uniform@1 | 1,261,757.3 | 1,152,531.0 | 1,148,133.5 | -8.66% | -9.01% | +| resident | point:uniform@8 | 6,445,634.1 | 5,270,841.2 | 5,463,995.8 | -18.23% | -15.23% | +| resident | range:uniform:1@1 | 114,690.1 | 105,447.5 | 105,112.6 | -8.06% | -8.35% | +| resident | range:uniform:1@8 | 600,376.6 | 532,530.7 | 549,552.3 | -11.30% | -8.47% | +| resident | range:uniform:10000@8 | 4,305.3 | 3,785.8 | 3,929.7 | -12.07% | -8.72% | +| disk | point:uniform@8 | 153,061.5 | 162,899.9 | 168,110.0 | +6.43% | +9.83% | +| disk | range:uniform:1@8 | 63,458.1 | 63,625.4 | 64,975.4 | +0.26% | +2.39% | +| disk | range:uniform:10000@8 | 841.4 | 835.2 | 840.0 | -0.74% | -0.17% | + +## Interpretation and limits + +These samples support feature tradeoffs, not an isolated estimate of merge overhead. There are two samples per candidate and two or three main controls; non-overlap is a descriptive range check, not a statistical confidence interval. Unchanged main controls drifted substantially, including in nearby repeats. The cause is unproven. + +The invalid-address checks reproduce a roughly 4.5% resident single-thread point-query loss: broad screening -4.60%, nearby warmed ABBA -4.54% with non-overlapping sample ranges. Its repeated eight-thread point query (-3.64%) overlaps main, and repeated one-result/eight-thread range query (+0.03%) does not reproduce the initial apparent loss. Atomic directory nearby repeats (-6.18% point and -3.89% short range) and async-barrier ordinary-query repeats (-0.52% point and -2.28% short range) all overlap main; their smaller effects remain inconclusive. The scalar/SIMD directory variants show larger resident feature costs in the broad screen. These are comparisons of the whole features to main, not newly introduced merge regressions. + +Prefix compression improves several common-prefix workloads, but the random-key resident long-range screen loses 10.88% with non-overlapping samples. The default 64-page automatic log-group durability threshold loses 42.7% insert throughput in the short WAL-enabled screen; this is the cost of added automatic durability versus main without automatic barriers. Disabling the threshold gives -1.3% with overlapping ranges. These tradeoffs require workload-specific judgment, not a blanket performance-neutral claim. + +All timed processes passed their declared verification. Query runs validate keys/values/ranges and close successfully. Insert-only runs validate API returns, positive insert counts and close, but do not perform readback. Barrier runs crash after the final completed barrier, reopen and verify every key/value. Hash-filter runs compare checksums and expected output counts. Every measured ordinary resident and focused-repeat query phase has zero physical read bytes. The retained pre-repeat warmup runs point and one-result range queries at eight threads for 10 seconds each; it is not pooled with comparison controls. + +Separate correctness testing reproduced an inherited small-cache async progress stall on main and invalid-addresses under restricted CPU scheduling. That limitation is documented in the branch correctness evidence; it was not repaired in these merges, and no benchmark run stalled. + +Barrier measurements are short and show substantial run-to-run I/O variation in some configurations. In particular, WAL-enabled/no-background-worker candidate sync spans about 60k to 194k updates/s, while main spans 63k to 65k; its +99% mean alone is not a stable speedup estimate. Async launch latency is distinct from durability completion. These runs use depth 1: completion is observed by waiting or polling before the next insert batch. Greater-depth harness modes may include intervening inserts in observed completion, but were not measured here. + +| Barrier config | Candidate mode | Main updates/s range | Candidate updates/s range | Mean change | Mean launch p99 us | Mean observed completion p99 us | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| wal0-workers0 | sync | 31,072–40,897 | 57,552–58,998 | +61.95% | 15769.34 | 15769.34 | +| wal0-workers0 | async | 31,072–40,897 | 59,568–60,181 | +66.39% | 0.98 | 14594.80 | +| wal0-workers1 | sync | 32,686–33,069 | 32,567–33,066 | -0.19% | 21671.51 | 21671.51 | +| wal0-workers1 | async | 32,686–33,069 | 31,925–33,322 | -0.77% | 2.90 | 23050.30 | +| wal1-workers0 | sync | 63,288–64,560 | 60,090–194,153 | +98.86% | 4118.68 | 4118.68 | +| wal1-workers0 | async | 63,288–64,560 | 200,298–273,650 | +270.71% | 3.51 | 2190.87 | +| wal1-workers1 | sync | 61,607–63,583 | 67,545–69,246 | +9.27% | 4940.58 | 4940.58 | +| wal1-workers1 | async | 61,607–63,583 | 78,489–80,079 | +26.66% | 6.17 | 4128.32 | + +Hash-filter API versus application filtering: mean latency changes for limits 1/2/10/100 are -5.70%/-16.71%/+0.56%/-9.59%; output checksums match. Full ranges, visited tuple counts, CPU/I/O and provenance are retained in the raw results and feature-summaries.json. diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/report.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/report.py new file mode 100644 index 000000000..eb2212fd6 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/report.py @@ -0,0 +1,127 @@ +import csv,json,statistics +from pathlib import Path +OUT=Path('/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab') +COORD=Path(__file__).resolve().parent +allrows=[] +def read(path): + with path.open() as f:return list(csv.DictReader(f)) +def verified_manifest(folder): + p=folder/'manifest.json' + if not p.exists():return None + m=json.loads(p.read_text()) + if not m.get('complete'):return None + assert m['runs'] and all(r.get('verified') and r.get('returncode')==0 for r in m['runs']),folder + return m +def summary(mode,group,version,rs,base,rate='ops_s'): + rates=[float(r[rate]) for r in rs];baseline=[float(r[rate]) for r in base] + avg=statistics.fmean + return dict(workload=mode,phase=group,version=version,samples=len(rs),ops_s_mean=avg(rates),ops_s_min=min(rates),ops_s_max=max(rates),baseline_ops_s_mean=avg(baseline),baseline_ops_s_min=min(baseline),baseline_ops_s_max=max(baseline),delta_pct=100*(avg(rates)/avg(baseline)-1),ranges_overlap=max(min(rates),min(baseline))<=min(max(rates),max(baseline))) +for mode in ['resident','disk','prefix-resident-common','prefix-resident-random','prefix-disk-common','prefix-disk-random',*[p.name for p in OUT.glob('repeat-*')]]: + folder=OUT/mode + if not (folder/'results.csv').exists() or not (folder/'manifest.json').exists():continue + m=verified_manifest(folder) + if m is None:continue + rr=read(folder/'results.csv') + for run in m['runs']: + command=run['command']; order=command[command.index('--otag')+1] + own=[r for r in rr if r['order']==order] + phases=command[command.index('warm'):] + assert len(own)==len(phases) and [r['phase'] for r in own]==phases and all(r['version']==run['profile'] for r in own),run['label'] + assert len(rr)==sum(len(run['command'][run['command'].index('warm'):]) for run in m['runs']) + for phase in dict.fromkeys(r['phase'] for r in rr if r['phase']!='warm'): + bas=[r for r in rr if r['version']=='main' and r['phase']==phase] + if mode=='resident' or mode.startswith('repeat-'):assert all(int(r['rd_bytes'])==0 for r in rr if r['phase']!='warm'), 'resident queries performed physical reads' + for version in dict.fromkeys(r['version'] for r in rr): + rs=[r for r in rr if r['version']==version and r['phase']==phase] + if not rs or not bas:continue + s=summary(mode,phase,version,rs,bas) + s.update(errors=sum(int(r['errors']) for r in rs),violations=sum(int(r['violations']) for r in rs),cpu_us_op=statistics.fmean(1e6*float(r['cpu_s'])/int(r['ops']) for r in rs),read_B_op=statistics.fmean(float(r['rd_B_op']) for r in rs),write_B_op=statistics.fmean(float(r['wr_B_op']) for r in rs),p99_us=statistics.fmean(float(r['p99_us']) for r in rs));allrows.append(s) +folder=OUT/'insert' +if (folder/'results.csv').exists() and verified_manifest(folder): + m=verified_manifest(folder);rr=read(folder/'results.csv') + assert len(rr)==len(m['runs']) + assert {f"wal{r['wal']}-{int(r['index']):02d}-{r['version']}" for r in rr}=={r['label'] for r in m['runs']} + for wal in dict.fromkeys(r['wal'] for r in rr): + bas=[r for r in rr if r['version']=='main' and r['wal']==wal] + for version in dict.fromkeys(r['version'] for r in rr if r['wal']==wal): + rs=[r for r in rr if r['version']==version and r['wal']==wal];s=summary('insert','WAL'+wal,version,rs,bas) + s.update(errors=sum(int(r['errors']) for r in rs),violations=sum(int(r['violations']) for r in rs));s.update({k:statistics.fmean(float(r[k]) for r in rs) for k in ['cpu_us_op','read_B_op','write_B_op','p99_us']});allrows.append(s) +if allrows: + with (COORD/'summary.csv').open('w') as f: + w=csv.DictWriter(f,fieldnames=list(allrows[0]),lineterminator='\n');w.writeheader();w.writerows(allrows) +lines=['# Fresh main integration benchmark screening','','These compare current feature branches with main 7bb5fab. They do not isolate the cost of merging main. Short runs are screening evidence, not proof of performance neutrality. All databases are on verified real disk with O_DIRECT; copied fixtures are fsynced before opening. Query samples use serial-built immutable fixtures, with separate equivalent fixtures for incompatible formats. Main controls bracket forward and reverse variant sweeps.','', '| Workload | Phase | Branch | Change | Sample range overlap |','| --- | --- | --- | ---: | --- |'] +for r in allrows: + if r['version']=='main':continue + lines.append(f"| {r['workload']} | {r['phase']} | {r['version']} | {r['delta_pct']:+.2f}% | {r['ranges_overlap']} |") +counts={} +for p in OUT.glob('*/manifest.json'): + m=json.loads(p.read_text());counts[p.parent.name]=dict(runs=len(m['runs']),complete=m.get('complete',False),verified=sum(r.get('verified',False) for r in m['runs'])) +(COORD/'counts.json').write_text(json.dumps(counts,indent=2)+'\n') +lines+=['','Detailed throughput ranges, CPU/op, I/O/op and p99 are in summary.csv. Raw commands, source/library/executable hashes, verification and logs remain in the per-workload manifest and adjacent artifacts.','',json.dumps(counts,indent=2)] +(COORD/'report.md').write_text('\n'.join(lines)+'\n') +print(json.dumps(counts,indent=2)) +feature_summaries={} +p=OUT/'hash/manifest.json' +if p.exists() and verified_manifest(p.parent): + m=verified_manifest(p.parent);hh=[] + for limit in ['1','2','10','100']: + groups={profile:[r for run in m['runs'] if run['profile']==profile and run['verified'] for r in run['rows'] if r['limit']==limit] for profile in ['main','filtered-ranges']} + if not all(groups.values()):continue + mean=lambda name:statistics.fmean(float(r['ns_per_query']) for r in groups[name]) + for profile,rr in groups.items(): + costs=[float(r['ns_per_query']) for r in rr] + hh.append(dict(profile=profile,limit=int(limit),samples=len(rr),ns_per_query_mean=statistics.fmean(costs),ns_per_query_min=min(costs),ns_per_query_max=max(costs),latency_delta_main_pct=100*(mean(profile)/mean('main')-1),visited_mean=statistics.fmean(int(r['visited']) for r in rr),verified=True,verification_kind='successful process with harness assertions and matching result checksums')) + feature_summaries['hash']=hh +p=OUT/'barrier/results.csv' +if p.exists() and verified_manifest(p.parent): + m=verified_manifest(p.parent);rr=read(p);bb=[] + assert len(rr)==len(m['runs']) and {r['label'] for r in rr}=={r['label'] for r in m['runs']} + for wal in ['0','1']: + for workers in ['0','1']: + cases=[('main','sync'),('async-durable-barrier','sync'),('async-durable-barrier','async')] + base=[r for r in rr if r['log']==wal and r['workers']==workers and '-main-' in r['label']] + if not base:continue + for profile,mode in cases: + rs=[r for r in rr if r['log']==wal and r['workers']==workers and f'-{profile}-' in r['label'] and r['mode']==mode] + if not rs:continue + s=summary('barrier',f'wal{wal}-workers{workers}-{mode}',profile,rs,base,rate='updates_s') + s.update(launch_p99_us=statistics.fmean(float(r['launch_p99_us']) for r in rs),completion_p99_us=statistics.fmean(float(r['observed_completion_p99_us']) for r in rs),verified=all(int(r['verified'])==1 for r in rs),verification_kind='crash after completed final barrier, reopen and full key/value readback',cpu_us_update=statistics.fmean(1e6*float(r['cpu_s'])/(int(r['batch'])*int(r['barriers'])) for r in rs),read_bytes_update=statistics.fmean(int(r['read_bytes'])/(int(r['batch'])*int(r['barriers'])) for r in rs),write_bytes_update=statistics.fmean(int(r['write_bytes'])/(int(r['batch'])*int(r['barriers'])) for r in rs));bb.append(s) + feature_summaries['barrier']=bb +(COORD/'feature-summaries.json').write_text(json.dumps(feature_summaries,indent=2)+'\n') +controls=[] +for folder in OUT.iterdir(): + p=folder/'results.csv' + if not folder.is_dir() or not p.exists():continue + rr=read(p) + if not rr or not {'version','phase','ops_s'}.issubset(rr[0]):continue + for phase in dict.fromkeys(r['phase'] for r in rr if r['phase']!='warm'): + rs=[r for r in rr if r['version']=='main' and r['phase']==phase] + if not rs:continue + first=float(rs[0]['ops_s']) + for r in rs:controls.append(dict(workload=folder.name,phase=phase,order=r['order'],ops_s=float(r['ops_s']),change_from_first_pct=100*(float(r['ops_s'])/first-1),cpu_us_op=1e6*float(r['cpu_s'])/int(r['ops']),read_bytes=int(r['rd_bytes']))) +if controls: + with (COORD/'control-drift.csv').open('w') as f: + w=csv.DictWriter(f,fieldnames=list(controls[0]),lineterminator='\n');w.writeheader();w.writerows(controls) +with (COORD/'report.md').open('a') as f: + f.write('\n## Main control drift\n\n') + f.write('All controls are retained. Large variation in these unchanged-main samples prevents attributing small feature differences to code. The measured cause of this variation is unknown. Nearby ABBA repeats are separate evidence; they do not replace these results.\n\n') + f.write('| Regime | Phase | First ops/s | Middle ops/s | Last ops/s | Middle vs first | Last vs first |\n| --- | --- | ---: | ---: | ---: | ---: | ---: |\n') + for mode in ['resident','disk']: + for phase in dict.fromkeys(r['phase'] for r in controls if r['workload']==mode): + rr=[r for r in controls if r['workload']==mode and r['phase']==phase] + if len(rr)!=3:continue + f.write(f"| {mode} | {phase} | {rr[0]['ops_s']:,.1f} | {rr[1]['ops_s']:,.1f} | {rr[2]['ops_s']:,.1f} | {rr[1]['change_from_first_pct']:+.2f}% | {rr[2]['change_from_first_pct']:+.2f}% |\n") + +with (COORD/'report.md').open('a') as f: + f.write('\n## Interpretation and limits\n\n') + f.write('These samples support feature tradeoffs, not an isolated estimate of merge overhead. There are two samples per candidate and two or three main controls; non-overlap is a descriptive range check, not a statistical confidence interval. Unchanged main controls drifted substantially, including in nearby repeats. The cause is unproven.\n\n') + f.write('The invalid-address checks reproduce a roughly 4.5% resident single-thread point-query loss: broad screening -4.60%, nearby warmed ABBA -4.54% with non-overlapping sample ranges. Its repeated eight-thread point query (-3.64%) overlaps main, and repeated one-result/eight-thread range query (+0.03%) does not reproduce the initial apparent loss. Atomic directory nearby repeats (-6.18% point and -3.89% short range) and async-barrier ordinary-query repeats (-0.52% point and -2.28% short range) all overlap main; their smaller effects remain inconclusive. The scalar/SIMD directory variants show larger resident feature costs in the broad screen. These are comparisons of the whole features to main, not newly introduced merge regressions.\n\n') + f.write('Prefix compression improves several common-prefix workloads, but the random-key resident long-range screen loses 10.88% with non-overlapping samples. The default 64-page automatic log-group durability threshold loses 42.7% insert throughput in the short WAL-enabled screen; this is the cost of added automatic durability versus main without automatic barriers. Disabling the threshold gives -1.3% with overlapping ranges. These tradeoffs require workload-specific judgment, not a blanket performance-neutral claim.\n\n') + f.write('All timed processes passed their declared verification. Query runs validate keys/values/ranges and close successfully. Insert-only runs validate API returns, positive insert counts and close, but do not perform readback. Barrier runs crash after the final completed barrier, reopen and verify every key/value. Hash-filter runs compare checksums and expected output counts. Every measured ordinary resident and focused-repeat query phase has zero physical read bytes. The retained pre-repeat warmup runs point and one-result range queries at eight threads for 10 seconds each; it is not pooled with comparison controls.\n\n') + f.write('Separate correctness testing reproduced an inherited small-cache async progress stall on main and invalid-addresses under restricted CPU scheduling. That limitation is documented in the branch correctness evidence; it was not repaired in these merges, and no benchmark run stalled.\n\n') + f.write('Barrier measurements are short and show substantial run-to-run I/O variation in some configurations. In particular, WAL-enabled/no-background-worker candidate sync spans about 60k to 194k updates/s, while main spans 63k to 65k; its +99% mean alone is not a stable speedup estimate. Async launch latency is distinct from durability completion. These runs use depth 1: completion is observed by waiting or polling before the next insert batch. Greater-depth harness modes may include intervening inserts in observed completion, but were not measured here.\n\n') + f.write('| Barrier config | Candidate mode | Main updates/s range | Candidate updates/s range | Mean change | Mean launch p99 us | Mean observed completion p99 us |\n| --- | --- | ---: | ---: | ---: | ---: | ---: |\n') + for r in feature_summaries.get('barrier',[]): + if r['version']=='main':continue + f.write(f"| {r['phase'].rsplit('-',1)[0]} | {r['phase'].rsplit('-',1)[1]} | {r['baseline_ops_s_min']:,.0f}–{r['baseline_ops_s_max']:,.0f} | {r['ops_s_min']:,.0f}–{r['ops_s_max']:,.0f} | {r['delta_pct']:+.2f}% | {r['launch_p99_us']:.2f} | {r['completion_p99_us']:.2f} |\n") + f.write('\nHash-filter API versus application filtering: mean latency changes for limits 1/2/10/100 are -5.70%/-16.71%/+0.56%/-9.59%; output checksums match. Full ranges, visited tuple counts, CPU/I/O and provenance are retained in the raw results and feature-summaries.json.\n') diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/00-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/00-main.log new file mode 100644 index 000000000..873e8f345 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/00-main.log @@ -0,0 +1,30 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=0 sabotage=0 +open took 0.006 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 7786147 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.05s ops/cpu-s=3668715 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=1 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=7570779 notfound=0 +== point:uniform@1 T=1 ops=7570779 secs= 6.00 ops/s= 1261757 lat us p50=0.74 p99=1.07 p99.9=5.18 max=62.6 mean=0.77 + cpu=6.00s ops/cpu-s=1261769 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 26246664 ops (+2624370 ops/s) + found=38675055 notfound=0 +== point:uniform@8 T=8 ops=38675055 secs= 6.00 ops/s= 6445634 lat us p50=1.17 p99=1.71 p99.9=2.21 max=95.5 mean=1.21 + cpu=47.99s ops/cpu-s=805965 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 799349 ops (+1844558182126266112 ops/s) +== range:uniform:1@1 T=1 ops=688170 secs= 6.00 ops/s= 114690 lat us p50=8.58 p99=13.18 p99.9=36.35 max=108.9 mean=8.69 + cpu=6.00s ops/cpu-s=114695 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.32 p99=12.93 mean=8.43 per-tuple=nan ns deinit mean=0.24 us tuples/s=114690 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) +== range:uniform:1@8 T=8 ops=3602377 secs= 6.00 ops/s= 600377 lat us p50=12.42 p99=17.66 p99.9=26.37 max=485.9 mean=13.28 + cpu=47.99s ops/cpu-s=75070 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=11.90 p99=17.15 mean=12.72 per-tuple=nan ns deinit mean=0.53 us tuples/s=600377 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 13084 ops (+1844550238704732416 ops/s) +close took 0.141 s +== range:uniform:10000@8 T=8 ops=25833 secs= 6.00 ops/s= 4305 lat us p50=1753.09 p99=2260.99 p99.9=2326.53 max=2498.1 mean=1857.98 + cpu=47.99s ops/cpu-s=538 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=16.26 p99=25.86 mean=16.46 per-tuple=188.8 ns deinit mean=0.80 us tuples/s=41992304 + insert/run=6.00s drain(close)=0.14s total=6.14s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/01-disk-format.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/01-disk-format.log new file mode 100644 index 000000000..d69d8f628 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/01-disk-format.log @@ -0,0 +1,30 @@ +kvbench version=disk-format splinter=splinterdb_build_version fb878f13-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=1 sabotage=0 +open took 0.006 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 6877194 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.07s ops/cpu-s=2894984 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=5 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=7211899 notfound=0 +== point:uniform@1 T=1 ops=7211899 secs= 6.00 ops/s= 1201927 lat us p50=0.78 p99=1.14 p99.9=4.42 max=128.4 mean=0.81 + cpu=6.00s ops/cpu-s=1201963 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 23771278 ops (+2376828 ops/s) + found=35384682 notfound=0 +== point:uniform@8 T=8 ops=35384682 secs= 6.00 ops/s= 5897246 lat us p50=1.30 p99=1.87 p99.9=2.66 max=452.4 mean=1.32 + cpu=47.99s ops/cpu-s=737367 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 780537 ops (+1844458626551731712 ops/s) +== range:uniform:1@1 T=1 ops=674821 secs= 6.00 ops/s= 112465 lat us p50=8.83 p99=10.37 p99.9=16.64 max=46.5 mean=8.87 + cpu=6.00s ops/cpu-s=112388 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.58 p99=10.11 mean=8.59 per-tuple=nan ns deinit mean=0.25 us tuples/s=112465 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) +== range:uniform:1@8 T=8 ops=3368698 secs= 6.00 ops/s= 561429 lat us p50=12.93 p99=18.69 p99.9=27.39 max=120.9 mean=14.21 + cpu=47.99s ops/cpu-s=70193 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.42 p99=18.18 mean=13.59 per-tuple=nan ns deinit mean=0.58 us tuples/s=561429 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 12416 ops (+1844547406777281536 ops/s) +close took 0.096 s +== range:uniform:10000@8 T=8 ops=24625 secs= 6.00 ops/s= 4104 lat us p50=1818.62 p99=2392.06 p99.9=2523.14 max=2704.5 mean=1949.13 + cpu=47.99s ops/cpu-s=513 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=16.64 p99=25.86 mean=16.92 per-tuple=198.0 ns deinit mean=0.81 us tuples/s=40033646 + insert/run=6.00s drain(close)=0.10s total=6.10s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/11-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/11-main.log new file mode 100644 index 000000000..6977aae92 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/11-main.log @@ -0,0 +1,30 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=11 sabotage=0 +open took 0.007 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 7472109 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3393972 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=6915404 notfound=0 +== point:uniform@1 T=1 ops=6915404 secs= 6.00 ops/s= 1152531 lat us p50=0.82 p99=1.14 p99.9=1.42 max=55.5 mean=0.84 + cpu=6.00s ops/cpu-s=1152568 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 21133749 ops (+2113088 ops/s) + found=31626695 notfound=0 +== point:uniform@8 T=8 ops=31626695 secs= 6.00 ops/s= 5270841 lat us p50=1.42 p99=2.08 p99.9=3.10 max=435.7 mean=1.47 + cpu=47.98s ops/cpu-s=659104 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 730121 ops (+1844496310552051968 ops/s) +== range:uniform:1@1 T=1 ops=632712 secs= 6.00 ops/s= 105447 lat us p50=9.34 p99=10.88 p99.9=15.49 max=695.7 mean=9.46 + cpu=6.00s ops/cpu-s=105461 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=9.09 p99=10.62 mean=9.16 per-tuple=nan ns deinit mean=0.27 us tuples/s=105447 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) +== range:uniform:1@8 T=8 ops=3195303 secs= 6.00 ops/s= 532531 lat us p50=13.95 p99=19.71 p99.9=29.95 max=100.9 mean=14.98 + cpu=47.99s ops/cpu-s=66577 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=13.44 p99=19.20 mean=14.33 per-tuple=nan ns deinit mean=0.60 us tuples/s=532531 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11306 ops (+1844551191351300864 ops/s) +close took 0.115 s +== range:uniform:10000@8 T=8 ops=22716 secs= 6.00 ops/s= 3786 lat us p50=1982.46 p99=2523.14 p99.9=2654.21 max=2800.7 mean=2112.86 + cpu=47.99s ops/cpu-s=473 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=18.18 p99=28.42 mean=18.41 per-tuple=214.6 ns deinit mean=0.91 us tuples/s=36934012 + insert/run=6.00s drain(close)=0.11s total=6.12s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/21-disk-format.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/21-disk-format.log new file mode 100644 index 000000000..487b3e2f2 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/21-disk-format.log @@ -0,0 +1,30 @@ +kvbench version=disk-format splinter=splinterdb_build_version fb878f13-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=21 sabotage=0 +open took 0.007 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 7942406 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3469331 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=6866473 notfound=0 +== point:uniform@1 T=1 ops=6866473 secs= 6.00 ops/s= 1144383 lat us p50=0.84 p99=1.20 p99.9=1.68 max=34.2 mean=0.85 + cpu=6.00s ops/cpu-s=1144371 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 21657061 ops (+2165456 ops/s) + found=32023217 notfound=0 +== point:uniform@8 T=8 ops=32023217 secs= 6.00 ops/s= 5337004 lat us p50=1.42 p99=2.03 p99.9=4.42 max=146.5 mean=1.46 + cpu=47.99s ops/cpu-s=667243 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 746428 ops (+1844469563345863168 ops/s) +== range:uniform:1@1 T=1 ops=646324 secs= 6.00 ops/s= 107715 lat us p50=9.09 p99=11.14 p99.9=16.00 max=269.2 mean=9.26 + cpu=6.00s ops/cpu-s=107744 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=8.83 p99=10.62 mean=8.97 per-tuple=nan ns deinit mean=0.26 us tuples/s=107715 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) +== range:uniform:1@8 T=8 ops=3087323 secs= 6.00 ops/s= 514530 lat us p50=14.46 p99=20.74 p99.9=32.00 max=149.6 mean=15.50 + cpu=47.99s ops/cpu-s=64336 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=13.70 p99=19.71 mean=14.83 per-tuple=nan ns deinit mean=0.63 us tuples/s=514530 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11109 ops (+1844546814534079744 ops/s) +close took 0.113 s +== range:uniform:10000@8 T=8 ops=22300 secs= 6.00 ops/s= 3717 lat us p50=1982.46 p99=2588.67 p99.9=2654.21 max=4356.7 mean=2152.26 + cpu=47.99s ops/cpu-s=465 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=18.18 p99=24.83 mean=18.48 per-tuple=218.6 ns deinit mean=0.90 us tuples/s=36253487 + insert/run=6.00s drain(close)=0.11s total=6.11s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/22-main.log b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/22-main.log new file mode 100644 index 000000000..d59419ef3 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/22-main.log @@ -0,0 +1,30 @@ +kvbench version=main splinter=splinterdb_build_version 7bb5fab3 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=22 sabotage=0 +open took 0.006 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.02 ops/s= 8000871 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.06s ops/cpu-s=3196011 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=6889105 notfound=0 +== point:uniform@1 T=1 ops=6889105 secs= 6.00 ops/s= 1148133 lat us p50=0.82 p99=1.17 p99.9=1.46 max=36.6 mean=0.85 + cpu=6.00s ops/cpu-s=1148182 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 21983968 ops (+2198121 ops/s) + found=32785181 notfound=0 +== point:uniform@8 T=8 ops=32785181 secs= 6.00 ops/s= 5463996 lat us p50=1.39 p99=1.94 p99.9=2.53 max=139.6 mean=1.42 + cpu=47.99s ops/cpu-s=683146 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@1: T=1 dist=uniform (fixed duration) + [range:uniform:1@1] progress 730182 ops (+1844484954785668608 ops/s) +== range:uniform:1@1 T=1 ops=630703 secs= 6.00 ops/s= 105113 lat us p50=9.34 p99=10.88 p99.9=15.49 max=224.8 mean=9.49 + cpu=6.00s ops/cpu-s=105052 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=9.09 p99=10.62 mean=9.19 per-tuple=nan ns deinit mean=0.27 us tuples/s=105113 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) +== range:uniform:1@8 T=8 ops=3297430 secs= 6.00 ops/s= 549552 lat us p50=13.44 p99=19.20 p99.9=27.39 max=99.6 mean=14.51 + cpu=47.99s ops/cpu-s=68709 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=12.93 p99=18.18 mean=13.89 per-tuple=nan ns deinit mean=0.58 us tuples/s=549552 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11763 ops (+1844547284492220928 ops/s) +close took 0.076 s +== range:uniform:10000@8 T=8 ops=23579 secs= 6.00 ops/s= 3930 lat us p50=1916.93 p99=2392.06 p99.9=2457.60 max=2477.0 mean=2035.76 + cpu=47.99s ops/cpu-s=491 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=17.15 p99=23.81 mean=17.58 per-tuple=206.8 ns deinit mean=0.84 us tuples/s=38341201 + insert/run=6.00s drain(close)=0.08s total=6.08s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/manifest.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/manifest.json new file mode 100644 index 000000000..413dfcbc8 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/manifest.json @@ -0,0 +1,429 @@ +{ + "mode": "resident", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "scope": "Updated feature branches versus fresh main; not an isolation of merge cost", + "runner_sha256": "536f10a1a7fd4b1e8e0250ea01cfe36793c7d6e2345f141092616ec06f64ee50", + "runs": [ + { + "label": "00-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/results.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791906088.0460572, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791906123.264936, + "log_sha256": "0861c0ade82e24816323caa4da330c4cac0c53104f1160285c1cf8a8df22251a", + "process_cpu_s": 182.022043, + "process_read_bytes": 28295168, + "process_write_bytes": 344064, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-main-ordinary.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 28655616 + }, + { + "label": "01-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/results.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "1", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791906123.396471, + "binary_sha256": "733c71174e9356864c326570c69279aacafc92f2dd7286813f9a21cdf9335ddc", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791906158.5575716, + "log_sha256": "e94a6af4a231157eeac29ed3e093bf33df2acfa168489bc4518d37315075178e", + "process_cpu_s": 182.067102, + "process_read_bytes": 28295168, + "process_write_bytes": 344064, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-disk-format-ordinary.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 28655616 + }, + { + "label": "11-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/results.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "11", + "--rep", + "11", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791906475.645752, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791906510.8186312, + "log_sha256": "034a238757bbc021b03c57327181adea2db55fca23bb013540bf3f4ab4091d19", + "process_cpu_s": 182.05580900000004, + "process_read_bytes": 28295168, + "process_write_bytes": 335872, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-main-ordinary.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 28655616 + }, + { + "label": "21-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/results.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "21", + "--rep", + "21", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791906827.7381353, + "binary_sha256": "733c71174e9356864c326570c69279aacafc92f2dd7286813f9a21cdf9335ddc", + "library_sha256": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "source_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "verified": true, + "returncode": 0, + "finished": 1791906862.8999107, + "log_sha256": "b4bff98b28bcae54d590f70bc4098eb4e952d4fc69aa0a383c0da4f9a8c2154a", + "process_cpu_s": 182.0508850000001, + "process_read_bytes": 28295168, + "process_write_bytes": 339968, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-disk-format-ordinary.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 28655616 + }, + { + "label": "22-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident/results.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "22", + "--rep", + "22", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "started": 1791906862.9535198, + "binary_sha256": "f9a3dbd472dd5a6773abbd986dee047aec294108779eb731f678943aaeb89c5e", + "library_sha256": "20944ac1b2f8420bd8f868d20894e462f9c3c240e4380e9c1242287cbb394f9a", + "source_head": "7bb5fab305b9c6179e31f75998dabea833537a08", + "verified": true, + "returncode": 0, + "finished": 1791906898.0791984, + "log_sha256": "12947e19e2731c0041d5caf9492a0f102ac3edab312e9eb80d03bcbe7275fea0", + "process_cpu_s": 182.05485299999998, + "process_read_bytes": 28295168, + "process_write_bytes": 335872, + "verification_kind": "query key/value/range validation and successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/main-7bb5fab/fixtures/resident-main-ordinary.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "db_allocated_bytes": 28655616 + } + ], + "order": [ + "main", + "disk-format", + "filtered-ranges", + "log-group-limit", + "lazy-merge", + "async-durable-barrier", + "cache-invalid-addresses", + "directory-direct", + "directory-bucket", + "directory-bucket-simd", + "directory-bucket-atomic", + "main", + "directory-bucket-atomic", + "directory-bucket-simd", + "directory-bucket", + "directory-direct", + "cache-invalid-addresses", + "async-durable-barrier", + "lazy-merge", + "log-group-limit", + "filtered-ranges", + "disk-format", + "main" + ], + "phases": [ + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@1", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "complete": true +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/results.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/results.csv new file mode 100644 index 000000000..bd438e693 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/resident/results.csv @@ -0,0 +1,31 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,resident,0,0,warm,8,0,200000,200000,0.026,7786146.6,nan,nan,nan,0.000,nan,0.055,3668715.0,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,7786146.6,0.026,0.000,0.026,0.000,0,0,nan,0,0,0.006,0,0,0,0,1 +main,resident,0,0,point:uniform@1,1,0,200000,7570779,6.000,1261757.3,0.744,1.072,5.184,62.610,0.771,6.000,1261769.0,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,0,0,point:uniform@8,8,0,200000,38675055,6.000,6445634.1,1.168,1.712,2.208,95.451,1.206,47.986,805964.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,0,0,range:uniform:1@1,1,0,200000,688170,6.000,114690.1,8.576,13.184,36.352,108.924,8.694,6.000,114695.4,0,4096,0.00,0.01,8.320,12.928,8.430,nan,0.239,688170,114690.1,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,0,0,range:uniform:1@8,8,0,200000,3602377,6.000,600376.6,12.416,17.664,26.368,485.919,13.285,47.987,75070.0,0,0,0.00,0.00,11.904,17.152,12.722,nan,0.526,3602377,600376.6,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,0,0,range:uniform:10000@8,8,0,200000,25833,6.000,4305.3,1753.088,2260.992,2326.528,2498.138,1857.975,47.994,538.3,0,4096,0.00,0.16,16.256,25.856,16.463,188.75,0.797,251963190,41992304.1,6.000,0.141,6.141,0.013,0,307200,nan,162664448,28655616,0.006,0,0,0,0,0 +disk-format,resident,1,0,warm,8,1,200000,200000,0.029,6877193.8,nan,nan,nan,0.000,nan,0.069,2894984.4,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,6877193.8,0.029,0.000,0.029,0.000,0,0,nan,0,0,0.006,0,0,0,0,5 +disk-format,resident,1,0,point:uniform@1,1,1,200000,7211899,6.000,1201926.5,0.776,1.136,4.416,128.428,0.809,6.000,1201962.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +disk-format,resident,1,0,point:uniform@8,8,1,200000,35384682,6.000,5897246.4,1.296,1.872,2.656,452.389,1.320,47.988,737366.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +disk-format,resident,1,0,range:uniform:1@1,1,1,200000,674821,6.000,112465.1,8.832,10.368,16.640,46.538,8.865,6.004,112387.8,0,4096,0.00,0.01,8.576,10.112,8.586,nan,0.252,674821,112465.1,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +disk-format,resident,1,0,range:uniform:1@8,8,1,200000,3368698,6.000,561429.3,12.928,18.688,27.392,120.886,14.206,47.992,70193.0,0,0,0.00,0.00,12.416,18.176,13.588,nan,0.578,3368698,561429.3,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +disk-format,resident,1,0,range:uniform:10000@8,8,1,200000,24625,6.000,4104.0,1818.624,2392.064,2523.136,2704.509,1949.135,47.992,513.1,0,4096,0.00,0.17,16.640,25.856,16.921,198.01,0.808,240211349,40033646.2,6.000,0.096,6.097,0.011,0,311296,nan,162664448,28655616,0.006,0,0,0,0,0 +main,resident,11,0,warm,8,11,200000,200000,0.027,7472108.7,nan,nan,nan,0.000,nan,0.059,3393972.3,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,7472108.7,0.027,0.000,0.027,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,resident,11,0,point:uniform@1,1,11,200000,6915404,6.000,1152531.0,0.824,1.136,1.424,55.546,0.843,6.000,1152567.5,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,resident,11,0,point:uniform@8,8,11,200000,31626695,6.000,5270841.2,1.424,2.080,3.104,435.656,1.474,47.984,659103.9,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,resident,11,0,range:uniform:1@1,1,11,200000,632712,6.000,105447.5,9.344,10.880,15.488,695.659,9.455,5.999,105461.4,0,0,0.00,0.00,9.088,10.624,9.160,nan,0.267,632712,105447.5,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,resident,11,0,range:uniform:1@8,8,11,200000,3195303,6.000,532530.7,13.952,19.712,29.952,100.913,14.977,47.994,66577.2,0,0,0.00,0.00,13.440,19.200,14.334,nan,0.601,3195303,532530.7,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +main,resident,11,0,range:uniform:10000@8,8,11,200000,22716,6.000,3785.8,1982.464,2523.136,2654.208,2800.712,2112.859,47.991,473.3,0,0,0.00,0.00,18.176,28.416,18.413,214.61,0.909,221613100,36934011.9,6.000,0.115,6.115,0.012,0,311296,nan,162664448,28655616,0.007,0,0,0,0,0 +disk-format,resident,21,0,warm,8,21,200000,200000,0.025,7942405.8,nan,nan,nan,0.000,nan,0.058,3469331.1,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,7942405.8,0.025,0.000,0.025,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +disk-format,resident,21,0,point:uniform@1,1,21,200000,6866473,6.000,1144382.6,0.840,1.200,1.680,34.233,0.848,6.000,1144370.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +disk-format,resident,21,0,point:uniform@8,8,21,200000,32023217,6.000,5337003.8,1.424,2.032,4.416,146.466,1.458,47.993,667243.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +disk-format,resident,21,0,range:uniform:1@1,1,21,200000,646324,6.000,107715.5,9.088,11.136,16.000,269.160,9.257,5.999,107743.9,0,4096,0.00,0.01,8.832,10.624,8.972,nan,0.257,646324,107715.5,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +disk-format,resident,21,0,range:uniform:1@8,8,21,200000,3087323,6.000,514530.3,14.464,20.736,32.000,149.561,15.500,47.987,64336.2,0,0,0.00,0.00,13.696,19.712,14.830,nan,0.627,3087323,514530.3,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.007,0,0,0,0,0 +disk-format,resident,21,0,range:uniform:10000@8,8,21,200000,22300,6.000,3716.5,1982.464,2588.672,2654.208,4356.730,2152.255,47.991,464.7,0,4096,0.00,0.18,18.176,24.832,18.482,218.63,0.900,217529609,36253486.7,6.000,0.113,6.113,0.012,0,307200,nan,162664448,28655616,0.007,0,0,0,0,0 +main,resident,22,0,warm,8,22,200000,200000,0.025,8000871.1,nan,nan,nan,0.000,nan,0.063,3196011.4,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,8000871.1,0.025,0.000,0.025,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,22,0,point:uniform@1,1,22,200000,6889105,6.000,1148133.5,0.824,1.168,1.456,36.575,0.846,6.000,1148181.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,22,0,point:uniform@8,8,22,200000,32785181,6.000,5463995.8,1.392,1.936,2.528,139.647,1.424,47.992,683145.5,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,22,0,range:uniform:1@1,1,22,200000,630703,6.000,105112.6,9.344,10.880,15.488,224.800,9.486,6.004,105051.9,0,0,0.00,0.00,9.088,10.624,9.189,nan,0.268,630703,105112.6,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,22,0,range:uniform:1@8,8,22,200000,3297430,6.000,549552.3,13.440,19.200,27.392,99.564,14.513,47.991,68709.3,0,0,0.00,0.00,12.928,18.176,13.891,nan,0.582,3297430,549552.3,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.006,0,0,0,0,0 +main,resident,22,0,range:uniform:10000@8,8,22,200000,23579,6.000,3929.7,1916.928,2392.064,2457.600,2476.953,2035.763,47.994,491.3,0,0,0.00,0.00,17.152,23.808,17.581,206.78,0.845,230056482,38341201.1,6.000,0.076,6.077,0.012,0,311296,nan,162664448,28655616,0.006,0,0,0,0,0 diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_job.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_job.py new file mode 100644 index 000000000..299c91084 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_job.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Coordinate this campaign's builds/tests and disk-backed benchmarks.""" +import argparse +import fcntl +import json +import os +from pathlib import Path +import subprocess +import tempfile +import time + +ROOT = Path(__file__).resolve().parent.parent +p = argparse.ArgumentParser() +p.add_argument('mode', choices=['build', 'test', 'benchmark']) +p.add_argument('--name', required=True) +p.add_argument('--db-dir', type=Path) +p.add_argument('command', nargs=argparse.REMAINDER) +a = p.parse_args() +command = a.command[1:] if a.command[:1] == ['--'] else a.command +if not command: + p.error('command required after --') +if a.mode == 'benchmark': + if a.db_dir is None: + p.error('benchmark requires --db-dir') + a.db_dir.mkdir(parents=True, exist_ok=True) + db_dir = a.db_dir.resolve() + if not db_dir.is_relative_to(ROOT / 'benchmarks'): + p.error('benchmark database directory must be under the campaign benchmarks directory') + fs = subprocess.check_output(['findmnt', '-n', '-o', 'FSTYPE,SOURCE', '-T', str(db_dir)], text=True).strip() + if fs.split()[0] in {'tmpfs', 'ramfs'} or not fs.split()[1].startswith('/dev/'): + p.error('benchmark storage must be a verified disk filesystem: ' + fs) + bench_lock = open(ROOT / 'coordination' / 'benchmark.lock', 'a+') + fcntl.flock(bench_lock, fcntl.LOCK_EX) + print('Benchmark database storage:', fs, str(db_dir), flush=True) +machine_lock = open(ROOT / 'coordination' / 'machine.lock', 'a+') +fcntl.flock(machine_lock, fcntl.LOCK_EX if a.mode == 'benchmark' else fcntl.LOCK_SH) +record = dict(name=a.name, mode=a.mode, cwd=os.getcwd(), command=command, + started=time.time(), pid=os.getpid()) +if a.db_dir is not None: + record['db_dir'] = str(a.db_dir.resolve()) +print('Starting', a.name, a.mode, flush=True) +try: + # Tool commands run in distinct PID namespaces; GNU make's FIFO names + # otherwise collide when several builds choose /tmp/GMfifo8. + with tempfile.TemporaryDirectory(prefix='splinter-' + a.name + '-', dir='/tmp') as temporary: + environment = dict(os.environ, TMPDIR=temporary) + result = subprocess.run(command, env=environment) + record['returncode'] = result.returncode +finally: + record['finished'] = time.time() + with open(ROOT / 'coordination' / 'jobs.jsonl', 'a') as journal: + fcntl.flock(journal, fcntl.LOCK_EX) + journal.write(json.dumps(record) + '\n') +raise SystemExit(result.returncode) diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix-fixtures.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix-fixtures.py new file mode 100644 index 000000000..e6ff54325 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix-fixtures.py @@ -0,0 +1,152 @@ +"""Fresh main integration screening. Invoke every mode under run_job.py benchmark.""" +import argparse,csv,hashlib,json,os,re,resource,signal,statistics,subprocess,time +from pathlib import Path +ROOT=Path('/home/rob/projects/splinterdb'); CAMP=ROOT/'.cache/projects-20261011' +COORD=Path(__file__).resolve().parent; OUT=CAMP/'benchmarks/main-7bb5fab' +VARIANTS=['disk-format','filtered-ranges','log-group-limit','lazy-merge','async-durable-barrier','cache-invalid-addresses','directory-direct','directory-bucket','directory-bucket-simd','directory-bucket-atomic'] +ORDER=['main',*VARIANTS,'main',*reversed(VARIANTS),'main'] +def sha(path): + h=hashlib.sha256() + with Path(path).open('rb') as f: + for b in iter(lambda:f.read(1024*1024),b''):h.update(b) + return h.hexdigest() +def save(path,value):path.write_text(json.dumps(value,indent=2)+'\n') +def rows(path): + with path.open() as f:return list(csv.DictReader(f)) +def csvwrite(path,data): + if not data:return + with path.open('w') as f: + w=csv.DictWriter(f,fieldnames=list(data[0]),lineterminator='\n');w.writeheader();w.writerows(data) +def build(name): + r=json.loads((COORD/(name+'-build.json')).read_text()) + assert sha(r['library'])==r['library_sha256'],name + for filename,expected in r['source_files_sha256'].items():assert sha(Path(r['tree'])/filename)==expected,(name,filename) + for b in r['binaries'].values():assert sha(b['binary'])==b['binary_sha256'],name + return r +def cmdrun(folder,manifest,label,profile,binary,args,env=None,timeout=600): + b=build(profile);exe=b['binaries'][binary] + cmd=['taskset','-c','0-11',exe['binary'],*map(str,args)] + rec=dict(label=label,profile=profile,command=cmd,started=time.time(),binary_sha256=exe['binary_sha256'],library_sha256=b['library_sha256'],source_head=b['head'],verified=False) + print(label,flush=True) + log=folder/(label+'.log') + try: + before=resource.getrusage(resource.RUSAGE_CHILDREN) + with log.open('w') as f: + p=subprocess.Popen(cmd,env=dict(os.environ,**(env or {})),stdout=f,stderr=subprocess.STDOUT,start_new_session=True) + try:p.wait(timeout=timeout) + except BaseException: + try:os.killpg(p.pid,signal.SIGKILL) + except ProcessLookupError:pass + p.wait() + raise + after=resource.getrusage(resource.RUSAGE_CHILDREN) + rec.update(returncode=p.returncode,finished=time.time(),log_sha256=sha(log),process_cpu_s=(after.ru_utime+after.ru_stime-before.ru_utime-before.ru_stime),process_read_bytes=512*(after.ru_inblock-before.ru_inblock),process_write_bytes=512*(after.ru_oublock-before.ru_oublock)) + except BaseException as e: + rec.update(exception=str(e),finished=time.time());manifest['runs'].append(rec);save(folder/'manifest.json',manifest);raise + manifest['runs'].append(rec);save(folder/'manifest.json',manifest) + assert p.returncode==0,rec + assert 'FINDING: splinterdb_close' not in log.read_text(),rec + return rec + +def newmode(mode): + folder=OUT/mode;folder.mkdir(parents=True,exist_ok=False) + fs=subprocess.check_output(['findmnt','-n','-o','FSTYPE,SOURCE','-T',str(folder)],text=True).strip();assert fs.split()[0] not in ['tmpfs','ramfs'] and fs.split()[1].startswith('/dev/') + manifest=dict(mode=mode,storage=fs,io='O_DIRECT',cpu_affinity='0-11',scope='Updated feature branches versus fresh main; not an isolation of merge cost',runner_sha256=sha(__file__),runs=[]) + save(folder/'manifest.json',manifest);return folder,manifest + +def kvargs(db,csvpath,profile,regime,index,secs=6,prefix=False,create=False): + return ['--db',db,'--create' if create else '--open','--n',200000 if regime=='resident' else 2000000,'--cache-mib',128 if regime=='resident' else 64,'--disk-gib',8,'--log',0,'--threads',1 if create else 8,'--mt-bg',0,'--normal-bg',0,'--memtable-mib',4,'--secs',secs,'--warmup',1,'--csv',csvpath,'--vtag',profile,'--stag',regime,'--otag',index,'--rep',index,'--seed',42,'--hang-secs',60,'--hang-dir',str(csvpath.parent)] +def verifykv(csvpath,index,count): + rr=[r for r in rows(csvpath) if r['order']==str(index)] + assert len(rr)==count,(index,len(rr),count) + assert all(int(r['ops'])>0 and int(r['errors'])==0 and int(r['violations'])==0 for r in rr),rr + return rr + +def fixtures(): + folder,m=newmode('fixtures');ff={} + for regime in ['resident','disk']: + for profile,pattern in [('main','ordinary'),('disk-format','ordinary'),('main','common'),('branch-prefix','common'),('main','random'),('branch-prefix','random')]: + label=f'{regime}-{profile}-{pattern}';db=folder/(label+'.db');binary='kvbench' if pattern=='ordinary' else 'prefix_kvbench' + rec=cmdrun(folder,m,label,profile,binary,kvargs(db,folder/(label+'.csv'),profile,regime,0,secs=1,create=True)+['load:random'],dict(PREFIX_PATTERN=pattern) if pattern!='ordinary' else None) + verifykv(folder/(label+'.csv'),0,1) + with db.open('rb') as f:os.fsync(f.fileno()) + rec.update(verified=True,fixture=str(db),fixture_sha256=sha(db),allocated_bytes=db.stat().st_blocks*512,apparent_bytes=db.stat().st_size) + ff[label]={k:rec[k] for k in ['fixture','fixture_sha256','allocated_bytes','apparent_bytes']};save(folder/'manifest.json',m);save(folder/'fixtures.json',ff) + m['complete']=True;save(folder/'manifest.json',m) + +def query_one(folder,m,index,profile,regime,phases,pattern='ordinary'): + f=json.loads((OUT/'fixtures/fixtures.json').read_text());owner=profile if profile in ['disk-format','branch-prefix'] else 'main' + fixture=f[f'{regime}-{owner}-{pattern}'];src=Path(fixture['fixture']);before=src.stat() + db=folder/'scratch.db';assert not db.exists() + subprocess.run(['cp','--reflink=auto','--sparse=always',str(src),str(db)],check=True) + with db.open('rb') as image:os.fsync(image.fileno()) + label=f'{index:02d}-{profile}';csvpath=folder/'results.csv';binary='kvbench' if pattern=='ordinary' else 'prefix_kvbench' + rec=cmdrun(folder,m,label,profile,binary,kvargs(db,csvpath,profile,regime,index,secs=6 if regime=='resident' else 5)+phases,dict(PREFIX_PATTERN=pattern) if pattern!='ordinary' else None) + rr=verifykv(csvpath,index,len(phases)) + rec.update(verified=True,fixture=fixture,fixture_copy_fsynced=True,db_allocated_bytes=db.stat().st_blocks*512) + assert src.stat().st_mtime_ns==before.st_mtime_ns + save(folder/'manifest.json',m);db.unlink() + +def queries(regime): + folder,m=newmode(regime) + phases=['warm','point:uniform@1','point:uniform@8','range:uniform:1@1','range:uniform:1@8','range:uniform:10000@8'] if regime=='resident' else ['warm','point:uniform@8','range:uniform:1@8','range:uniform:10000@8'] + m.update(order=ORDER,phases=phases) + for i,profile in enumerate(ORDER):query_one(folder,m,i,profile,regime,phases) + m['complete']=True;save(folder/'manifest.json',m) + +def prefix(): + for regime in ['resident','disk']: + for pattern in ['common','random']: + folder,m=newmode(f'prefix-{regime}-{pattern}') + phases=['warm','point:uniform@8','range:uniform:1@8','range:uniform:10000@8'] + if regime=='resident':phases+=['ow:uniform@8'] + for i,profile in enumerate(['main','branch-prefix','branch-prefix','main']):query_one(folder,m,i,profile,regime,phases,pattern) + m['complete']=True;save(folder/'manifest.json',m) + +def insert(): + folder,m=newmode('insert');data=[] + profiles=['disk-format','filtered-ranges','lazy-merge','async-durable-barrier','cache-invalid-addresses','directory-direct','directory-bucket','directory-bucket-simd','directory-bucket-atomic','log-disabled','log-default'] + for wal,versions in [(1,['main',*profiles,'main',*reversed(profiles),'main']),(0,['main','cache-invalid-addresses','directory-bucket-atomic','directory-bucket-atomic','cache-invalid-addresses','main'])]: + for i,version in enumerate(versions): + profile='log-group-limit' if version.startswith('log-') else version + label=f'wal{wal}-{i:02d}-{version}';stem=folder/label;db=folder/'scratch.db';assert not db.exists() + args=['--db',db,'--csv',str(stem)+'.csv','--summary',str(stem)+'.summary.csv','--records',100000000,'--seconds',5,'--report-ms',100,'--cache-mib',256,'--disk-gib',16,'--direct','--inserters',8,'--readers',0,'--memtable-bg',1,'--normal-bg',3,'--memtable-mib',8,'--key-size',16,'--value-size',16,'--min-free-gib',12,'--seed',42] + if not wal:args+=['--no-log'] + if version.startswith('log-'):args+=['--log-group-size-pages',2**64-1 if version=='log-disabled' else 0] + rec=cmdrun(folder,m,label,profile,'insert_benchmark',args,timeout=240) + summary=rows(Path(str(stem)+'.summary.csv'));assert len(summary)==1;summary=summary[0];inserted=int(summary['inserted']);assert inserted>0 + log=(folder/(label+'.log')).read_text();assert re.search(r'load_end .* error=0(?: |\n)',log) + close=re.search(r'close_s=([\d.]+) close_rc=0',log);assert close + interval=rows(Path(str(stem)+'.csv'));cpu=sum(float(r['cpu_cores'])*float(r['interval_s']) for r in interval) + data.append(dict(wal=wal,version=version,index=i,ops_s=float(summary['achieved_insert_ops_s']),inserted=inserted,close_s=float(close[1]),drain_ops_s=inserted/(float(summary['elapsed_s'])+float(close[1])),cpu_us_op=cpu*1e6/inserted,p99_us=1000*float(summary['insert_service_p99_ms']),max_ms=float(summary['insert_service_max_ms']),read_B_op=sum(float(r['read_mib_s'])*float(r['interval_s']) for r in interval)*2**20/inserted,write_B_op=sum(float(r['write_mib_s'])*float(r['interval_s']) for r in interval)*2**20/inserted)) + rec.update(verified=True,allocated_bytes=db.stat().st_blocks*512);save(folder/'manifest.json',m);csvwrite(folder/'results.csv',data);db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +def hash_range(): + folder,m=newmode('hash');reference=None + for i,profile in enumerate(['main','filtered-ranges','filtered-ranges','main']): + label=f'{i}-{profile}';db=folder/'scratch.db';rec=cmdrun(folder,m,label,profile,'hash_range_benchmark',[db,16384,1000],timeout=900) + lines=(folder/(label+'.log')).read_text().splitlines();start=next(j for j,x in enumerate(lines) if x.startswith('mode,keys_per_tenant,'));rr=list(csv.DictReader(lines[start:]));assert len(rr)==4 + checks=[x['checksum'] for x in rr] + if reference is None:reference=checks + assert checks==reference + rec.update(verified=True,rows=rr,allocated_bytes=db.stat().st_blocks*512);save(folder/'manifest.json',m);db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +def barrier(): + folder,m=newmode('barrier') + for wal in [0,1]: + for workers in [0,1]: + order=[('main','sync'),('async-durable-barrier','sync'),('async-durable-barrier','async'),('async-durable-barrier','async'),('async-durable-barrier','sync'),('main','sync')] + for i,(profile,mode) in enumerate(order): + label=f'wal{wal}-workers{workers}-{i}-{profile}-{mode}';db=folder/'scratch.db';csvpath=folder/'results.csv' + rec=cmdrun(folder,m,label,profile,'barrier_bench',[db,csvpath,label,mode,wal,workers,256,2048 if wal else 512],dict(BARRIER_DEPTH='1')) + rr=rows(csvpath);assert rr[-1]['label']==label and int(rr[-1]['verified'])==1 + rec.update(verified=True,allocated_bytes=db.stat().st_blocks*512);save(folder/'manifest.json',m);db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +if __name__=='__main__': + def interrupted(signum,frame):raise InterruptedError(f'benchmark interrupted by signal {signum}') + for signum in [signal.SIGTERM,signal.SIGHUP]:signal.signal(signum,interrupted) + p=argparse.ArgumentParser();p.add_argument('mode',choices=['fixtures','resident','disk','prefix','insert','hash','barrier']);a=p.parse_args() + {'fixtures':fixtures,'resident':lambda:queries('resident'),'disk':lambda:queries('disk'),'prefix':prefix,'insert':insert,'hash':hash_range,'barrier':barrier}[a.mode]() diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix.py b/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix.py new file mode 100644 index 000000000..532cb8667 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/run_matrix.py @@ -0,0 +1,156 @@ +"""Fresh main integration screening. Invoke every mode under run_job.py benchmark.""" +import argparse,csv,hashlib,json,os,re,resource,signal,statistics,subprocess,time +from pathlib import Path +ROOT=Path('/home/rob/projects/splinterdb'); CAMP=ROOT/'.cache/projects-20261011' +COORD=Path(__file__).resolve().parent; OUT=CAMP/'benchmarks/main-7bb5fab' +VARIANTS=['disk-format','filtered-ranges','log-group-limit','lazy-merge','async-durable-barrier','cache-invalid-addresses','directory-direct','directory-bucket','directory-bucket-simd','directory-bucket-atomic'] +ORDER=['main',*VARIANTS,'main',*reversed(VARIANTS),'main'] +def sha(path): + h=hashlib.sha256() + with Path(path).open('rb') as f: + for b in iter(lambda:f.read(1024*1024),b''):h.update(b) + return h.hexdigest() +def save(path,value):path.write_text(json.dumps(value,indent=2)+'\n') +def rows(path): + with path.open() as f:return list(csv.DictReader(f)) +def csvwrite(path,data): + if not data:return + with path.open('w') as f: + w=csv.DictWriter(f,fieldnames=list(data[0]),lineterminator='\n');w.writeheader();w.writerows(data) +def build(name): + r=json.loads((COORD/(name+'-build.json')).read_text()) + assert sha(r['library'])==r['library_sha256'],name + for filename,expected in r['source_files_sha256'].items():assert sha(Path(r['tree'])/filename)==expected,(name,filename) + for b in r['binaries'].values(): + assert sha(b['binary'])==b['binary_sha256'],name + assert sha(b['source'])==b['source_sha256'],(name,b['source']) + return r +def cmdrun(folder,manifest,label,profile,binary,args,env=None,timeout=600): + b=build(profile);exe=b['binaries'][binary] + cmd=['taskset','-c','0-11',exe['binary'],*map(str,args)] + rec=dict(label=label,profile=profile,command=cmd,started=time.time(),binary_sha256=exe['binary_sha256'],library_sha256=b['library_sha256'],source_head=b['head'],verified=False) + print(label,flush=True) + log=folder/(label+'.log') + try: + before=resource.getrusage(resource.RUSAGE_CHILDREN) + with log.open('w') as f: + p=subprocess.Popen(cmd,env=dict(os.environ,**(env or {})),stdout=f,stderr=subprocess.STDOUT,start_new_session=True) + try:p.wait(timeout=timeout) + except BaseException: + try:os.killpg(p.pid,signal.SIGKILL) + except ProcessLookupError:pass + p.wait() + raise + after=resource.getrusage(resource.RUSAGE_CHILDREN) + rec.update(returncode=p.returncode,finished=time.time(),log_sha256=sha(log),process_cpu_s=(after.ru_utime+after.ru_stime-before.ru_utime-before.ru_stime),process_read_bytes=512*(after.ru_inblock-before.ru_inblock),process_write_bytes=512*(after.ru_oublock-before.ru_oublock)) + except BaseException as e: + rec.update(exception=str(e),finished=time.time());manifest['runs'].append(rec);save(folder/'manifest.json',manifest);raise + manifest['runs'].append(rec);save(folder/'manifest.json',manifest) + assert p.returncode==0,rec + assert 'FINDING: splinterdb_close' not in log.read_text(),rec + return rec + +def newmode(mode): + folder=OUT/mode;folder.mkdir(parents=True,exist_ok=False) + fs=subprocess.check_output(['findmnt','-n','-o','FSTYPE,SOURCE','-T',str(folder)],text=True).strip();assert fs.split()[0] not in ['tmpfs','ramfs'] and fs.split()[1].startswith('/dev/') + manifest=dict(mode=mode,storage=fs,io='O_DIRECT',cpu_affinity='0-11',scope='Updated feature branches versus fresh main; not an isolation of merge cost',runner_sha256=sha(__file__),runs=[]) + save(folder/'manifest.json',manifest);return folder,manifest + +def kvargs(db,csvpath,profile,regime,index,secs=6,prefix=False,create=False): + return ['--db',db,'--create' if create else '--open','--n',200000 if regime=='resident' else 2000000,'--cache-mib',128 if regime=='resident' else 64,'--disk-gib',8,'--log',0,'--threads',1 if create else 8,'--mt-bg',0,'--normal-bg',0,'--memtable-mib',4,'--secs',secs,'--warmup',1,'--csv',csvpath,'--vtag',profile,'--stag',regime,'--otag',index,'--rep',index,'--seed',42,'--hang-secs',60,'--hang-dir',str(csvpath.parent)] +def verifykv(csvpath,index,count): + rr=[r for r in rows(csvpath) if r['order']==str(index)] + assert len(rr)==count,(index,len(rr),count) + assert all(int(r['ops'])>0 and int(r['errors'])==0 and int(r['violations'])==0 for r in rr),rr + assert all(int(r['ops'])==int(r['n']) for r in rr if r['phase']=='warm' or r['phase'].startswith('load:')),rr + return rr + +def fixtures(): + folder,m=newmode('fixtures');ff={} + for regime in ['resident','disk']: + for profile,pattern in [('main','ordinary'),('disk-format','ordinary'),('main','common'),('branch-prefix','common'),('main','random'),('branch-prefix','random')]: + label=f'{regime}-{profile}-{pattern}';db=folder/(label+'.db');binary='kvbench' if pattern=='ordinary' else 'prefix_kvbench' + rec=cmdrun(folder,m,label,profile,binary,kvargs(db,folder/(label+'.csv'),profile,regime,0,secs=1,create=True)+['load:random'],dict(PREFIX_PATTERN=pattern) if pattern!='ordinary' else None) + verifykv(folder/(label+'.csv'),0,1) + with db.open('rb') as f:os.fsync(f.fileno()) + rec.update(verified=True,verification_kind='load API errors and operation count; query phases later validate data',errors=0,violations=0,fixture=str(db),fixture_sha256=sha(db),allocated_bytes=db.stat().st_blocks*512,apparent_bytes=db.stat().st_size) + ff[label]={k:rec[k] for k in ['fixture','fixture_sha256','allocated_bytes','apparent_bytes']};save(folder/'manifest.json',m);save(folder/'fixtures.json',ff) + m['complete']=True;save(folder/'manifest.json',m) + +def query_one(folder,m,index,profile,regime,phases,pattern='ordinary'): + f=json.loads((OUT/'fixtures/fixtures.json').read_text());owner=profile if profile in ['disk-format','branch-prefix'] else 'main' + fixture=f[f'{regime}-{owner}-{pattern}'];src=Path(fixture['fixture']);before=src.stat() + db=folder/'scratch.db';assert not db.exists() + subprocess.run(['cp','--reflink=auto','--sparse=always',str(src),str(db)],check=True) + with db.open('rb') as image:os.fsync(image.fileno()) + label=f'{index:02d}-{profile}';csvpath=folder/'results.csv';binary='kvbench' if pattern=='ordinary' else 'prefix_kvbench' + rec=cmdrun(folder,m,label,profile,binary,kvargs(db,csvpath,profile,regime,index,secs=6 if regime=='resident' else 5)+phases,dict(PREFIX_PATTERN=pattern) if pattern!='ordinary' else None) + rr=verifykv(csvpath,index,len(phases)) + rec.update(verified=True,verification_kind='query key/value/range validation and successful close',errors=sum(int(r['errors']) for r in rr),violations=sum(int(r['violations']) for r in rr),fixture=fixture,fixture_copy_fsynced=True,db_allocated_bytes=db.stat().st_blocks*512) + assert src.stat().st_mtime_ns==before.st_mtime_ns + save(folder/'manifest.json',m);db.unlink() + +def queries(regime): + folder,m=newmode(regime) + phases=['warm','point:uniform@1','point:uniform@8','range:uniform:1@1','range:uniform:1@8','range:uniform:10000@8'] if regime=='resident' else ['warm','point:uniform@8','range:uniform:1@8','range:uniform:10000@8'] + m.update(order=ORDER,phases=phases) + for i,profile in enumerate(ORDER):query_one(folder,m,i,profile,regime,phases) + m['complete']=True;save(folder/'manifest.json',m) + +def prefix(): + for regime in ['resident','disk']: + for pattern in ['common','random']: + folder,m=newmode(f'prefix-{regime}-{pattern}') + phases=['warm','point:uniform@8','range:uniform:1@8','range:uniform:10000@8'] + if regime=='resident':phases+=['ow:uniform@8'] + for i,profile in enumerate(['main','branch-prefix','branch-prefix','main']):query_one(folder,m,i,profile,regime,phases,pattern) + m['complete']=True;save(folder/'manifest.json',m) + +def insert(): + folder,m=newmode('insert');data=[] + profiles=['disk-format','filtered-ranges','lazy-merge','async-durable-barrier','cache-invalid-addresses','directory-direct','directory-bucket','directory-bucket-simd','directory-bucket-atomic','log-disabled','log-default'] + for wal,versions in [(1,['main',*profiles,'main',*reversed(profiles),'main']),(0,['main','cache-invalid-addresses','directory-bucket-atomic','directory-bucket-atomic','cache-invalid-addresses','main'])]: + for i,version in enumerate(versions): + profile='log-group-limit' if version.startswith('log-') else version + label=f'wal{wal}-{i:02d}-{version}';stem=folder/label;db=folder/'scratch.db';assert not db.exists() + args=['--db',db,'--csv',str(stem)+'.csv','--summary',str(stem)+'.summary.csv','--records',100000000,'--seconds',5,'--report-ms',100,'--cache-mib',256,'--disk-gib',16,'--direct','--inserters',8,'--readers',0,'--memtable-bg',1,'--normal-bg',3,'--memtable-mib',8,'--key-size',16,'--value-size',16,'--min-free-gib',12,'--seed',42] + if not wal:args+=['--no-log'] + if version.startswith('log-'):args+=['--log-group-size-pages',2**64-1 if version=='log-disabled' else 0] + rec=cmdrun(folder,m,label,profile,'insert_benchmark',args,timeout=240) + summary=rows(Path(str(stem)+'.summary.csv'));assert len(summary)==1;summary=summary[0];inserted=int(summary['inserted']);assert inserted>0 + log=(folder/(label+'.log')).read_text();error_match=re.search(r'load_end .* error=([0-9]+)(?: |\n)',log);assert error_match and int(error_match[1])==0 + close=re.search(r'close_s=([\d.]+) close_rc=0',log);assert close + interval=rows(Path(str(stem)+'.csv'));cpu=sum(float(r['cpu_cores'])*float(r['interval_s']) for r in interval) + data.append(dict(wal=wal,version=version,index=i,errors=int(error_match[1]),violations=0,verification_kind='API return, positive insert count, successful close; no readback',ops_s=float(summary['achieved_insert_ops_s']),inserted=inserted,close_s=float(close[1]),drain_ops_s=inserted/(float(summary['elapsed_s'])+float(close[1])),cpu_us_op=cpu*1e6/inserted,p99_us=1000*float(summary['insert_service_p99_ms']),max_ms=float(summary['insert_service_max_ms']),read_B_op=sum(float(r['read_mib_s'])*float(r['interval_s']) for r in interval)*2**20/inserted,write_B_op=sum(float(r['write_mib_s'])*float(r['interval_s']) for r in interval)*2**20/inserted)) + rec.update(verified=True,errors=int(error_match[1]),verification_kind='API return, positive insert count, successful close; no readback',allocated_bytes=db.stat().st_blocks*512);save(folder/'manifest.json',m);csvwrite(folder/'results.csv',data);db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +def hash_range(): + folder,m=newmode('hash');reference=None + for i,profile in enumerate(['main','filtered-ranges','filtered-ranges','main']): + label=f'{i}-{profile}';db=folder/'scratch.db';rec=cmdrun(folder,m,label,profile,'hash_range_benchmark',[db,16384,1000],timeout=900) + lines=(folder/(label+'.log')).read_text().splitlines();start=next(j for j,x in enumerate(lines) if x.startswith('mode,keys_per_tenant,'));rr=list(csv.DictReader(lines[start:]));assert len(rr)==4 + assert [int(r['limit']) for r in rr]==[1,2,10,100] and all(int(r['queries'])==1000 and int(r['keys_per_tenant'])==16384 for r in rr),rr + checks=[x['checksum'] for x in rr] + if reference is None:reference=checks + assert checks==reference + rec.update(verified=True,rows=rr,allocated_bytes=db.stat().st_blocks*512);save(folder/'manifest.json',m);db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +def barrier(): + folder,m=newmode('barrier') + for wal in [0,1]: + for workers in [0,1]: + order=[('main','sync'),('async-durable-barrier','sync'),('async-durable-barrier','async'),('async-durable-barrier','async'),('async-durable-barrier','sync'),('main','sync')] + for i,(profile,mode) in enumerate(order): + label=f'wal{wal}-workers{workers}-{i}-{profile}-{mode}';db=folder/'scratch.db';csvpath=folder/'results.csv' + rec=cmdrun(folder,m,label,profile,'barrier_bench',[db,csvpath,label,mode,wal,workers,256,2048 if wal else 512],dict(BARRIER_DEPTH='1')) + rr=rows(csvpath);assert rr[-1]['label']==label and int(rr[-1]['verified'])==1 + rec.update(verified=True,allocated_bytes=db.stat().st_blocks*512);save(folder/'manifest.json',m);db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +if __name__=='__main__': + def interrupted(signum,frame):raise InterruptedError(f'benchmark interrupted by signal {signum}') + for signum in [signal.SIGTERM,signal.SIGHUP]:signal.signal(signum,interrupted) + p=argparse.ArgumentParser();p.add_argument('mode',choices=['fixtures','resident','disk','prefix','insert','hash','barrier']);a=p.parse_args() + {'fixtures':fixtures,'resident':lambda:queries('resident'),'disk':lambda:queries('disk'),'prefix':prefix,'insert':insert,'hash':hash_range,'barrier':barrier}[a.mode]() diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/status.json b/benchmarks/results-20261013/main-7bb5fab/benchmarks/status.json new file mode 100644 index 000000000..1ec2659f5 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/status.json @@ -0,0 +1,33 @@ +{ + "project": "disk-format", + "scope": "new main versus updated feature; merge cost not isolated", + "workloads": [ + { + "workload": "insert", + "runs": 7, + "feature_runs": 2, + "verified": 7, + "complete": true, + "warmup_runs": 0 + }, + { + "workload": "disk", + "runs": 5, + "feature_runs": 2, + "verified": 5, + "complete": true, + "warmup_runs": 0 + }, + { + "workload": "resident", + "runs": 5, + "feature_runs": 2, + "verified": 5, + "complete": true, + "warmup_runs": 0 + } + ], + "reported_errors": 0, + "reported_violations": 0, + "all_runs_verified": true +} diff --git a/benchmarks/results-20261013/main-7bb5fab/benchmarks/summary.csv b/benchmarks/results-20261013/main-7bb5fab/benchmarks/summary.csv new file mode 100644 index 000000000..ab642b32d --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/benchmarks/summary.csv @@ -0,0 +1,10 @@ +workload,phase,version,samples,ops_s_mean,ops_s_min,ops_s_max,baseline_ops_s_mean,baseline_ops_s_min,baseline_ops_s_max,delta_pct,ranges_overlap,errors,violations,cpu_us_op,read_B_op,write_B_op,p99_us,scope +resident,point:uniform@1,disk-format,2,1173154.55,1144382.6,1201926.5,1187473.9333333333,1148133.5,1261757.3,-1.2058692769059465,True,0,0,0.8528847295744506,0.0,0.0,1.168,new main versus updated feature; merge cost not isolated +resident,point:uniform@8,disk-format,2,5617125.1,5337003.8,5897246.4,5726823.7,5270841.2,6445634.1,-1.91552256096168,True,0,0,1.4274369781400575,0.0,0.0,1.952,new main versus updated feature; merge cost not isolated +resident,range:uniform:1@1,disk-format,2,110090.3,107715.5,112465.1,108416.73333333334,105112.6,114690.1,1.5436424020646156,True,0,0,9.089448346676637,0.0,0.01,10.751999999999999,new main versus updated feature; merge cost not isolated +resident,range:uniform:1@8,disk-format,2,537979.8,514530.3,561429.3,560819.8666666667,532530.7,600376.6,-4.072620822514839,True,0,0,14.894846457168455,0.0,0.0,19.712,new main versus updated feature; merge cost not isolated +resident,range:uniform:10000@8,disk-format,2,3910.25,3716.5,4104.0,4006.933333333333,3785.8,4305.3,-2.4129009716491345,True,0,0,2050.488242926407,0.0,0.175,2490.368,new main versus updated feature; merge cost not isolated +disk,point:uniform@8,disk-format,2,160050.35,152478.2,167622.5,161357.13333333333,153061.5,168110.0,-0.8098701968345945,True,0,0,5.764932708266299,3229.14,0.0,101.376,new main versus updated feature; merge cost not isolated +disk,range:uniform:1@8,disk-format,2,63523.45,62315.3,64731.6,64019.63333333333,63458.1,64975.4,-0.7750486959989922,True,0,0,133.3279180665298,19797.525,0.0,247.808,new main versus updated feature; merge cost not isolated +disk,range:uniform:10000@8,disk-format,2,835.4,821.0,849.8,838.8666666666667,835.2,841.4,-0.41325598029087196,True,0,0,3349.4787768362203,2375536.705,0.0,23330.816,new main versus updated feature; merge cost not isolated +insert,WAL1,disk-format,2,2263352.0526735,2207885.214204,2318818.891143,2283542.087203333,2261423.232769,2296488.128709,-0.8841542550485637,True,0,0,2.6033216004679063,11.656607718383256,118.31430321239577,63.488,new main versus updated feature; merge cost not isolated diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/README.md b/benchmarks/results-20261013/main-7bb5fab/tests/README.md new file mode 100644 index 000000000..1f158cd9e --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/README.md @@ -0,0 +1,11 @@ +# disk-format: correctness evidence for main integration + +Branch: `robj/disk-format`. Merge head when archived: `d21b87f5cfac709fb0fa74c3585fb38936c08852`. Integrated main: `7bb5fab305b9c6179e31f75998dabea833537a08`. + +`test_integration.py debug` and `test_integration.py release` selected btree, superblock, core, quick, trunk async, filter, and log suites. `validation-debug.json` and `validation-release.json` record final commands and passing outcomes. Initial compilation failures during merge resolution remain in the build logs. `*-4gib.*` preserves the first core suite runs: the new large blob test exceeded the configured 4 GiB logical capacity. The corrected 64 GiB logical fixtures passed in debug and release. + +Build logs describe compilation; JSON records retain the exact executed commands and available source, binary, and log hashes. The archive includes intermediate failures as well as successful final validation; an early failed log is not a claim that the final build failed. Tests with databases used real disk storage under the workspace. `shared/run_job.py` is the coordination wrapper used to serialize timed work against builds and tests. Its syntax is `python3 run_job.py --name NAME test -- COMMAND`; build jobs use the `build` mode. + +`shared/before.json` records the original project heads and backup refs. The scripts are archived byte for byte for command provenance; they contain original workspace paths and may depend on their original directory layout. Do not run them directly from this archive without adapting paths and choosing fresh scratch locations. + +Original evidence directory: `/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format`. Only regular files with `.json`, `.log`, `.py`, `.gdb`, or `.md` extensions and at most 1 MiB each were copied. Scratch directories, databases, executables, core dumps, symlinks, bytecode, and shared benchmark outputs are excluded. Raw logs retain their original bytes. `archive-manifest.json` hashes every archived file except itself; it does not hash database files. diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/archive-manifest.json b/benchmarks/results-20261013/main-7bb5fab/tests/archive-manifest.json new file mode 100644 index 000000000..1b5e7007a --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/archive-manifest.json @@ -0,0 +1,214 @@ +{ + "project": "disk-format", + "branch": "robj/disk-format", + "merge_head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "integrated_main": "7bb5fab305b9c6179e31f75998dabea833537a08", + "files": [ + { + "path": "README.md", + "bytes": 2087, + "sha256": "c6973738a9ff770ca29cf6d23c581beb9795c2c114572bb1fd0ebb23432d1e3e", + "source": "generated archive guide" + }, + { + "path": "build-debug-3.log", + "bytes": 2645, + "sha256": "0ca67435c379811d39b3c6447642a4aa39cc7fad4909812f7a15366000fe74a0", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-debug-3.log" + }, + { + "path": "build-debug-4.log", + "bytes": 1050, + "sha256": "2d2a2ae60de93416062f879361aeaac014528ae8147922a943e833933facdf0e", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-debug-4.log" + }, + { + "path": "build-debug-5.log", + "bytes": 615, + "sha256": "87e86d57017778cb00c49bdbb29447f0d50505de79049dc22bc77d3f2a98aad9", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-debug-5.log" + }, + { + "path": "build-debug-final.log", + "bytes": 9826, + "sha256": "7b33ebeed7ab9431c221c444ef151ae009548064525fec25ab12f4d9b95a3305", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-debug-final.log" + }, + { + "path": "build-debug-formatted.log", + "bytes": 2014, + "sha256": "be00255ea6bc75794bfa659f28ceb9e28a2144f8552518ae8b01e5b6ffc5c005", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-debug-formatted.log" + }, + { + "path": "build-debug-resolved.log", + "bytes": 1808, + "sha256": "9a3be29c63ca79da78381300ed25248f7c6089e4613f64ae58999f98b04a66fc", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-debug-resolved.log" + }, + { + "path": "build-debug.log", + "bytes": 2695, + "sha256": "72da8e8587c1a9bd7946519a223a788f72b64320727055aa3d97e79c7ee9aeef", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-debug.log" + }, + { + "path": "build-release.log", + "bytes": 13641, + "sha256": "c6cabf428fd6973cd2bd1fa01b04be25c538f891ac547c2cf7f16834c3301bf7", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/build-release.log" + }, + { + "path": "debug-btree.log", + "bytes": 2305, + "sha256": "f8ca9a5b8c682cd418b940a5afef55253c9dc165dced3c780c25b4b141e8d05e", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-btree.log" + }, + { + "path": "debug-filter.log", + "bytes": 3865, + "sha256": "49259f7e14bc368c5ba8d9410dd23b58d270333eaf6b13736bbd038ca9b1d924", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-filter.log" + }, + { + "path": "debug-log.log", + "bytes": 752, + "sha256": "372a0c0e44520c2d299d25bde942a0dfdd13119b77cfa2b5e4cc032e3e4d8095", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-log.log" + }, + { + "path": "debug-quick.log", + "bytes": 3943, + "sha256": "2e348ee7750ccc4c95d7c839ff752e2bb23183c605af0d0df0efabcb46033421", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-quick.log" + }, + { + "path": "debug-splinter-4gib.log", + "bytes": 6635, + "sha256": "19abfb216913e1d89c0d6ef7f7dc7cd0ff4275f8de44e5e2cd7a3ca094b90f1f", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-splinter-4gib.log" + }, + { + "path": "debug-splinter.log", + "bytes": 6580, + "sha256": "20208a30e6dcf9076de207d0712849e24113084851af3813c3bdbd9d7e492a7f", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-splinter.log" + }, + { + "path": "debug-superblock.log", + "bytes": 2781, + "sha256": "417133bc50d178e582c1bdcbff4182a07ced3994a81d29eb409f4b713300f7d2", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-superblock.log" + }, + { + "path": "debug-trunk-async.log", + "bytes": 438, + "sha256": "358786a8840bf748a73c6f3d58cb3eb2bac80a7107a6387bfa11a167fe5265ed", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/debug-trunk-async.log" + }, + { + "path": "manifest.json", + "bytes": 27637, + "sha256": "e8214285ef7e8a67ecb37a56cb076f1a47c08a84048e6f0a6bc24d2769708970", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/manifest.json" + }, + { + "path": "release-btree.log", + "bytes": 2304, + "sha256": "43c08095c908b960b7c40ce73c84804496b2538fd4e5d1e6d67cf78f14d311f5", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-btree.log" + }, + { + "path": "release-filter.log", + "bytes": 3867, + "sha256": "bcfd592c4f9020413d7751bbf1e69d0de0a8c21e9953b19768e35c17564cd7e5", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-filter.log" + }, + { + "path": "release-log.log", + "bytes": 754, + "sha256": "94d99f7647332b2b2a3310da728d30fddc459ed2c722060f5d0fc1bc4003e55e", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-log.log" + }, + { + "path": "release-quick.log", + "bytes": 3943, + "sha256": "fb9f96d901f9d6653e0295bcffe6220ded0a75f71db48ec3a5c763be57a50b0c", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-quick.log" + }, + { + "path": "release-splinter-4gib.log", + "bytes": 6786, + "sha256": "40f1c0c881534b1c7584bf5db4e324181858eb9668642ca7d83649627dd82664", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-splinter-4gib.log" + }, + { + "path": "release-splinter.log", + "bytes": 6580, + "sha256": "64e3e5c3f4c796d2be3661ddbc06e3da816516e1497a2b98aafc2a84a6947de7", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-splinter.log" + }, + { + "path": "release-superblock.log", + "bytes": 2781, + "sha256": "e8f058f14fdc7b2ae6c78a4983c56e5ae3c3cee7cdc4b46a303efa3a18cb2b1c", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-superblock.log" + }, + { + "path": "release-trunk-async.log", + "bytes": 438, + "sha256": "58337bde38e57564caa2561b024f9e05bedf6d7c6d6f1b220fee48ef6c622449", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/release-trunk-async.log" + }, + { + "path": "shared/before.json", + "bytes": 4460, + "sha256": "133ae8f9f5fda20e884fd1d283d857d389a96cef6a74a82b7a17fa094a0cc6e5", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/before.json" + }, + { + "path": "shared/run_job.py", + "bytes": 2461, + "sha256": "908d86de805e6efd41b423c919187f2df14715104f047e10fa8a6c57129cc773", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/run_job.py" + }, + { + "path": "shared/test_root_branches.py", + "bytes": 3223, + "sha256": "36f5b93940442f3806266d12284fe4da2927b56b4f1882c957df7519696d3da7", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/test_root_branches.py" + }, + { + "path": "test_integration.py", + "bytes": 1600, + "sha256": "cabf042d58089771e6d134241f594e273d93bd259b47d77a660c69424a9ed298", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/test_integration.py" + }, + { + "path": "validation-debug-4gib.json", + "bytes": 1319, + "sha256": "985687f55c536e818cab8a71be2af260dbe1a53c8a74fd3a041ad4ed58cea79c", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/validation-debug-4gib.json" + }, + { + "path": "validation-debug.json", + "bytes": 3094, + "sha256": "d1be87ee55ad22eb4d14b295ff62b78e8e77ca02e9031cc13aefdfb3e325f853", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/validation-debug.json" + }, + { + "path": "validation-release-4gib.json", + "bytes": 1252, + "sha256": "6333e64bcc5246e1accccbb2058edbb599cecc1b089e77ee9a82116d420ffda0", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/validation-release-4gib.json" + }, + { + "path": "validation-release.json", + "bytes": 3111, + "sha256": "ecfd4bd663739ae819754ef4239ebcde4796d3cef6615d0ac291b547c46ccbb0", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/main-7bb5fab/disk-format/validation-release.json" + } + ], + "manifest_excludes_itself": true, + "archived_payload_bytes": 139290 +} diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-3.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-3.log new file mode 100644 index 000000000..985e73c43 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-3.log @@ -0,0 +1,38 @@ +Starting mainmerge-disk-format-debug-build-3 build +Checking for mismatched config...No mismatched config found +Compiling tests/unit/splinter_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_quick_test +Linking build-main-7bb5fab/debug/bin/unit/superblock_test +Linking build-main-7bb5fab/debug/bin/unit/task_system_test +Linking build-main-7bb5fab/debug/bin/unit/trunk_async_test +Linking build-main-7bb5fab/debug/bin/unit/trunk_gc_test +Linking build-main-7bb5fab/debug/bin/unit/trunk_split_test +Linking build-main-7bb5fab/debug/bin/unit/util_test +Linking build-main-7bb5fab/debug/bin/unit/vector_test +Linking build-main-7bb5fab/debug/bin/unit/writable_buffer_test +Linking build-main-7bb5fab/debug/bin/unit/writeback_set_test +Compiling tools/splinter_inspect.c [build-main-7bb5fab/debug/obj/tools/splinter_inspect.o] +Building archive build-main-7bb5fab/debug/lib/libsplinterdb.a +Linking build-main-7bb5fab/debug/bin/driver_test +Linking build-main-7bb5fab/debug/bin/unit_test +In file included from tests/unit/unit_tests.h:8, + from tests/unit/splinter_test.c:38: +tests/unit/splinter_test.c: In function ‘ctest_splinter_test_blob_height_three_and_large_stats_run’: +src/data_internal.h:45:4: error: incompatible type for argument 4 of ‘btree_count_in_range’ + 45 | ((key){.kind = NEGATIVE_INFINITY, \ + | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + | | + | key + 46 | .is_query_key = FALSE, \ + | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + 47 | .user_slice = INVALID_SLICE}) + | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +tests/unit/ctest.h:292:16: note: in definition of macro ‘ASSERT_TRUE’ + 292 | assert_true(real, __FILE__, __LINE__, "" __VA_ARGS__) + | ^~~~ +tests/unit/splinter_test.c:6048:45: note: in expansion of macro ‘NEGATIVE_INFINITY_KEY’ + 6048 | NEGATIVE_INFINITY_KEY, + | ^~~~~~~~~~~~~~~~~~~~~ +compilation terminated due to -Wfatal-errors. +make: *** [Makefile:365: build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] Error 1 +make: *** Waiting for unfinished jobs.... diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-4.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-4.log new file mode 100644 index 000000000..24dcb6662 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-4.log @@ -0,0 +1,14 @@ +Starting mainmerge-disk-format-debug-build-4 build +Checking for mismatched config...No mismatched config found +Compiling tests/unit/splinter_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] +In file included from tests/unit/unit_tests.h:8, + from tests/unit/splinter_test.c:38: +tests/unit/splinter_test.c: In function ‘ctest_splinter_test_blob_height_three_and_large_stats_run’: +tests/unit/splinter_test.c:6048:45: error: ‘BTREE_FORMAT_VERSION’ undeclared (first use in this function); did you mean ‘TRUNK_FORMAT_VERSION’? + 6048 | BTREE_FORMAT_VERSION, + | ^~~~~~~~~~~~~~~~~~~~ +tests/unit/ctest.h:292:16: note: in definition of macro ‘ASSERT_TRUE’ + 292 | assert_true(real, __FILE__, __LINE__, "" __VA_ARGS__) + | ^~~~ +compilation terminated due to -Wfatal-errors. +make: *** [Makefile:365: build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] Error 1 diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-5.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-5.log new file mode 100644 index 000000000..916e6b6f6 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-5.log @@ -0,0 +1,8 @@ +Starting mainmerge-disk-format-debug-build-5 build +Checking for mismatched config...No mismatched config found +Compiling tests/unit/splinter_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] +Linking build-main-7bb5fab/debug/bin/unit/config_parse_test +Linking build-main-7bb5fab/debug/bin/unit/large_inserts_stress_test +Linking build-main-7bb5fab/debug/bin/unit/splinter_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_forked_child_test +Linking build-main-7bb5fab/debug/bin/splinter_inspect diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-final.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-final.log new file mode 100644 index 000000000..a13f8de08 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-final.log @@ -0,0 +1,84 @@ +Starting mainmerge-disk-format-debug-build-final build +Checking for mismatched config...No mismatched config found +Compiling tests/functional/filter_test.c [build-main-7bb5fab/debug/obj/tests/functional/filter_test.o] +Compiling tests/functional/log_test.c [build-main-7bb5fab/debug/obj/tests/functional/log_test.o] +Compiling tests/functional/scan_benchmark.c [build-main-7bb5fab/debug/obj/tests/functional/scan_benchmark.o] +Compiling tests/functional/splinter_test.c [build-main-7bb5fab/debug/obj/tests/functional/splinter_test.o] +Compiling tests/functional/test_async.c [build-main-7bb5fab/debug/obj/tests/functional/test_async.o] +Compiling tests/functional/test_dispatcher.c [build-main-7bb5fab/debug/obj/tests/functional/test_dispatcher.o] +Compiling tests/functional/test_functionality.c [build-main-7bb5fab/debug/obj/tests/functional/test_functionality.o] +Compiling tests/functional/test_splinter_shadow.c [build-main-7bb5fab/debug/obj/tests/functional/test_splinter_shadow.o] +Compiling tests/functional/ycsb_test.c [build-main-7bb5fab/debug/obj/tests/functional/ycsb_test.o] +Compiling tests/config.c [build-main-7bb5fab/debug/obj/tests/config.o] +Compiling tests/test_common.c [build-main-7bb5fab/debug/obj/tests/test_common.o] +Compiling tests/test_data.c [build-main-7bb5fab/debug/obj/tests/test_data.o] +Compiling src/allocator.c [build-main-7bb5fab/debug/obj/src/allocator.o] +Compiling src/blob.c [build-main-7bb5fab/debug/obj/src/blob.o] +Compiling src/blob_build.c [build-main-7bb5fab/debug/obj/src/blob_build.o] +Compiling src/btree.c [build-main-7bb5fab/debug/obj/src/btree.o] +Compiling src/clockcache.c [build-main-7bb5fab/debug/obj/src/clockcache.o] +Compiling src/data_blob_build.c [build-main-7bb5fab/debug/obj/src/data_blob_build.o] +Compiling src/data_internal.c [build-main-7bb5fab/debug/obj/src/data_internal.o] +Compiling src/histogram.c [build-main-7bb5fab/debug/obj/src/histogram.o] +Compiling src/mini_allocator.c [build-main-7bb5fab/debug/obj/src/mini_allocator.o] +Compiling src/ondisk_ref.c [build-main-7bb5fab/debug/obj/src/ondisk_ref.o] +Compiling src/platform_linux/laio.c [build-main-7bb5fab/debug/obj/src/platform_linux/laio.o] +Compiling src/platform_linux/platform_assert.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_assert.o] +Compiling src/platform_linux/platform_buffer.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_buffer.o] +Compiling src/platform_linux/platform_condvar.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_condvar.o] +Compiling src/platform_linux/platform_heap.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_heap.o] +Compiling src/platform_linux/platform_io.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_io.o] +Compiling src/platform_linux/platform_log.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_log.o] +Compiling src/platform_linux/platform_mutex.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_mutex.o] +Compiling src/platform_linux/platform_threads.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_threads.o] +Compiling src/platform_linux/platform_trace.c [build-main-7bb5fab/debug/obj/src/platform_linux/platform_trace.o] +Compiling src/platform_linux/shmalloc.c [build-main-7bb5fab/debug/obj/src/platform_linux/shmalloc.o] +Compiling src/rc_allocator.c [build-main-7bb5fab/debug/obj/src/rc_allocator.o] +Compiling src/task.c [build-main-7bb5fab/debug/obj/src/task.o] +Compiling src/util.c [build-main-7bb5fab/debug/obj/src/util.o] +Compiling src/writeback_set.c [build-main-7bb5fab/debug/obj/src/writeback_set.o] +Compiling tests/unit/btree_stress_test.c [build-main-7bb5fab/debug/obj/tests/unit/btree_stress_test.o] +Compiling tests/unit/btree_test.c [build-main-7bb5fab/debug/obj/tests/unit/btree_test.o] +Compiling tests/unit/btree_test_common.c [build-main-7bb5fab/debug/obj/tests/unit/btree_test_common.o] +Compiling tests/unit/limitations_test.c [build-main-7bb5fab/debug/obj/tests/unit/limitations_test.o] +Compiling tests/unit/main.c [build-main-7bb5fab/debug/obj/tests/unit/main.o] +Compiling tests/unit/merge_test.c [build-main-7bb5fab/debug/obj/tests/unit/merge_test.o] +Compiling tests/unit/misc_test.c [build-main-7bb5fab/debug/obj/tests/unit/misc_test.o] +Compiling tests/unit/platform_apis_test.c [build-main-7bb5fab/debug/obj/tests/unit/platform_apis_test.o] +Compiling tests/unit/platform_threads_test.c [build-main-7bb5fab/debug/obj/tests/unit/platform_threads_test.o] +Compiling tests/unit/splinterdb_heap_id_mgmt_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinterdb_heap_id_mgmt_test.o] +Compiling tests/unit/splinterdb_optimize_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinterdb_optimize_test.o] +Compiling tests/unit/splinterdb_quick_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinterdb_quick_test.o] +Compiling tests/unit/splinterdb_stress_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinterdb_stress_test.o] +Compiling tests/unit/superblock_test.c [build-main-7bb5fab/debug/obj/tests/unit/superblock_test.o] +Compiling tests/unit/task_system_test.c [build-main-7bb5fab/debug/obj/tests/unit/task_system_test.o] +Compiling tests/unit/trunk_async_test.c [build-main-7bb5fab/debug/obj/tests/unit/trunk_async_test.o] +Compiling tests/unit/trunk_gc_test.c [build-main-7bb5fab/debug/obj/tests/unit/trunk_gc_test.o] +Compiling tests/unit/trunk_split_test.c [build-main-7bb5fab/debug/obj/tests/unit/trunk_split_test.o] +Compiling tests/unit/unit_tests_common.c [build-main-7bb5fab/debug/obj/tests/unit/unit_tests_common.o] +Compiling tests/unit/util_test.c [build-main-7bb5fab/debug/obj/tests/unit/util_test.o] +Compiling tests/unit/vector_test.c [build-main-7bb5fab/debug/obj/tests/unit/vector_test.o] +Compiling tests/unit/writable_buffer_test.c [build-main-7bb5fab/debug/obj/tests/unit/writable_buffer_test.o] +Compiling tests/unit/writeback_set_test.c [build-main-7bb5fab/debug/obj/tests/unit/writeback_set_test.o] +Linking build-main-7bb5fab/debug/lib/libsplinterdb.so +Linking build-main-7bb5fab/debug/bin/unit/btree_stress_test +Linking build-main-7bb5fab/debug/bin/unit/btree_test +Compiling tests/unit/config_parse_test.c [build-main-7bb5fab/debug/obj/tests/unit/config_parse_test.o] +Compiling tests/unit/large_inserts_stress_test.c [build-main-7bb5fab/debug/obj/tests/unit/large_inserts_stress_test.o] +Linking build-main-7bb5fab/debug/bin/unit/limitations_test +Linking build-main-7bb5fab/debug/bin/unit/merge_test +Linking build-main-7bb5fab/debug/bin/unit/misc_test +Linking build-main-7bb5fab/debug/bin/unit/platform_apis_test +Linking build-main-7bb5fab/debug/bin/unit/platform_threads_test +Compiling tests/unit/splinter_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] +Compiling tests/unit/splinterdb_forked_child_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinterdb_forked_child_test.o] +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_heap_id_mgmt_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_optimize_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_stress_test +tests/unit/splinter_test.c: In function ‘ctest_splinter_test_invalid_branch_reference_rejected_run’: +tests/unit/splinter_test.c:1533:7: error: incompatible types when initializing type ‘page_handle *’ using type ‘page_handle’ + 1533 | test_splinter_claim_page(cc, root_addr, PAGE_TYPE_TRUNK); + | ^~~~~~~~~~~~~~~~~~~~~~~~ +compilation terminated due to -Wfatal-errors. +make: *** [Makefile:364: build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] Error 1 +make: *** Waiting for unfinished jobs.... diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-formatted.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-formatted.log new file mode 100644 index 000000000..41437a6d7 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-formatted.log @@ -0,0 +1,26 @@ +Starting mainmerge-disk-format-debug-formatted build +Checking for mismatched config...No mismatched config found +Compiling tests/functional/filter_test.c [build-main-7bb5fab/debug/obj/tests/functional/filter_test.o] +Compiling src/shard_log.c [build-main-7bb5fab/debug/obj/src/shard_log.o] +Compiling tests/unit/splinter_test.c [build-main-7bb5fab/debug/obj/tests/unit/splinter_test.o] +Linking build-main-7bb5fab/debug/lib/libsplinterdb.so +Building archive build-main-7bb5fab/debug/lib/libsplinterdb.a +Linking build-main-7bb5fab/debug/bin/splinter_inspect +Linking build-main-7bb5fab/debug/bin/driver_test +Linking build-main-7bb5fab/debug/bin/unit_test +Linking build-main-7bb5fab/debug/bin/unit/config_parse_test +Linking build-main-7bb5fab/debug/bin/unit/large_inserts_stress_test +Linking build-main-7bb5fab/debug/bin/unit/limitations_test +Linking build-main-7bb5fab/debug/bin/unit/merge_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_forked_child_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_heap_id_mgmt_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_optimize_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_quick_test +Linking build-main-7bb5fab/debug/bin/unit/splinterdb_stress_test +Linking build-main-7bb5fab/debug/bin/unit/superblock_test +Linking build-main-7bb5fab/debug/bin/unit/task_system_test +Linking build-main-7bb5fab/debug/bin/unit/trunk_async_test +Linking build-main-7bb5fab/debug/bin/unit/trunk_gc_test +Linking build-main-7bb5fab/debug/bin/unit/trunk_split_test +Linking build-main-7bb5fab/debug/bin/unit/writable_buffer_test +Linking build-main-7bb5fab/debug/bin/unit/splinter_test diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-resolved.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-resolved.log new file mode 100644 index 000000000..3b16c15b5 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug-resolved.log @@ -0,0 +1,18 @@ +Starting mainmerge-disk-format-debug-build-resolved build +Checking for mismatched config...No mismatched config found +Compiling src/routing_filter.c [build-main-7bb5fab/debug/obj/src/routing_filter.o] +Compiling src/shard_log.c [build-main-7bb5fab/debug/obj/src/shard_log.o] +Compiling src/trunk.c [build-main-7bb5fab/debug/obj/src/trunk.o] +Compiling tests/functional/avlTree.c [build-main-7bb5fab/debug/obj/tests/functional/avlTree.o] +Compiling tests/functional/btree_test.c [build-main-7bb5fab/debug/obj/tests/functional/btree_test.o] +Compiling tests/functional/cache_test.c [build-main-7bb5fab/debug/obj/tests/functional/cache_test.o] +Compiling tests/functional/driver_test.c [build-main-7bb5fab/debug/obj/tests/functional/driver_test.o] +Compiling tests/functional/filter_test.c [build-main-7bb5fab/debug/obj/tests/functional/filter_test.o] +Compiling tests/functional/io_apis_test.c [build-main-7bb5fab/debug/obj/tests/functional/io_apis_test.o] +tests/functional/filter_test.c: In function ‘test_filter_malicious_metadata’: +tests/functional/filter_test.c:362:16: error: incompatible types when assigning to type ‘page_handle *’ from type ‘page_handle’ + 362 | meta_page = test_filter_claim_page(cc, filter.meta_head, PAGE_TYPE_FILTER); + | ^~~~~~~~~~~~~~~~~~~~~~ +compilation terminated due to -Wfatal-errors. +make: *** [Makefile:365: build-main-7bb5fab/debug/obj/tests/functional/filter_test.o] Error 1 +make: *** Waiting for unfinished jobs.... diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-debug.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug.log new file mode 100644 index 000000000..c7ff70b15 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-debug.log @@ -0,0 +1,33 @@ +Starting mainmerge-disk-format-debug-build build +Checking for mismatched config...No mismatched config found +Saving config to build-main-7bb5fab/debug/build-config.e33dfc688a9c2bf2dad4e17ef758ec3e +Compiling src/PackedArray.c [build-main-7bb5fab/debug/obj/src/PackedArray.o] +Compiling src/batch_rwlock.c [build-main-7bb5fab/debug/obj/src/batch_rwlock.o] +Compiling src/core.c [build-main-7bb5fab/debug/obj/src/core.o] +Compiling src/default_data_config.c [build-main-7bb5fab/debug/obj/src/default_data_config.o] +Compiling src/memtable.c [build-main-7bb5fab/debug/obj/src/memtable.o] +Compiling src/merge.c [build-main-7bb5fab/debug/obj/src/merge.o] +Compiling src/notification.c [build-main-7bb5fab/debug/obj/src/notification.o] +Compiling src/routing_filter.c [build-main-7bb5fab/debug/obj/src/routing_filter.o] +Compiling src/shard_log.c [build-main-7bb5fab/debug/obj/src/shard_log.o] +Compiling src/splinterdb.c [build-main-7bb5fab/debug/obj/src/splinterdb.o] +Compiling src/superblock.c [build-main-7bb5fab/debug/obj/src/superblock.o] +src/routing_filter.c: In function ‘routing_filter_add’: +src/routing_filter.c:849:1: error: version control conflict marker in file + 849 | <<<<<<< HEAD + | ^~~~~~~ +compilation terminated due to -Wfatal-errors. +make: *** [Makefile:365: build-main-7bb5fab/debug/obj/src/routing_filter.o] Error 1 +make: *** Waiting for unfinished jobs.... +src/shard_log.c: In function ‘shard_log_iterator_create_recovered’: +src/shard_log.c:3101:1: error: version control conflict marker in file + 3101 | <<<<<<< HEAD + | ^~~~~~~ +compilation terminated due to -Wfatal-errors. +make: *** [Makefile:365: build-main-7bb5fab/debug/obj/src/shard_log.o] Error 1 +In file included from src/PackedArray.c:296, + from src/PackedArray.c:356: +src/PackedArray.c: In function ‘__PackedArray_unpack_21’: +src/PackedArray.c:177: note: ‘-Wmisleading-indentation’ is disabled from this point onwards, since column-tracking was disabled due to the size of the code/headers + 177 | PACKEDARRAY_IMPL_UNPACK_CASE_BREAK +src/PackedArray.c:177: note: adding ‘-flarge-source-files’ will allow for more column-tracking support, at the expense of compilation time and memory diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/build-release.log b/benchmarks/results-20261013/main-7bb5fab/tests/build-release.log new file mode 100644 index 000000000..0399907c0 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/build-release.log @@ -0,0 +1,120 @@ +Starting mainmerge-disk-format-release-build build +Checking for mismatched config...No mismatched config found +Saving config to build-main-7bb5fab/release/build-config.fb6bc8eeb8390429253883c09491f5bd +Compiling src/PackedArray.c [build-main-7bb5fab/release/obj/src/PackedArray.o] +Compiling src/batch_rwlock.c [build-main-7bb5fab/release/obj/src/batch_rwlock.o] +Compiling src/core.c [build-main-7bb5fab/release/obj/src/core.o] +Compiling src/default_data_config.c [build-main-7bb5fab/release/obj/src/default_data_config.o] +Compiling src/memtable.c [build-main-7bb5fab/release/obj/src/memtable.o] +Compiling src/merge.c [build-main-7bb5fab/release/obj/src/merge.o] +Compiling src/notification.c [build-main-7bb5fab/release/obj/src/notification.o] +Compiling src/routing_filter.c [build-main-7bb5fab/release/obj/src/routing_filter.o] +Compiling src/shard_log.c [build-main-7bb5fab/release/obj/src/shard_log.o] +Compiling src/splinterdb.c [build-main-7bb5fab/release/obj/src/splinterdb.o] +Compiling src/superblock.c [build-main-7bb5fab/release/obj/src/superblock.o] +Compiling src/trunk.c [build-main-7bb5fab/release/obj/src/trunk.o] +Compiling tests/functional/avlTree.c [build-main-7bb5fab/release/obj/tests/functional/avlTree.o] +Compiling tests/functional/btree_test.c [build-main-7bb5fab/release/obj/tests/functional/btree_test.o] +Compiling tests/functional/cache_test.c [build-main-7bb5fab/release/obj/tests/functional/cache_test.o] +Compiling tests/functional/driver_test.c [build-main-7bb5fab/release/obj/tests/functional/driver_test.o] +Compiling tests/functional/filter_test.c [build-main-7bb5fab/release/obj/tests/functional/filter_test.o] +Compiling tests/functional/io_apis_test.c [build-main-7bb5fab/release/obj/tests/functional/io_apis_test.o] +Compiling tests/functional/log_test.c [build-main-7bb5fab/release/obj/tests/functional/log_test.o] +Compiling tests/functional/scan_benchmark.c [build-main-7bb5fab/release/obj/tests/functional/scan_benchmark.o] +Compiling tests/functional/splinter_test.c [build-main-7bb5fab/release/obj/tests/functional/splinter_test.o] +Compiling tests/functional/test_async.c [build-main-7bb5fab/release/obj/tests/functional/test_async.o] +Compiling tests/functional/test_dispatcher.c [build-main-7bb5fab/release/obj/tests/functional/test_dispatcher.o] +Compiling tests/functional/test_functionality.c [build-main-7bb5fab/release/obj/tests/functional/test_functionality.o] +Compiling tests/functional/test_splinter_shadow.c [build-main-7bb5fab/release/obj/tests/functional/test_splinter_shadow.o] +Compiling tests/functional/ycsb_test.c [build-main-7bb5fab/release/obj/tests/functional/ycsb_test.o] +Compiling tests/config.c [build-main-7bb5fab/release/obj/tests/config.o] +Compiling tests/test_common.c [build-main-7bb5fab/release/obj/tests/test_common.o] +Compiling tests/test_data.c [build-main-7bb5fab/release/obj/tests/test_data.o] +Compiling src/allocator.c [build-main-7bb5fab/release/obj/src/allocator.o] +Compiling src/blob.c [build-main-7bb5fab/release/obj/src/blob.o] +Compiling src/blob_build.c [build-main-7bb5fab/release/obj/src/blob_build.o] +Compiling src/btree.c [build-main-7bb5fab/release/obj/src/btree.o] +Compiling src/clockcache.c [build-main-7bb5fab/release/obj/src/clockcache.o] +Compiling src/data_blob_build.c [build-main-7bb5fab/release/obj/src/data_blob_build.o] +Compiling src/data_internal.c [build-main-7bb5fab/release/obj/src/data_internal.o] +Compiling src/histogram.c [build-main-7bb5fab/release/obj/src/histogram.o] +Compiling src/mini_allocator.c [build-main-7bb5fab/release/obj/src/mini_allocator.o] +Compiling src/ondisk_ref.c [build-main-7bb5fab/release/obj/src/ondisk_ref.o] +Compiling src/platform_linux/laio.c [build-main-7bb5fab/release/obj/src/platform_linux/laio.o] +Compiling src/platform_linux/platform_assert.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_assert.o] +Compiling src/platform_linux/platform_buffer.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_buffer.o] +Compiling src/platform_linux/platform_condvar.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_condvar.o] +Compiling src/platform_linux/platform_heap.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_heap.o] +Compiling src/platform_linux/platform_io.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_io.o] +Compiling src/platform_linux/platform_log.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_log.o] +Compiling src/platform_linux/platform_mutex.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_mutex.o] +Compiling src/platform_linux/platform_threads.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_threads.o] +Compiling src/platform_linux/platform_trace.c [build-main-7bb5fab/release/obj/src/platform_linux/platform_trace.o] +Compiling src/platform_linux/shmalloc.c [build-main-7bb5fab/release/obj/src/platform_linux/shmalloc.o] +Compiling src/rc_allocator.c [build-main-7bb5fab/release/obj/src/rc_allocator.o] +Compiling src/task.c [build-main-7bb5fab/release/obj/src/task.o] +Compiling src/util.c [build-main-7bb5fab/release/obj/src/util.o] +Compiling src/writeback_set.c [build-main-7bb5fab/release/obj/src/writeback_set.o] +In file included from src/PackedArray.c:296, + from src/PackedArray.c:356: +src/PackedArray.c: In function ‘__PackedArray_unpack_21’: +src/PackedArray.c:177: note: ‘-Wmisleading-indentation’ is disabled from this point onwards, since column-tracking was disabled due to the size of the code/headers + 177 | PACKEDARRAY_IMPL_UNPACK_CASE_BREAK +src/PackedArray.c:177: note: adding ‘-flarge-source-files’ will allow for more column-tracking support, at the expense of compilation time and memory +Compiling tests/unit/btree_stress_test.c [build-main-7bb5fab/release/obj/tests/unit/btree_stress_test.o] +Compiling tests/unit/btree_test.c [build-main-7bb5fab/release/obj/tests/unit/btree_test.o] +Compiling tests/unit/btree_test_common.c [build-main-7bb5fab/release/obj/tests/unit/btree_test_common.o] +Compiling tests/unit/limitations_test.c [build-main-7bb5fab/release/obj/tests/unit/limitations_test.o] +Compiling tests/unit/main.c [build-main-7bb5fab/release/obj/tests/unit/main.o] +Compiling tests/unit/merge_test.c [build-main-7bb5fab/release/obj/tests/unit/merge_test.o] +Compiling tests/unit/misc_test.c [build-main-7bb5fab/release/obj/tests/unit/misc_test.o] +Compiling tests/unit/platform_apis_test.c [build-main-7bb5fab/release/obj/tests/unit/platform_apis_test.o] +Compiling tests/unit/platform_threads_test.c [build-main-7bb5fab/release/obj/tests/unit/platform_threads_test.o] +Compiling tests/unit/splinterdb_heap_id_mgmt_test.c [build-main-7bb5fab/release/obj/tests/unit/splinterdb_heap_id_mgmt_test.o] +Compiling tests/unit/splinterdb_optimize_test.c [build-main-7bb5fab/release/obj/tests/unit/splinterdb_optimize_test.o] +Compiling tests/unit/splinterdb_quick_test.c [build-main-7bb5fab/release/obj/tests/unit/splinterdb_quick_test.o] +Compiling tests/unit/splinterdb_stress_test.c [build-main-7bb5fab/release/obj/tests/unit/splinterdb_stress_test.o] +Compiling tests/unit/superblock_test.c [build-main-7bb5fab/release/obj/tests/unit/superblock_test.o] +Compiling tests/unit/task_system_test.c [build-main-7bb5fab/release/obj/tests/unit/task_system_test.o] +Compiling tests/unit/trunk_async_test.c [build-main-7bb5fab/release/obj/tests/unit/trunk_async_test.o] +Compiling tests/unit/trunk_gc_test.c [build-main-7bb5fab/release/obj/tests/unit/trunk_gc_test.o] +Compiling tests/unit/trunk_split_test.c [build-main-7bb5fab/release/obj/tests/unit/trunk_split_test.o] +Compiling tests/unit/unit_tests_common.c [build-main-7bb5fab/release/obj/tests/unit/unit_tests_common.o] +Compiling tests/unit/util_test.c [build-main-7bb5fab/release/obj/tests/unit/util_test.o] +Compiling tests/unit/vector_test.c [build-main-7bb5fab/release/obj/tests/unit/vector_test.o] +Compiling tests/unit/writable_buffer_test.c [build-main-7bb5fab/release/obj/tests/unit/writable_buffer_test.o] +Compiling tests/unit/writeback_set_test.c [build-main-7bb5fab/release/obj/tests/unit/writeback_set_test.o] +Linking build-main-7bb5fab/release/bin/unit/btree_stress_test +Linking build-main-7bb5fab/release/bin/unit/btree_test +Compiling tests/unit/config_parse_test.c [build-main-7bb5fab/release/obj/tests/unit/config_parse_test.o] +Compiling tests/unit/large_inserts_stress_test.c [build-main-7bb5fab/release/obj/tests/unit/large_inserts_stress_test.o] +Linking build-main-7bb5fab/release/bin/unit/misc_test +Linking build-main-7bb5fab/release/bin/unit/platform_apis_test +Linking build-main-7bb5fab/release/bin/unit/platform_threads_test +Compiling tests/unit/splinter_test.c [build-main-7bb5fab/release/obj/tests/unit/splinter_test.o] +Compiling tests/unit/splinterdb_forked_child_test.c [build-main-7bb5fab/release/obj/tests/unit/splinterdb_forked_child_test.o] +Linking build-main-7bb5fab/release/bin/unit/util_test +Linking build-main-7bb5fab/release/bin/unit/vector_test +Linking build-main-7bb5fab/release/bin/unit/writeback_set_test +Compiling tools/splinter_inspect.c [build-main-7bb5fab/release/obj/tools/splinter_inspect.o] +Building archive build-main-7bb5fab/release/lib/libsplinterdb.a +Linking build-main-7bb5fab/release/lib/libsplinterdb.so +Linking build-main-7bb5fab/release/bin/splinter_inspect +Linking build-main-7bb5fab/release/bin/driver_test +Linking build-main-7bb5fab/release/bin/unit_test +Linking build-main-7bb5fab/release/bin/unit/config_parse_test +Linking build-main-7bb5fab/release/bin/unit/large_inserts_stress_test +Linking build-main-7bb5fab/release/bin/unit/limitations_test +Linking build-main-7bb5fab/release/bin/unit/merge_test +Linking build-main-7bb5fab/release/bin/unit/splinter_test +Linking build-main-7bb5fab/release/bin/unit/splinterdb_forked_child_test +Linking build-main-7bb5fab/release/bin/unit/splinterdb_heap_id_mgmt_test +Linking build-main-7bb5fab/release/bin/unit/splinterdb_optimize_test +Linking build-main-7bb5fab/release/bin/unit/splinterdb_quick_test +Linking build-main-7bb5fab/release/bin/unit/splinterdb_stress_test +Linking build-main-7bb5fab/release/bin/unit/superblock_test +Linking build-main-7bb5fab/release/bin/unit/task_system_test +Linking build-main-7bb5fab/release/bin/unit/trunk_async_test +Linking build-main-7bb5fab/release/bin/unit/trunk_gc_test +Linking build-main-7bb5fab/release/bin/unit/trunk_split_test +Linking build-main-7bb5fab/release/bin/unit/writable_buffer_test diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-btree.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-btree.log new file mode 100644 index 000000000..ac7cd762e --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-btree.log @@ -0,0 +1,35 @@ +Running 1 CTests, suite name 'btree', test case 'all'. +TEST 1/11 btree:test_leaf_hdr [OK] +TEST 2/11 btree:test_leaf_hdr_search [OK] +TEST 3/11 btree:test_index_hdr [OK] +TEST 4/11 btree:test_index_hdr_search [OK] +TEST 5/11 btree:test_leaf_split [OK] +TEST 6/11 btree:test_defragmented_leaf_bytes [OK] +TEST 7/11 btree:test_defragmented_index_bytes [OK] +TEST 8/11 btree:test_invalid_branch_point_lookup Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 9/11 btree:test_invalid_branch_async_point_lookup Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 10/11 btree:test_invalid_branch_iterator Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 11/11 btree:test_invalid_branch_metadata_queries Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +RESULTS: 11 tests (11 ok, 0 failed, 0 skipped) ran in 11 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-filter.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-filter.log new file mode 100644 index 000000000..31920f0f9 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-filter.log @@ -0,0 +1,65 @@ +build-main-7bb5fab/debug/bin/driver_test: splinterdb_build_version fb878f13-dirty +Dispatch test filter_test +filter_test: routing filter index expansion test started +filter_test: mixed routing filter fingerprint sizes test started +filter_test: routing filter arithmetic limits test started +filter_test: malicious routing/mini metadata test started +Malformed mini allocator metadata: unsupported format version (addr=4980736). +Malformed mini allocator metadata: metadata page number disagrees with predecessor link (addr=4980736). +Malformed mini allocator metadata: metadata page number is not contiguous (addr=4984832). +Malformed mini allocator metadata: metadata page number is not contiguous (addr=4980736). +Malformed mini allocator metadata: metadata entry position is invalid (addr=4980736). +Malformed mini allocator metadata: metadata entry position is invalid (addr=4980736). +Malformed mini allocator metadata: metadata extent entry is invalid (addr=4980736). +filter_test: routing filter basic test started +num input keys 1048575 estimate 1048653 num_unique 1040503 +num input keys 1572862 estimate 1573255 num_unique 1554958 +num input keys 2097150 estimate 2097684 num_unique 2065239 +num input keys 2534056 estimate 2534923 num_unique 2487644 +num input keys 3075820 estimate 3077087 num_unique 3007608 +num input keys 3460297 estimate 3461800 num_unique 3374028 +num input keys 4014546 estimate 4016523 num_unique 3898690 +num input keys 4392783 estimate 4395331 num_unique 4254486 +across filters: num input keys 4392783 estimate 4408343 +routing filter basic test: false positive rate 0.0625 +filter_test: routing filter basic test started +num input keys 100 estimate 100 num_unique 100 +num input keys 150 estimate 150 num_unique 150 +num input keys 199 estimate 199 num_unique 199 +num input keys 241 estimate 241 num_unique 241 +num input keys 292 estimate 292 num_unique 292 +num input keys 328 estimate 328 num_unique 328 +num input keys 379 estimate 379 num_unique 379 +num input keys 415 estimate 415 num_unique 415 +across filters: num input keys 415 estimate 0 +routing filter basic test: false positive rate 0.0000 +filter_test: routing filter basic test started +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +across filters: num input keys 1 estimate 0 +routing filter basic test: false positive rate 0.0000 +filter_test: routing filter basic test started +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +num input keys 1 estimate 0 num_unique 1 +across filters: num input keys 1 estimate 0 +routing filter basic test: false positive rate 0.0000 diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-log.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-log.log new file mode 100644 index 000000000..fdad25457 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-log.log @@ -0,0 +1,11 @@ +build-main-7bb5fab/debug/bin/driver_test: splinterdb_build_version fb878f13-dirty +Dispatch test log_test + +Started log_test!! +Malformed log page sequence at page 1843200: group=3 page=0 follows group=1 page=0 (terminated). +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +shard_log_iterator_init: record validation failed in group 1 at log page 139591680: Invalid argument; discarding this group and the rest +Malformed log page sequence at page 140378112: group=1 page=0 follows group=1 page=0. +shard_log_iterator_init: record validation failed in group 2 at log page 141168640: Input/output error; discarding this group and the rest +log returned 500000 of 500000 entries diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-quick.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-quick.log new file mode 100644 index 000000000..793ea7c46 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-quick.log @@ -0,0 +1,55 @@ +Running 1 CTests, suite name 'splinterdb_quick', test case 'all'. +TEST 1/49 splinterdb_quick:test_get_config_returns_effective_config [OK] +TEST 2/49 splinterdb_quick:test_get_cost_stats [OK] +TEST 3/49 splinterdb_quick:test_checkpoint_log_grace_default [OK] +TEST 4/49 splinterdb_quick:test_durable_barrier [OK] +TEST 5/49 splinterdb_quick:test_logged_large_value_replaces_old_value [OK] +TEST 6/49 splinterdb_quick:test_basic_flow [OK] +TEST 7/49 splinterdb_quick:test_pthread_auto_registration [OK] +TEST 8/49 splinterdb_quick:test_result_auto_registration_with_stats [OK] +TEST 9/49 splinterdb_quick:test_stats_reset_clears_cache_io [OK] +TEST 10/49 splinterdb_quick:test_final_merge_error_propagation [OK] +TEST 11/49 splinterdb_quick:test_fetch_and_update_catches_up_after_rotation [OK] +TEST 12/49 splinterdb_quick:test_fetch_and_update_rejects_incorporated_boundary [OK] +TEST 13/49 splinterdb_quick:test_range_init_error_cleanup_preserves_branches setup_ordered_iterators: exception: Cannot allocate memory +core_apply_to_range: range iterator init failed: Cannot allocate memory +[OK] +TEST 14/49 splinterdb_quick:stats_reset_excludes_active_operations [OK] +TEST 15/49 splinterdb_quick:test_apis_for_large_key [OK] +TEST 16/49 splinterdb_quick:test_key_size_gt_max_inline_key_size [OK] +TEST 17/49 splinterdb_quick:test_value_size_gt_max_value_size [OK] +TEST 18/49 splinterdb_quick:test_large_lookup_survives_memtable_recycle [OK] +TEST 19/49 splinterdb_quick:test_iterator_survives_memtable_recycle [OK] +TEST 20/49 splinterdb_quick:test_variable_length_values [OK] +TEST 21/49 splinterdb_quick:test_basic_iterator [OK] +TEST 22/49 splinterdb_quick:test_iterator_delete_current [OK] +TEST 23/49 splinterdb_quick:test_empty_iterator [OK] +TEST 24/49 splinterdb_quick:test_bounded_iterator [OK] +TEST 25/49 splinterdb_quick:test_bounded_iterator_start_normalization [OK] +TEST 26/49 splinterdb_quick:test_splinterdb_iterator_with_startkey [OK] +TEST 27/49 splinterdb_quick:test_splinterdb_iterator_with_non_existent_startkey [OK] +TEST 28/49 splinterdb_quick:test_splinterdb_iterator_with_missing_startkey_in_sequence [OK] +TEST 29/49 splinterdb_quick:test_iterator_prev_and_next [OK] +TEST 30/49 splinterdb_quick:test_iterator_reversal_on_key_in_two_branches [OK] +TEST 31/49 splinterdb_quick:test_iterator_reversal_with_three_branches [OK] +TEST 32/49 splinterdb_quick:test_rewriting_keys_does_not_grow_trunk [OK] +TEST 33/49 splinterdb_quick:test_close_and_reopen [OK] +TEST 34/49 splinterdb_quick:test_logged_close_and_reopen [OK] +TEST 35/49 splinterdb_quick:test_repeated_insert_close_reopen [OK] +TEST 36/49 splinterdb_quick:test_custom_data_config [OK] +TEST 37/49 splinterdb_quick:test_existence_only_memtable_lookup [OK] +TEST 38/49 splinterdb_quick:test_existence_only_trunk_lookup_skips_branches [OK] +TEST 39/49 splinterdb_quick:test_write_api_returns_old_value [OK] +TEST 40/49 splinterdb_quick:test_write_api_old_result_existence_only [OK] +TEST 41/49 splinterdb_quick:test_write_api_old_result_custom_merge_semantics [OK] +TEST 42/49 splinterdb_quick:test_write_api_old_result_merges_memtable_and_trunk [OK] +TEST 43/49 splinterdb_quick:test_write_api_old_result_respects_trunk_delete_shadow [OK] +TEST 44/49 splinterdb_quick:test_iterator_custom_comparator [OK] +TEST 45/49 splinterdb_quick:test_iterator_init_bug [OK] +TEST 46/49 splinterdb_quick:test_splinterdb_create_w_background_threads [OK] +TEST 47/49 splinterdb_quick:test_splinterdb_create_w_all_background_threads [OK] +TEST 48/49 splinterdb_quick:test_iterator_thread_handoff [OK] +TEST 49/49 splinterdb_quick:test_blob_tree_crash_recovery Running 1 CTests, suite name 'splinterdb_quick', test case 'test_blob_tree_crash_recovery'. +TEST 1/1 splinterdb_quick:test_blob_tree_crash_recovery Running 1 CTests, suite name 'splinterdb_quick', test case 'test_blob_tree_crash_recovery'. +TEST 1/1 splinterdb_quick:test_blob_tree_crash_recovery [OK] +RESULTS: 49 tests (49 ok, 0 failed, 0 skipped) ran in 10478 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter-4gib.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter-4gib.log new file mode 100644 index 000000000..4afe8318f --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter-4gib.log @@ -0,0 +1,89 @@ +Running 1 CTests, suite name 'splinter', test case 'all'. +TEST 1/38 splinter:test_blob_checksums [OK] +TEST 2/38 splinter:test_inserts [OK] +TEST 3/38 splinter:test_btree_pack_abort_releases_reserved_extents btree_pack error status: 5 +[OK] +TEST 4/38 splinter:test_btree_pack_rejects_counter_overflow btree_pack: output statistics exceed the on-disk limit +btree_pack error status: 28 +[OK] +TEST 5/38 splinter:test_malformed_trunk_pivot_bundle [OK] +TEST 6/38 splinter:test_invalid_branch_reference_rejected Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +[OK] +TEST 7/38 splinter:test_malformed_trunk_pivot_stats trunk_merge_lookup: ondisk_node_find_pivot failed: 22 +trunk_merge_lookup_async: ondisk_node_find_pivot_async failed: 22 +[OK] +TEST 8/38 splinter:test_malformed_trunk_child_height trunk_merge_lookup_async: ondisk_node_handle_init_async failed: 22 +[OK] +TEST 9/38 splinter:test_two_log_checkpoint [OK] +TEST 10/38 splinter:test_reopen_with_changed_filter_config [OK] +TEST 11/38 splinter:test_logged_large_insert_shares_blob_extents [OK] +TEST 12/38 splinter:test_durable_barrier_waits_for_visible_insert_log_write [OK] +TEST 13/38 splinter:test_durable_barrier_packs_concurrent_small_tails [OK] +TEST 14/38 splinter:test_shard_log_first_fit_decreasing_packing [OK] +TEST 15/38 splinter:test_shard_log_page_alloc_failure_retries_open_buffer shard_log_graduate_buffer: out of log space +shard_log_graduate_group: failed to terminate group 1: No space left on device +[OK] +TEST 16/38 splinter:test_durable_barrier_without_log_publishes_root [OK] +TEST 17/38 splinter:test_checkpoint_reports_terminal_incorporation_failure trunk_node_serialize():2802: cache_alloc() failedserialize_nodes():2986: node_serialize() failedserialize_nodes_and_save_contingent_compactions: serialize_nodes failed: 12 +restore_balance_leaf():6155: serialize_nodes() failed: Cannot allocate memorytrunk_flush_prepare: flush_then_compact failed: 12 +trunk_incorporate_prepare failed: Cannot allocate memory +Memtable incorporation failed: generation=0 state=INCORPORATION_ASSIGNED status=Cannot allocate memory memtable_root=262144 +memtable flush failed: Cannot allocate memory +Shutdown found memtable from failed incorporation: generation=0 state=INCORPORATION_FAILED status=Cannot allocate memory memtable_root=262144 compacted_root=0 +[OK] +TEST 18/38 splinter:test_durable_barrier_reopens_writers_before_device_barrier [OK] +TEST 19/38 splinter:test_concurrent_durable_barriers_coalesce [OK] +TEST 20/38 splinter:test_durable_barrier_waits_for_checkpoint_publication [OK] +TEST 21/38 splinter:test_log_prepare_failure_leaves_update_unapplied core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_unmount: failed to publish the unmount root: Input/output error +core_unmount: shutdown retained recovery state; the next mount must recover +[OK] +TEST 22/38 splinter:test_checkpoint_advance_retries_cut_publish_failure core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +[OK] +TEST 23/38 splinter:test_checkpoint_advance_retries_completion_failure core_checkpoint_advance: completion publish failed: Input/output error +core_checkpoint_advance: completion publish failed: Input/output error +[OK] +TEST 24/38 splinter:test_self_managed_checkpoints_reclaim_log_space [OK] +TEST 25/38 splinter:test_auto_checkpoint [OK] +TEST 26/38 splinter:test_auto_checkpoint_grace_allows_natural_rotation [OK] +TEST 27/38 splinter:test_auto_checkpoint_grace_forces_overwrite_rotation [OK] +TEST 28/38 splinter:test_auto_checkpoint_on_overwrites [OK] +TEST 29/38 splinter:test_recover_allocations_reproduces_persisted_map [OK] +TEST 30/38 splinter:test_unmount_skips_allocator_map_that_needs_rebuild core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 31/38 splinter:test_late_reclamation_failure_requires_recovery WARNING: abandoned reclamation of mini allocator at 6553600 (type misc): Connection timed out. Remaining space is unavailable until reopen; clean allocation state will not be written. Recovery is required. +WARNING: core_unmount: reclamation failed (Connection timed out); remaining space was retained. Shutdown requires recovery and the next open will rebuild allocation state. +core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 32/38 splinter:test_mount_active_checkpoint_stays_mountable [OK] +TEST 33/38 splinter:test_lookups [OK] +TEST 34/38 splinter:test_splinter_print_diags [OK] +TEST 35/38 splinter:test_cache_read_errors_reach_core_operations ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 110 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 110 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 110 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 110 +[OK] +TEST 36/38 splinter:test_blob_descriptor_sizes [OK] +TEST 37/38 splinter:test_blob_trees [OK] +TEST 38/38 splinter:test_blob_height_three_and_large_stats [FAIL] + ERR: tests/unit/splinter_test.c:5941 should be true +RESULTS: 38 tests (37 ok, 1 failed, 0 skipped) ran in 85363 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter.log new file mode 100644 index 000000000..0e78e5011 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-splinter.log @@ -0,0 +1,88 @@ +Running 1 CTests, suite name 'splinter', test case 'all'. +TEST 1/38 splinter:test_blob_checksums [OK] +TEST 2/38 splinter:test_inserts [OK] +TEST 3/38 splinter:test_btree_pack_abort_releases_reserved_extents btree_pack error status: 5 +[OK] +TEST 4/38 splinter:test_btree_pack_rejects_counter_overflow btree_pack: output statistics exceed the on-disk limit +btree_pack error status: 28 +[OK] +TEST 5/38 splinter:test_malformed_trunk_pivot_bundle [OK] +TEST 6/38 splinter:test_invalid_branch_reference_rejected Unsupported branch format version 2 at 8519680; this build requires version 1. +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +Unsupported branch format version 2 at 8519680; this build requires version 1. +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +Unsupported branch format version 2 at 8519680; this build requires version 1. +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +[OK] +TEST 7/38 splinter:test_malformed_trunk_pivot_stats trunk_merge_lookup: ondisk_node_find_pivot failed: 22 +trunk_merge_lookup_async: ondisk_node_find_pivot_async failed: 22 +[OK] +TEST 8/38 splinter:test_malformed_trunk_child_height trunk_merge_lookup_async: ondisk_node_handle_init_async failed: 22 +[OK] +TEST 9/38 splinter:test_two_log_checkpoint [OK] +TEST 10/38 splinter:test_reopen_with_changed_filter_config [OK] +TEST 11/38 splinter:test_logged_large_insert_shares_blob_extents [OK] +TEST 12/38 splinter:test_durable_barrier_waits_for_visible_insert_log_write [OK] +TEST 13/38 splinter:test_durable_barrier_packs_concurrent_small_tails [OK] +TEST 14/38 splinter:test_shard_log_first_fit_decreasing_packing [OK] +TEST 15/38 splinter:test_shard_log_page_alloc_failure_retries_open_buffer shard_log_graduate_buffer: out of log space +shard_log_graduate_group: failed to terminate group 1: No space left on device +[OK] +TEST 16/38 splinter:test_durable_barrier_without_log_publishes_root [OK] +TEST 17/38 splinter:test_checkpoint_reports_terminal_incorporation_failure trunk_node_serialize():2802: cache_alloc() failedserialize_nodes():2986: node_serialize() failedserialize_nodes_and_save_contingent_compactions: serialize_nodes failed: 12 +restore_balance_leaf():6155: serialize_nodes() failed: Cannot allocate memorytrunk_flush_prepare: flush_then_compact failed: 12 +trunk_incorporate_prepare failed: Cannot allocate memory +Memtable incorporation failed: generation=0 state=INCORPORATION_ASSIGNED status=Cannot allocate memory memtable_root=2228224 +memtable flush failed: Cannot allocate memory +Shutdown found memtable from failed incorporation: generation=0 state=INCORPORATION_FAILED status=Cannot allocate memory memtable_root=2228224 compacted_root=0 +[OK] +TEST 18/38 splinter:test_durable_barrier_reopens_writers_before_device_barrier [OK] +TEST 19/38 splinter:test_concurrent_durable_barriers_coalesce [OK] +TEST 20/38 splinter:test_durable_barrier_waits_for_checkpoint_publication [OK] +TEST 21/38 splinter:test_log_prepare_failure_leaves_update_unapplied core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_unmount: failed to publish the unmount root: Input/output error +core_unmount: shutdown retained recovery state; the next mount must recover +[OK] +TEST 22/38 splinter:test_checkpoint_advance_retries_cut_publish_failure core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +[OK] +TEST 23/38 splinter:test_checkpoint_advance_retries_completion_failure core_checkpoint_advance: completion publish failed: Input/output error +core_checkpoint_advance: completion publish failed: Input/output error +[OK] +TEST 24/38 splinter:test_self_managed_checkpoints_reclaim_log_space [OK] +TEST 25/38 splinter:test_auto_checkpoint [OK] +TEST 26/38 splinter:test_auto_checkpoint_grace_allows_natural_rotation [OK] +TEST 27/38 splinter:test_auto_checkpoint_grace_forces_overwrite_rotation [OK] +TEST 28/38 splinter:test_auto_checkpoint_on_overwrites [OK] +TEST 29/38 splinter:test_recover_allocations_reproduces_persisted_map [OK] +TEST 30/38 splinter:test_unmount_skips_allocator_map_that_needs_rebuild core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 31/38 splinter:test_late_reclamation_failure_requires_recovery WARNING: abandoned reclamation of mini allocator at 8519680 (type misc): Connection timed out. Remaining space is unavailable until reopen; clean allocation state will not be written. Recovery is required. +WARNING: core_unmount: reclamation failed (Connection timed out); remaining space was retained. Shutdown requires recovery and the next open will rebuild allocation state. +core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 32/38 splinter:test_mount_active_checkpoint_stays_mountable [OK] +TEST 33/38 splinter:test_lookups [OK] +TEST 34/38 splinter:test_splinter_print_diags [OK] +TEST 35/38 splinter:test_cache_read_errors_reach_core_operations ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 110 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 110 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 110 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 110 +[OK] +TEST 36/38 splinter:test_blob_descriptor_sizes [OK] +TEST 37/38 splinter:test_blob_trees [OK] +TEST 38/38 splinter:test_blob_height_three_and_large_stats [OK] +RESULTS: 38 tests (38 ok, 0 failed, 0 skipped) ran in 89322 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-superblock.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-superblock.log new file mode 100644 index 000000000..01b516af9 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-superblock.log @@ -0,0 +1,43 @@ +Running 1 CTests, suite name 'superblock', test case 'all'. +TEST 1/16 superblock:test_format_sets_fresh_state [OK] +TEST 2/16 superblock:test_snapshot_persists_state [OK] +TEST 3/16 superblock:test_two_log_checkpoint_transitions [OK] +TEST 4/16 superblock:test_snapshot_preserves_unincorporated_sealed_log [OK] +TEST 5/16 superblock:test_two_log_checkpoint_torn_begin [OK] +TEST 6/16 superblock:test_torn_write_falls_back_to_older_generation [OK] +TEST 7/16 superblock:test_generation_wrap_selects_newest_slot [OK] +TEST 8/16 superblock:test_both_slots_corrupt_is_not_found superblock_mount: no valid superblock found +[OK] +TEST 9/16 superblock:test_mount_rejects_invalid_metadata superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +[OK] +TEST 10/16 superblock:test_slot_read_error_falls_back_to_peer [OK] +TEST 11/16 superblock:test_both_slot_read_errors_are_reported superblock_mount: no readable valid superblock found: Input/output error +[OK] +TEST 12/16 superblock:test_read_geometry superblock_read_bootstrap: no valid superblock found +[OK] +TEST 13/16 superblock:test_read_bootstrap_allocation_state superblock_read_bootstrap: no valid superblock found +[OK] +TEST 14/16 superblock:test_bootstrap_geometry_metadata_budget Device geometry needs 134217728 bytes of cache lookup metadata, which exceeds the configured cache budget of 67108864 bytes. +Failed to open existing SplinterDB device 'db' with specified configuration: No space left on device +[OK] +TEST 15/16 superblock:test_unknown_structure_version_rejected Unsupported trunk format version 2; this build requires version 1. +Unsupported trunk format version 2; this build requires version 1. +Unsupported allocator format version 2; this build requires version 1. +Unsupported allocator format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +[OK] +TEST 16/16 superblock:test_unsupported_superblock_version_rejected Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +[OK] +RESULTS: 16 tests (16 ok, 0 failed, 0 skipped) ran in 28 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/debug-trunk-async.log b/benchmarks/results-20261013/main-7bb5fab/tests/debug-trunk-async.log new file mode 100644 index 000000000..03ab76cd8 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/debug-trunk-async.log @@ -0,0 +1,5 @@ +Running 1 CTests, suite name 'trunk_async', test case 'all'. +TEST 1/2 trunk_async:pivot_header_survives_cross_thread_yield [OK] +TEST 2/2 trunk_async:pivot_read_error_survives_cross_thread_yield trunk_ondisk_node_read_pivot_key_async():1462: ondisk_node_handle_setup_content_page() failed: Connection timed outtrunk_merge_lookup_async: ondisk_node_find_pivot_async failed: 110 +[OK] +RESULTS: 2 tests (2 ok, 0 failed, 0 skipped) ran in 0 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/manifest.json b/benchmarks/results-20261013/main-7bb5fab/tests/manifest.json new file mode 100644 index 000000000..df589e7b4 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/manifest.json @@ -0,0 +1,273 @@ +{ + "base": "7bb5fab305b9c6179e31f75998dabea833537a08", + "previous_head": "fb878f1", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "sources": { + "include/splinterdb/data.h": "57be61933a057f52f679dfcf21755a8645e6228cd1a7cdee983f5d57768923ab", + "include/splinterdb/default_data_config.h": "179386f3e224d322a9d7ac97afd929292994a1d890a3254705f9ebea525bce32", + "include/splinterdb/platform_linux/public_platform.h": "c5ded82c5b5bb5bbcedf06a66951eb98dc8892f7b3c4f8ce07c474db98ac68a0", + "include/splinterdb/public_platform.h": "4ce9da3fea9a54fb7bb613ec1458f11f043525efb7d0bc55182f7183dd4a4f6e", + "include/splinterdb/public_util.h": "4e8f6b30408fb4174894d4ba809acff4e96f559370c7749b65bb3724a853603f", + "include/splinterdb/splinterdb.h": "364073691c390663f6b732f57c3ee4bfecc9f68aca33ef6750d6106e04462dc8", + "src/PackedArray.c": "97b64eb0a4f550e55f3385d39d729a878fbc581909ea16ec3b8987a7f46604cc", + "src/PackedArray.h": "e46f854b487a8b130f0613efcab11836012d27c8aacf9ca9e76a2de1a7eb6874", + "src/allocator.c": "96c7f0a5e0808719cf114d267c84da1f057b5eed748b0672d60686d7f130b96f", + "src/allocator.h": "06da1d6a491a15ef5229e8e2e4b4ad6468cb3c9ba620e0d956032dc0a1b6aec1", + "src/batch_rwlock.c": "c043d9e5ead75cd9e1266a6f51fb555012269f3822d7f0233dc2abbd962c3aae", + "src/batch_rwlock.h": "0ff5f0bd7fdf5f9a63ca2f77c387d574d8009ad726b47283c93566cf06bc0927", + "src/blob.c": "75e01e92c06d775b1cb36ea63a7cddf191b5179b94c3e9cd6b5b23ab0480485b", + "src/blob.h": "8f68d2f67de8a9d81a422b28d47aa636cc7234fccc3dbabe59f2355bc361e93c", + "src/blob_build.c": "e484e9c14fd8a9b3c1d211485c35931c944fc66ac4ee78ea20ffc5b3f074b10b", + "src/blob_build.h": "38176881cf9159247e6f578bbc7abb725cfa969c493715df7d1388cb7cbfd119", + "src/btree.c": "8b059dfa240564a7fca21379b467afc7635b9128d2ca72555300333ae06dbc97", + "src/btree.h": "2dc405429d4d84f606412d3a16dc07dd2aee2877524d658e946cb1a8bf1d865e", + "src/btree_private.h": "43693dd3048ae050b3358ccfc9b91371a3d9dff4f3d6d5c8b07445897dee09f6", + "src/cache.h": "d4c1e133d6423fff324d0ebcf5c95e13647c0022a3458935156746b90d816d6d", + "src/clockcache.c": "e0ef2e19a8280849df4ee80d9b8a54a0faaf56a13faefe7a91d54f5b7269f933", + "src/clockcache.h": "68f4e7b019d0440f85929bfc0da1891ce941318ee7dfd14a3d4d923e3d4e72d8", + "src/core.c": "b8d5d7e2432841485468bf958c23f27e112e0370e6929715e44fb76b1b06f7a4", + "src/core.h": "7cf4d086f6d4dcaa4cbd161fef786dbe3cc502ced4512b7fe547c2aa2020462e", + "src/data_blob_build.c": "13475faa56bba94ee26ef7fec4e4da8a5045e0efe510a63291fd5e2697cb5d22", + "src/data_blob_build.h": "dd7f263a0b1138325b211c561ba7289183a2cc8472978cba3ca430295e11e5f3", + "src/data_internal.c": "0a42dd477aaa6ad6c4910841045c4c768ad8d248c0126c64c51035f53e769cde", + "src/data_internal.h": "98b4ebe77576c4cbb7dd9984ca5e297851175494062073f1a4c137c5afb8b587", + "src/default_data_config.c": "161352f5247978f3d57033818d05a79d0a1a0d0ffac2bc9f25f4f33eabca2227", + "src/histogram.c": "24e09dfabd6085021d8a06ef9b10f6e85b24914d95831be359df09f8a9729aff", + "src/histogram.h": "7ccfde6d3ada7ead18fe477c06a6b021db83ca8c041ab0998ca023b945a3c1c7", + "src/iterator.h": "2fcddeb9fa6bd49f4e3da933224219bdb830f9e2f11638ea1ed13eba534fa27f", + "src/log.h": "49a2b0aec91a1c40784b8b3b54f016ef5504c79492b1393674123e27656a09bb", + "src/log_data.h": "8b9198c49c95367f5c66cfa23ec1a3b6cab2abd875fe74c4afb4dafb5a5c44fe", + "src/lookup_result.h": "7528b988a28ef91b6bd7a4fa10dfa51b9ce2475e801b3b6a8bf30adf88f1ef16", + "src/memtable.c": "7db3b4754497c73b4071c7e7e591cc76fc226197fb70d98fb46caf139b3953f4", + "src/memtable.h": "244cac7cf304df85a25c8bede2df106d6f840d65bbd1a3551b3d9f2ed9129fc1", + "src/merge.c": "df98421e597d1edeac871755f0473055b590791af8efad495575d20260e46a12", + "src/merge.h": "c17e2a0b6272ec52e6d1e80acd9df7dc85221f186e6c60e323c37b467dc5c41a", + "src/mini_allocator.c": "d620cbdff9867ce64c752992ede58783010fb09366220ad3bd6d8518911205ca", + "src/mini_allocator.h": "39d245b58914d28b26dfe230b9de15582382cec1c2a90fa14db19f5f2c203632", + "src/notification.c": "2f3bcda49d8ce35ad532fef8ec8f4aeca48bec280cdf6210c21d55cf2aa4a661", + "src/notification.h": "47eb73f5813d6f54658f339ce44d6a475f8f64f2b2eb6368261e839a090f083f", + "src/ondisk_ref.c": "2ace8cbbce3f80a54ff7762ad03a79a564665fa660394f1d3446235a3e40ed7b", + "src/ondisk_ref.h": "643325c8b4f16fab41d8c884c97bdfcf6c86e5178398508fa0f81b6b0606926d", + "src/pcq.h": "862cd346b0097014ccae797dc1463e7b3fc387f409319d5adcef7613b6f82f84", + "src/platform_linux/async.h": "41383b57e21f0f51d737749a4ce7c2c5330ac9363e742330625c9aedd4f13a52", + "src/platform_linux/laio.c": "0ce628806e6896070359a9e39d069f326f460ab697c6c17ff17d171614290881", + "src/platform_linux/laio.h": "f3987e22f1fd7a9d6b46c79f84e75559aec5410239a13d52a7a5343ba792b28e", + "src/platform_linux/platform.h": "f1c3d7c5bcd089ed7a7df129d3b8d00d7617de9a7aa6f9d1f7deeb739281e768", + "src/platform_linux/platform_assert.c": "112e803b6db03dd8c14a3bfc3f1139bbcc4b81f1d5e24a232b7b99ce89146c2d", + "src/platform_linux/platform_assert.h": "751dccceac9ce4a97d25d441b5216ced5ee5f079a13c9ccd43551b0994aa7cc2", + "src/platform_linux/platform_backtrace.h": "ef13ccbf4ae4747b0de244a37c026ad8769018d84f9fc4d6493db43d9714358c", + "src/platform_linux/platform_buffer.c": "b790d645b3dc5f638da871bf9611865dc700927f3b594a1d645b0350bd1bd027", + "src/platform_linux/platform_buffer.h": "72e016dbb19b8250378b3a5c79fbebf9c3d72a31d07a232f36431e04f0e4afa6", + "src/platform_linux/platform_condvar.c": "e595f69194accb3f3460d6fc9a4a9144bd7f495dbeb77f11bf3616dc0e9b03ec", + "src/platform_linux/platform_condvar.h": "a4ee02ae5069637bce4cae07f3ed9869925f17a9e6e1b04d69baedfc148fd493", + "src/platform_linux/platform_hash.h": "f69f5ed6218bfe62c62718b387e1c110e9fca46c07603d4d51ccc4fc782b09ab", + "src/platform_linux/platform_heap.c": "72bb2753f525e56f8c398988a787feb38f02efaa749e5d57cb476f967269c479", + "src/platform_linux/platform_heap.h": "921f3113d20b44c36817c284702e965e01769596ff012edd6ceaa70cbdbbd10a", + "src/platform_linux/platform_io.c": "875a7331f8bfa46e9a6961f50acb40b107d40883839c94def52a289a521c66be", + "src/platform_linux/platform_io.h": "69ae9556daf1c7a69e007abd494bf9c86ec81c08bca1e032663009cef95b2fac", + "src/platform_linux/platform_log.c": "318bd17c858475a3a91c5e9641b52808f7c4194e1c6fc9a7d801392b04c49538", + "src/platform_linux/platform_log.h": "569484386150fcfbf5dde722c5b8e34d67d65be3454e3e009cb9df41f11efea6", + "src/platform_linux/platform_machine.h": "c02745313c7268dbe99efa2b93dd4d37640b5a7aa2e88c78fd5b73742f252c7d", + "src/platform_linux/platform_mutex.c": "7ec1dd6a5397217d28fd85c10648d29a70bf66649f59f76d7387236031b15bf3", + "src/platform_linux/platform_mutex.h": "688991d6a7dc57a7a88f565541561344cdfbb1733ca6073f1655d6c436ed6544", + "src/platform_linux/platform_random.h": "184ac9da9c38a6c10275891be61ccb91beaf2234ebb9ce046d059cb16824d68f", + "src/platform_linux/platform_semaphore.h": "9fdb7cfeaf36c8bf9aa346a16466ed6bc50bea76fdb5722c86001563d55bedfe", + "src/platform_linux/platform_sleep.h": "34aeaea35133bf33e47eb1d9ace429297644e13bc100d90ebbd205bceb58d3ab", + "src/platform_linux/platform_sort.h": "d374788510f94a8c3b86ff6bfa1f6a3ff8ac33168eb26f6f7774984252f6d01e", + "src/platform_linux/platform_spinlock.h": "7b3f0409109f3341f2287fe7cd8fb69faac4a74b009a44c1eb83220f71e94de0", + "src/platform_linux/platform_status.h": "2749285ea1322e5c1639019b728fde4b87b0cae6f71a7fc16e420bd592d38c00", + "src/platform_linux/platform_string.h": "fa41faf9c6661c62ece783740250232709656b0a8677f542e390a5f3b0d834a6", + "src/platform_linux/platform_threads.c": "789465a2c6ac7ac501e5878f429cc0dd9b163145a522e247a4833e3de7af8f2a", + "src/platform_linux/platform_threads.h": "653e070115131756e6aaffc3c2764a6437fb39ff96df4afc8a01e6e751c9d622", + "src/platform_linux/platform_time.h": "f212a08cf8432693711cecfd389f9ed3d0a70e0c056dc062321bd11c4a8cbec7", + "src/platform_linux/platform_trace.c": "4a68e89f214920dd2762d072a0ad8e7c80d9b1733a64e4a7e3dd4bb04b91c352", + "src/platform_linux/platform_trace.h": "345f7283b01e5a9b4d5da7bea66ad44ff61f53155417c97990e23bb8fedadf33", + "src/platform_linux/platform_typed_alloc.h": "42c1d2bbd56c183d11768a5f43d440cf3c1bfe329c4352afa6acd3bc2b191a8e", + "src/platform_linux/platform_units.h": "1cb22a07b33ec8b58f9381438f480f561d12b838f315a117cc1f979e95b1ef65", + "src/platform_linux/platform_util.h": "4fa2702b3919eb1a7efcf29457f5ee9a3d7a7b4c38fed4606264562022fa2638", + "src/platform_linux/poison.h": "a32dfa445439eb08913e519117e7b11f72b2c3550577a74769ed935a792c42c5", + "src/platform_linux/shmalloc.c": "e4631d63a9db1ad58472ad3d034d081f2bb9a97f34c5898f7c848dda923727f8", + "src/platform_linux/shmalloc.h": "081a82473f6021f8ed628e79f8ae5dc7104988981f8c3b25749ea42c8158113d", + "src/prefetch.h": "9d55d9dad64e3795c81856bc38688936e4e74e488dfe2d29a1f5438ad000df42", + "src/rc_allocator.c": "23ef0da6fb2d5114fe403403c8e535bf8ba19ed2c8bef4397212933cedfc8b9e", + "src/rc_allocator.h": "5f1fae7e32207a4e0cd5c4f799c9b802ce116dd381ee5c074c21d64a229dc3ad", + "src/routing_filter.c": "a913ee83ca39267b6da882bae0db05b1453f39c863f31ec6fea9844998f72f03", + "src/routing_filter.h": "3104de8135817e33b585ba489bc4821a458f6a9c2ce9e561ea8be2fde28e43ba", + "src/shard_log.c": "551efce92b8e60bbc29d0a7724c0382f3d4e41c84e6e6915c2252e52410fae86", + "src/shard_log.h": "bde00a5cbe72e72ac7133006796e61f046ae758e8e4667aa610e184ed6ef4fd7", + "src/splinterdb.c": "1de96ba09714d40ee673ebdf05654ba85a447548667681476c7bd311025b67ca", + "src/splinterdb_tests_private.h": "579031767b8869f49cf1e3fc94caeec13a1362dfd047782538df40b281c93414", + "src/superblock.c": "e3fbe387ac4fef197c11977e35e3aecf0d31c27c9d4932e4ec54b1bfc391620a", + "src/superblock.h": "cec63cb28426e2d23140b3ac310a86ae9635d6b3ec429abe7c469bc0de06aa73", + "src/task.c": "35886e598b4464af16e76497917955d3d55c6c077fbcf80fb3fd2284c8ddc08d", + "src/task.h": "35405e04fbdcef9bc5c5d6ef43debc4b0889d6bc4d8d530649b0c45a34012825", + "src/trunk.c": "cff7a5eef6945c36b7784c888bdf6ab2510d132d87d963ba2c909322c13faf36", + "src/trunk.h": "5d8674472258da1be0a84e20a20869c84dc846254d7604e5799297491e6f91e1", + "src/util.c": "969df408c4c393f73feb05dfa37ebe1ed8be627cef630061830e4992c4c1fd9c", + "src/util.h": "9fa455f5a6268fa4a6864d444849b0872391af00512932c1587363cfd38304ef", + "src/vector.h": "6855dd0430f1a868d098797cbd4f3453ab542b5f6fbe716c79871124bee200a9", + "src/writeback_set.c": "cb485716f57a3183ce82543848edaf61f6568c9ff0445ea5ff989f95c00848ff", + "src/writeback_set.h": "68d6f589d24719a09e2680b05a1eb7e36c22b1fab978768f5b54f958f667aec9", + "tests/config.c": "e44ffe45687560ef6605bd086709251d53cccfbadbc94fd3bdfd0106721f3698", + "tests/config.h": "c1f80b62a51e77255d8d91ace4b2e6777c7311010167149430edd41a6d5ce142", + "tests/functional/README.md": "c95907cfe5f4aaebc303d0d94cdcbd51e78d717ec11e4c5ecea6b8efaf723be0", + "tests/functional/avlTree.c": "5c24865aa50a21c626dacb2bd6172ff182a25e3532f5aee73a25aeb47c6bb043", + "tests/functional/avlTree.h": "39d82dc2b4c99a635c095800efcd060e7c95822a1194de0f03524a970040ac86", + "tests/functional/btree_test.c": "c011e72c204cc7640d1d259c791ba7ead40ef2a2c01fcfff3140d988d6666547", + "tests/functional/cache_test.c": "549f326d035fbb81cf9f272a66948f34f30313495624c6320e4a75b47c042e48", + "tests/functional/driver_test.c": "aef8d611eae488b287a796007c2a4e2a5f2e7f3f3190dce51be0e2be78ecb765", + "tests/functional/filter_test.c": "113f4420a3ff87989bfcfa7cb1f88e2026805be1befbf04bbd084e7c3b63ae93", + "tests/functional/io_apis_test.c": "be9b36b3af71975782f42ecc8087a922dea828d82049844d0833845cc1698ec5", + "tests/functional/log_test.c": "fc67a09c147248b4a480bdf1a39a0810dd94ad3e8443b0da468e276defa3d5ec", + "tests/functional/random.h": "8b6567f2e066099fc21d9f6548b561169ffadb0da07150be7bf940c86d146082", + "tests/functional/scan_benchmark.c": "555f1157c220c4f2924bd2089abdcda527a3acde440bc0c922c9c58c2ddedfd1", + "tests/functional/splinter_test.c": "b6c2e683dd48e23bc5d035e5d00142012f64085a21cff563b44d54fbaf2c7b9c", + "tests/functional/splinter_test.h": "9eb16fb6c779b27df477213c341031421d383c1c72051f2f42abec4c3c5b0f79", + "tests/functional/test.h": "e495226245728b72d4108f627642e98d7fba399f6fca3b733e18e4d2b218fb30", + "tests/functional/test_async.c": "258f7c56c6c500c783e659b69b157d3c9a06812fcbc31a8fb165b6c66f213acd", + "tests/functional/test_async.h": "c3530305b730c3d9a2c3c73ed438ee7ccd6c5c45cd26e0e02e6a821350b1d778", + "tests/functional/test_dispatcher.c": "60eaa54ce5ffd0d50095722ecd488b448e875fa07a44acc4ba560534c16d67b9", + "tests/functional/test_functionality.c": "d7b71e53f78b88257a10d8cc4fddd3daa48d704a89f6b4c5590af507da4637cd", + "tests/functional/test_functionality.h": "d065e9b63d00f37a88ee9cf962a1280040d7ee4684a1141a506e393fa38913b0", + "tests/functional/test_splinter_shadow.c": "4b856d30739b5ae544b65f514a55cc9d91cbc550044c498bedc51e8d5da26bd5", + "tests/functional/test_splinter_shadow.h": "a1b23c891db62785db9c4f80035ea1060de68c79aa0230dd6dbc03a8fe0b118d", + "tests/functional/ycsb_test.c": "5b1be412e93cfe56b8578051583fa231c138e2acee1afe4b3d1dfb925e0ad26f", + "tests/test_common.c": "fd1d18926096852a7dcfb54622d534d76a8ec62437ce72143c261f402587e493", + "tests/test_common.h": "3691f4e6bdaedb246dc9edea3d62a5cc69b456240492ba6cca8a2320c27cd595", + "tests/test_data.c": "48433bcf29a2e5ce1540a7b479bf0ddf85278e3d2a3c5ca6d8e4538cd1f0012f", + "tests/test_data.h": "61af4814f5d37200fa2393fc40abc7aa19acf4af322ca159390da3332af2952a", + "tests/unit/README.md": "9d4dc976f63738ebe5b73d446e2d6de088f29b26e2683f3c3c552c964949de2d", + "tests/unit/btree_stress_test.c": "1aa7a844349a1070f497f8ded0be9a36a5cf492d8731b49d626a771664f3ba79", + "tests/unit/btree_test.c": "526b3cedc42ed434440cfbe2af832688a9610fa559c625aee5034db0f5f040ba", + "tests/unit/btree_test_common.c": "3fa17ff41ec7afa5e7204aeb68767a7fcb7e1c7cf81695ef9506876a2af324bf", + "tests/unit/btree_test_common.h": "b7b30bdc02449e81d2fe263c0ef74a4a28c728f717d1ea20d16e3f1acb10a6a2", + "tests/unit/config_parse_test.c": "700a56f940237508e1ce647cc011f2f42c503aab262f930fd1baf0f265a9a112", + "tests/unit/ctest.h": "db9f53f9da859ff1d72be00efb993ca551720e0acaf747236e34e031368c7a16", + "tests/unit/large_inserts_stress_test.c": "df90d2de303f72829779f23cad3881cc65465d5d1a22fcdaf10000b23430b7ae", + "tests/unit/limitations_test.c": "e851344911b8c835578ccd9ae777b79369d339a375cca1468a12c70f03a9efe3", + "tests/unit/main.c": "a4a3ca6e237e3f40f07682c6528b1ef6f32105afc6071810e15411ad297b7a38", + "tests/unit/merge_test.c": "cf5c1c8010324e1e7629b482bb77796c837abcc5e88eb2de8f97aed11f4bb692", + "tests/unit/misc_test.c": "4ec5cf57c933275ccdf532a529ae79909840f7d41d36b842a4cb9d9a291ea05b", + "tests/unit/platform_apis_test.c": "c3623021ae04b4305b0971cfb2760f895c5b6908310a6e4c192f65769262c860", + "tests/unit/platform_threads_test.c": "bd8e8bb58926344add7b38357c15237687165da8995b052f02612967b6bd8af5", + "tests/unit/splinter_test.c": "e147508f4f8ba4543c4c37f4ad4c2d44fb41e85b7058508b6630cd0d8551f399", + "tests/unit/splinterdb_forked_child_test.c": "32bf91450ba812947ecdc7551ea2996823c05808159786ce746eba0265010a72", + "tests/unit/splinterdb_heap_id_mgmt_test.c": "43729def7a6d91cf66f5bc1fe2e6f0a4a45d63ebbb55887ed10a5eafcdc8235e", + "tests/unit/splinterdb_optimize_test.c": "b5e3295bcc6ee44437f329005f92d02590d90ca1d96a98439721fff32649cd5d", + "tests/unit/splinterdb_quick_test.c": "2a408abe02a4d9390e3ca01b14ef2f3a14c4ba94a235e3f3c1223a6a8b6db6f9", + "tests/unit/splinterdb_stress_test.c": "21a7f8725c82e4d8ec780131eff514294304aba322d8a2650ae893efaaf66f3a", + "tests/unit/superblock_test.c": "d448fbb284ee9e93061a66426f03f7440eced487fff611f11988fb4717276a70", + "tests/unit/task_system_test.c": "bd8ad9ca2632f17a6353f3c1e4d0849d98604e6b3a30f9c0c0b7ee13904a3024", + "tests/unit/trunk_async_test.c": "00c51d4302a48e09bdfacf217d45fc3c935f59bac130ded3f3164cc64ec5713b", + "tests/unit/trunk_gc_test.c": "00d6134ad82ed061cbc16d4b19fc21f41432c4b7c5a4b7ade80c3bed3b2dfdca", + "tests/unit/trunk_split_test.c": "4237641a463b28b7be333ed5d4f80a4830c3b0ef75335b196da34318a9f58b18", + "tests/unit/unit_test_template_dot_c": "ffb5e31229ca59caaece995122fcefe11f360ae45953caae0b5d4a72b2a6bcf7", + "tests/unit/unit_tests.h": "4e15f58455d8d759e9fad8432a02b1ec70f3a47a3e74eb39ea11b6a2ad44af42", + "tests/unit/unit_tests_common.c": "7a47eb40fadea397842e2f56932083d7e8840f08834788d6774b1f688288e735", + "tests/unit/util_test.c": "e992dff255d18e84673ea3954eb2da314de0815e64d55fd1980018b22444508b", + "tests/unit/vector_test.c": "463c0ae5b0c7525844f198c6859988a1bc33274d26cace87b9229673bc7c5186", + "tests/unit/writable_buffer_test.c": "3418d6ae29c5bc09dda485c1c1fe1768a1a3fc66b9b6bfdd05aa50e52516241e", + "tests/unit/writeback_set_test.c": "7bef6ea2a4a9d13fe0d83549cadd8e9497018068ac397ac8c31edea6909ee398" + }, + "build_artifacts": { + "build-main-7bb5fab/debug/lib/libsplinterdb.so": "2eec768ad032c3715b492b2055a2f2ec1b0d5255a509c0f000fcf8dfdaa8bb00", + "build-main-7bb5fab/debug/lib/libsplinterdb.a": "f09956f5cf8845ffc47d246fd1c40e07e05c7176c35cd16ed19b407531554a37", + "build-main-7bb5fab/debug/bin/unit_test": "b1d2466dbb2aac9b13d59bfb8a3e8a10e2ea95af91cb86fc20b9e5943734ce9c", + "build-main-7bb5fab/debug/bin/splinter_inspect": "0b501a7d1854397ab7b5722a90a620c298e2f8e1f4fef98f58f8172676b0a9fd", + "build-main-7bb5fab/debug/bin/driver_test": "2505d2bef881cd2a40e292a0f218e9218d804592f3b5e6ddf617e0bf85d68c9f", + "build-main-7bb5fab/debug/bin/unit/splinter_test": "025ef3f65884899b5335c2a19e91ac639629e42f4eb5473ec56007d067b93767", + "build-main-7bb5fab/debug/bin/unit/splinterdb_forked_child_test": "3e4ec9f7f2537630ee2cf0922980e297ce728812f3af919ac7c25861517439b0", + "build-main-7bb5fab/debug/bin/unit/superblock_test": "3a72433e2ffbaf3eac7569dbd7075316a0f92cfb1800b3c2929b9fc4dc5d98fb", + "build-main-7bb5fab/debug/bin/unit/util_test": "8831eb1033f7f8b1044cdadb4855a01c7600860094f08e09c8777a3a5c35a7c7", + "build-main-7bb5fab/debug/bin/unit/misc_test": "9aa73e20f50f8167fe5e17fe38e5877b6618bd00f1dda014847b616e1688ac8e", + "build-main-7bb5fab/debug/bin/unit/platform_apis_test": "f8db8facf723398fe026afec40873b16f30d803f1330b4af15561f3209845575", + "build-main-7bb5fab/debug/bin/unit/splinterdb_optimize_test": "7da6aab3cc8adaffdb9cb2124f79b22d5a42580a4f6902785fdb98631271186a", + "build-main-7bb5fab/debug/bin/unit/limitations_test": "1c330cc9e33841bf533170aa9555946bf386b68f0160fe972ec699de63d60e11", + "build-main-7bb5fab/debug/bin/unit/vector_test": "d576cb5a32ed877bd4c132edddbf9be10133178a1d56d57df916036fb064bbfc", + "build-main-7bb5fab/debug/bin/unit/trunk_gc_test": "77468298d16d7e79e6bb70cdaa0469726966fa5adc32668b1abf921951364a4e", + "build-main-7bb5fab/debug/bin/unit/btree_test": "d7276cedc7a769ca9ec6b35d22fef21f4011383eeb65026b1eef748bbc14e23e", + "build-main-7bb5fab/debug/bin/unit/merge_test": "aa2d84669bb8c68eac7208e8f319e9167a016e1a8183cb6641547430cab4c3fa", + "build-main-7bb5fab/debug/bin/unit/config_parse_test": "fb502722d136960f4c68b1a3f05e625d171815ff5fcf14e8c2d2c0ddc1218525", + "build-main-7bb5fab/debug/bin/unit/splinterdb_heap_id_mgmt_test": "623aad39e0959d5f6858918bdc9076967b61a3057806541ae6c2b80ec80e70c3", + "build-main-7bb5fab/debug/bin/unit/trunk_split_test": "803cbde58462b06fd0e5c7bf54fbb8b5580044119de0d304ed56142c510332c3", + "build-main-7bb5fab/debug/bin/unit/writable_buffer_test": "e7950396a87ea056a0dcb0a69cce2cc269c3aee158e10343dc6a6acbb4ecf376", + "build-main-7bb5fab/debug/bin/unit/writeback_set_test": "830bee256063535d4db8283be55d23fde0d694e04801209bf7baec55e4113bbd", + "build-main-7bb5fab/debug/bin/unit/splinterdb_quick_test": "d593090210f421d1c7270988625392c3d28354d79beb24c9aef0ef52746a6fcf", + "build-main-7bb5fab/debug/bin/unit/splinterdb_stress_test": "b769a65e064ac62280df7f8e574925e6ceeebf483c0abe5dfaf89c28f32e2237", + "build-main-7bb5fab/debug/bin/unit/task_system_test": "b7986d5b61d3e2d5cd91a98525a3b942491fbfef1c6ce4520bbe4218984fe60b", + "build-main-7bb5fab/debug/bin/unit/platform_threads_test": "652d529375fb5e108a9e6ba401a9813f9d579ae3c5a5ce9f29a6544352793091", + "build-main-7bb5fab/debug/bin/unit/trunk_async_test": "a78ff24314c545cb352f77aaa728e7abb8aa928757ed9d82d88f8ecbaf653d78", + "build-main-7bb5fab/debug/bin/unit/btree_stress_test": "e23b28dddb515dcb1ba33d5890b407485ea92331cdbf839918f15e65ff57f0ff", + "build-main-7bb5fab/debug/bin/unit/large_inserts_stress_test": "5a30f7de8bebb594e6d8da6464ac481ca6224f81ca6babb6eb396429c69893c0", + "build-main-7bb5fab/release/lib/libsplinterdb.so": "646e398251e082ec29e484a309ad6b17bfceb0444f36d7ce845a606859a72b1f", + "build-main-7bb5fab/release/lib/libsplinterdb.a": "99282ddd7e3254565b67d2eab3fc64ee3286b4eaf278e802f902b871a7b341d6", + "build-main-7bb5fab/release/bin/unit_test": "c54b5e4c5ccdd40ae9405bfa395995f61840df45943e275832a099f711de29a0", + "build-main-7bb5fab/release/bin/splinter_inspect": "a3c659169cf37c2cbfc1994114745a45ea646126f2228f7194911560e8fd3662", + "build-main-7bb5fab/release/bin/driver_test": "e2710534ac545769edd345b6f95e2507c31899f31e865070e049c2014a4189d3", + "build-main-7bb5fab/release/bin/unit/splinter_test": "d5205810538dc7389d446542b6fe18d4801109c3e9c8cf25f763ded003a6c38e", + "build-main-7bb5fab/release/bin/unit/splinterdb_forked_child_test": "eea724fb51018dcb734693acdb89d681397446486dc55ce2d61c8fb30dbb225d", + "build-main-7bb5fab/release/bin/unit/superblock_test": "07ec2e8e93ce89872a41cdf4f83724f7b87b9a9393fe1af115d7e30d8184e188", + "build-main-7bb5fab/release/bin/unit/util_test": "484f3b0228fbb325c4f797187e6be6130c0bd15ac60afde720171116332bd6a7", + "build-main-7bb5fab/release/bin/unit/misc_test": "da87925eabd482f2f214e6e54a36e04e39beb77a475030ccf7d1800ea43277fa", + "build-main-7bb5fab/release/bin/unit/platform_apis_test": "2e019ae1546936b02ccaedee91442997ee1d9083372c411fd9b495dc01706f0f", + "build-main-7bb5fab/release/bin/unit/splinterdb_optimize_test": "8b5aebd869cec3592f404a7bd1caaa509eaeef024808cbcc9218f2109a49deb4", + "build-main-7bb5fab/release/bin/unit/limitations_test": "b7a0c1499bde53c34d643935c137443a030cdc6355124c8e0f4e3bb4237e44b9", + "build-main-7bb5fab/release/bin/unit/vector_test": "95f58033e00899bc3c09d2cf4a1dad0ee270920010d8c3381ed636d138032e7b", + "build-main-7bb5fab/release/bin/unit/trunk_gc_test": "9d946f0ee7b4d7252c6c885f9dfab5b298aa18cd98ecfc9b76bd2e81dd0b1d0d", + "build-main-7bb5fab/release/bin/unit/btree_test": "88fc8bdff24f56f8d33797032a63b7c1bd5ec8c594103e3ca16a7cdf71db26f7", + "build-main-7bb5fab/release/bin/unit/merge_test": "38e8f4ed6e908050ab5ba330e721d9ced0d8b9a7ee218d222301d68dbf2a0853", + "build-main-7bb5fab/release/bin/unit/config_parse_test": "0ea930f0bcff2b7378e0f71004182a0f96bee1528fc3cd8a45406af4796a8609", + "build-main-7bb5fab/release/bin/unit/splinterdb_heap_id_mgmt_test": "f0bfc8c4c44e6a8aabf726beacba9394704b4e89a2db65992e2db88d72431e8d", + "build-main-7bb5fab/release/bin/unit/trunk_split_test": "7b72527eb68c83417ce442e4871f8f12efca8f382ccc149e667d416c538c36ec", + "build-main-7bb5fab/release/bin/unit/writable_buffer_test": "2561b07ea680e836128d32dbfdec19f7d7c75c4d3b3974f720bced27b81fd62d", + "build-main-7bb5fab/release/bin/unit/writeback_set_test": "d3abb37737a2956b4fe59e812fb98217c59607f3f965938c7dee2a867063e471", + "build-main-7bb5fab/release/bin/unit/splinterdb_quick_test": "950085d00ee73fe28037bd719b59b0b8ea802b022bf5ea35318c451ed51692fe", + "build-main-7bb5fab/release/bin/unit/splinterdb_stress_test": "2915fc95e5d364f7674c0e8d20c0f961ad9f44846a1a21cd93b45496cfa6df38", + "build-main-7bb5fab/release/bin/unit/task_system_test": "d9b0078184b8ef21730043c32ec8465129ba2e8c3e02f688d1d9414d8e0a5504", + "build-main-7bb5fab/release/bin/unit/platform_threads_test": "ccfa707d6f0bdf388dba0d62845cf1e3e98cc6ec0fadfa9766a0e62a70bbeb74", + "build-main-7bb5fab/release/bin/unit/trunk_async_test": "d1670e5668ebb7473691a018b38251cc529975bf67b971467d88e3299a3cdeb4", + "build-main-7bb5fab/release/bin/unit/btree_stress_test": "9b544b72916db3dc769fc6ab25ab9df5841502cd5753a1c98b5a0258b4f4de1d", + "build-main-7bb5fab/release/bin/unit/large_inserts_stress_test": "4a1a76a4a34b28b32f1a6b221f09d5fa992433a6fe307914151c804f40776085" + }, + "logs": { + "build-debug-4.log": "2d2a2ae60de93416062f879361aeaac014528ae8147922a943e833933facdf0e", + "release-quick.log": "fb9f96d901f9d6653e0295bcffe6220ded0a75f71db48ec3a5c763be57a50b0c", + "build-debug-5.log": "87e86d57017778cb00c49bdbb29447f0d50505de79049dc22bc77d3f2a98aad9", + "validation-release.json": "ecfd4bd663739ae819754ef4239ebcde4796d3cef6615d0ac291b547c46ccbb0", + "debug-trunk-async.log": "358786a8840bf748a73c6f3d58cb3eb2bac80a7107a6387bfa11a167fe5265ed", + "release-btree.log": "43c08095c908b960b7c40ce73c84804496b2538fd4e5d1e6d67cf78f14d311f5", + "debug-quick.log": "2e348ee7750ccc4c95d7c839ff752e2bb23183c605af0d0df0efabcb46033421", + "validation-debug.json": "d1be87ee55ad22eb4d14b295ff62b78e8e77ca02e9031cc13aefdfb3e325f853", + "debug-filter.log": "49259f7e14bc368c5ba8d9410dd23b58d270333eaf6b13736bbd038ca9b1d924", + "release-splinter-4gib.log": "40f1c0c881534b1c7584bf5db4e324181858eb9668642ca7d83649627dd82664", + "build-debug-formatted.log": "be00255ea6bc75794bfa659f28ceb9e28a2144f8552518ae8b01e5b6ffc5c005", + "release-splinter.log": "64e3e5c3f4c796d2be3661ddbc06e3da816516e1497a2b98aafc2a84a6947de7", + "build-debug.log": "72da8e8587c1a9bd7946519a223a788f72b64320727055aa3d97e79c7ee9aeef", + "debug-splinter-4gib.log": "19abfb216913e1d89c0d6ef7f7dc7cd0ff4275f8de44e5e2cd7a3ca094b90f1f", + "release-superblock.log": "e8f058f14fdc7b2ae6c78a4983c56e5ae3c3cee7cdc4b46a303efa3a18cb2b1c", + "build-debug-3.log": "0ca67435c379811d39b3c6447642a4aa39cc7fad4909812f7a15366000fe74a0", + "test_integration.py": "cabf042d58089771e6d134241f594e273d93bd259b47d77a660c69424a9ed298", + "release-log.log": "94d99f7647332b2b2a3310da728d30fddc459ed2c722060f5d0fc1bc4003e55e", + "build-debug-final.log": "7b33ebeed7ab9431c221c444ef151ae009548064525fec25ab12f4d9b95a3305", + "debug-superblock.log": "417133bc50d178e582c1bdcbff4182a07ced3994a81d29eb409f4b713300f7d2", + "release-trunk-async.log": "58337bde38e57564caa2561b024f9e05bedf6d7c6d6f1b220fee48ef6c622449", + "validation-debug-4gib.json": "985687f55c536e818cab8a71be2af260dbe1a53c8a74fd3a041ad4ed58cea79c", + "build-release.log": "c6cabf428fd6973cd2bd1fa01b04be25c538f891ac547c2cf7f16834c3301bf7", + "debug-btree.log": "f8ca9a5b8c682cd418b940a5afef55253c9dc165dced3c780c25b4b141e8d05e", + "validation-release-4gib.json": "6333e64bcc5246e1accccbb2058edbb599cecc1b089e77ee9a82116d420ffda0", + "debug-splinter.log": "20208a30e6dcf9076de207d0712849e24113084851af3813c3bdbd9d7e492a7f", + "debug-log.log": "372a0c0e44520c2d299d25bde942a0dfdd13119b77cfa2b5e4cc032e3e4d8095", + "build-debug-resolved.log": "9a3be29c63ca79da78381300ed25248f7c6089e4613f64ae58999f98b04a66fc", + "release-filter.log": "bcfd592c4f9020413d7751bbf1e69d0de0a8c21e9953b19768e35c17564cd7e5" + }, + "notes": [ + "Initial 4 GiB core test configuration could not reserve the sparse height-three blob test logical extents; corrected to 64 GiB with no production source change.", + "No timed benchmarks run by this integration worker. Root coordinates benchmark serialization.", + "Old narrow B-tree index entries remain readable; upstream wide counters use flagged high words. Disk format remains 8." + ], + "head": "d21b87f5cfac709fb0fa74c3585fb38936c08852", + "parents": [ + "fb878f134c245276629828203a186dd2f26df981", + "7bb5fab305b9c6179e31f75998dabea833537a08" + ] +} diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-btree.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-btree.log new file mode 100644 index 000000000..7111e2425 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-btree.log @@ -0,0 +1,35 @@ +Running 1 CTests, suite name 'btree', test case 'all'. +TEST 1/11 btree:test_leaf_hdr_search [OK] +TEST 2/11 btree:test_index_hdr [OK] +TEST 3/11 btree:test_index_hdr_search [OK] +TEST 4/11 btree:test_leaf_split [OK] +TEST 5/11 btree:test_defragmented_leaf_bytes [OK] +TEST 6/11 btree:test_defragmented_index_bytes [OK] +TEST 7/11 btree:test_invalid_branch_point_lookup Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 8/11 btree:test_invalid_branch_async_point_lookup Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 9/11 btree:test_invalid_branch_iterator Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 10/11 btree:test_invalid_branch_metadata_queries Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 0 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +Unsupported branch format version 2 at 4096; this build requires version 1. +[OK] +TEST 11/11 btree:test_leaf_hdr [OK] +RESULTS: 11 tests (11 ok, 0 failed, 0 skipped) ran in 2 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-filter.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-filter.log new file mode 100644 index 000000000..5867d8e38 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-filter.log @@ -0,0 +1,65 @@ +build-main-7bb5fab/release/bin/driver_test: splinterdb_build_version fb878f13-dirty +Dispatch test filter_test +filter_test: routing filter index expansion test started +filter_test: mixed routing filter fingerprint sizes test started +filter_test: routing filter arithmetic limits test started +filter_test: malicious routing/mini metadata test started +Malformed mini allocator metadata: unsupported format version (addr=4980736). +Malformed mini allocator metadata: metadata page number disagrees with predecessor link (addr=4980736). +Malformed mini allocator metadata: metadata page number is not contiguous (addr=4984832). +Malformed mini allocator metadata: metadata page number is not contiguous (addr=4980736). +Malformed mini allocator metadata: metadata entry position is invalid (addr=4980736). +Malformed mini allocator metadata: metadata entry position is invalid (addr=4980736). +Malformed mini allocator metadata: metadata extent entry is invalid (addr=4980736). +filter_test: routing filter basic test started +num input keys 1048575 estimate 1048653 num_unique 1040503 +num input keys 1572862 estimate 1573255 num_unique 1554958 +num input keys 2097150 estimate 2097684 num_unique 2065239 +num input keys 2534056 estimate 2534923 num_unique 2487644 +num input keys 3075820 estimate 3077087 num_unique 3007608 +num input keys 3460297 estimate 3461800 num_unique 3374028 +num input keys 4014546 estimate 4016523 num_unique 3898690 +num input keys 4392783 estimate 4395331 num_unique 4254486 +across filters: num input keys 4392783 estimate 4408343 +routing filter basic test: false positive rate 0.0625 +filter_test: routing filter basic test started +num input keys 100 estimate 100 num_unique 100 +num input keys 150 estimate 150 num_unique 150 +num input keys 199 estimate 199 num_unique 199 +num input keys 241 estimate 241 num_unique 241 +num input keys 292 estimate 292 num_unique 292 +num input keys 328 estimate 328 num_unique 328 +num input keys 379 estimate 379 num_unique 379 +num input keys 415 estimate 415 num_unique 415 +across filters: num input keys 415 estimate 0 +routing filter basic test: false positive rate 0.0000 +filter_test: routing filter basic test started +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +across filters: num input keys 1 estimate 0 +routing filter basic test: false positive rate 0.0000 +filter_test: routing filter basic test started +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +num input keys 1 estimate 1 num_unique 1 +across filters: num input keys 1 estimate 0 +routing filter basic test: false positive rate 0.0000 diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-log.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-log.log new file mode 100644 index 000000000..b6656c744 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-log.log @@ -0,0 +1,11 @@ +build-main-7bb5fab/release/bin/driver_test: splinterdb_build_version fb878f13-dirty +Dispatch test log_test + +Started log_test!! +Malformed log page sequence at page 1843200: group=3 page=0 follows group=1 page=0 (terminated). +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +shard_log_iterator_init: record validation failed in group 1 at log page 139591680: Invalid argument; discarding this group and the rest +Malformed log page sequence at page 140378112: group=1 page=0 follows group=1 page=0. +shard_log_iterator_init: record validation failed in group 2 at log page 141168640: Input/output error; discarding this group and the rest +log returned 500000 of 500000 entries diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-quick.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-quick.log new file mode 100644 index 000000000..490cdb83d --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-quick.log @@ -0,0 +1,55 @@ +Running 1 CTests, suite name 'splinterdb_quick', test case 'all'. +TEST 1/49 splinterdb_quick:test_get_cost_stats [OK] +TEST 2/49 splinterdb_quick:test_checkpoint_log_grace_default [OK] +TEST 3/49 splinterdb_quick:test_durable_barrier [OK] +TEST 4/49 splinterdb_quick:test_logged_large_value_replaces_old_value [OK] +TEST 5/49 splinterdb_quick:test_basic_flow [OK] +TEST 6/49 splinterdb_quick:test_pthread_auto_registration [OK] +TEST 7/49 splinterdb_quick:test_result_auto_registration_with_stats [OK] +TEST 8/49 splinterdb_quick:test_stats_reset_clears_cache_io [OK] +TEST 9/49 splinterdb_quick:test_final_merge_error_propagation [OK] +TEST 10/49 splinterdb_quick:test_fetch_and_update_catches_up_after_rotation [OK] +TEST 11/49 splinterdb_quick:test_fetch_and_update_rejects_incorporated_boundary [OK] +TEST 12/49 splinterdb_quick:test_range_init_error_cleanup_preserves_branches setup_ordered_iterators: exception: Cannot allocate memory +core_apply_to_range: range iterator init failed: Cannot allocate memory +[OK] +TEST 13/49 splinterdb_quick:stats_reset_excludes_active_operations [OK] +TEST 14/49 splinterdb_quick:test_apis_for_large_key [OK] +TEST 15/49 splinterdb_quick:test_key_size_gt_max_inline_key_size [OK] +TEST 16/49 splinterdb_quick:test_value_size_gt_max_value_size [OK] +TEST 17/49 splinterdb_quick:test_large_lookup_survives_memtable_recycle [OK] +TEST 18/49 splinterdb_quick:test_iterator_survives_memtable_recycle [OK] +TEST 19/49 splinterdb_quick:test_variable_length_values [OK] +TEST 20/49 splinterdb_quick:test_basic_iterator [OK] +TEST 21/49 splinterdb_quick:test_iterator_delete_current [OK] +TEST 22/49 splinterdb_quick:test_empty_iterator [OK] +TEST 23/49 splinterdb_quick:test_bounded_iterator [OK] +TEST 24/49 splinterdb_quick:test_bounded_iterator_start_normalization [OK] +TEST 25/49 splinterdb_quick:test_splinterdb_iterator_with_startkey [OK] +TEST 26/49 splinterdb_quick:test_splinterdb_iterator_with_non_existent_startkey [OK] +TEST 27/49 splinterdb_quick:test_splinterdb_iterator_with_missing_startkey_in_sequence [OK] +TEST 28/49 splinterdb_quick:test_iterator_prev_and_next [OK] +TEST 29/49 splinterdb_quick:test_iterator_reversal_on_key_in_two_branches [OK] +TEST 30/49 splinterdb_quick:test_iterator_reversal_with_three_branches [OK] +TEST 31/49 splinterdb_quick:test_rewriting_keys_does_not_grow_trunk [OK] +TEST 32/49 splinterdb_quick:test_close_and_reopen [OK] +TEST 33/49 splinterdb_quick:test_logged_close_and_reopen [OK] +TEST 34/49 splinterdb_quick:test_repeated_insert_close_reopen [OK] +TEST 35/49 splinterdb_quick:test_custom_data_config [OK] +TEST 36/49 splinterdb_quick:test_existence_only_memtable_lookup [OK] +TEST 37/49 splinterdb_quick:test_existence_only_trunk_lookup_skips_branches [OK] +TEST 38/49 splinterdb_quick:test_write_api_returns_old_value [OK] +TEST 39/49 splinterdb_quick:test_write_api_old_result_existence_only [OK] +TEST 40/49 splinterdb_quick:test_write_api_old_result_custom_merge_semantics [OK] +TEST 41/49 splinterdb_quick:test_write_api_old_result_merges_memtable_and_trunk [OK] +TEST 42/49 splinterdb_quick:test_write_api_old_result_respects_trunk_delete_shadow [OK] +TEST 43/49 splinterdb_quick:test_iterator_custom_comparator [OK] +TEST 44/49 splinterdb_quick:test_iterator_init_bug [OK] +TEST 45/49 splinterdb_quick:test_splinterdb_create_w_background_threads [OK] +TEST 46/49 splinterdb_quick:test_splinterdb_create_w_all_background_threads [OK] +TEST 47/49 splinterdb_quick:test_iterator_thread_handoff [OK] +TEST 48/49 splinterdb_quick:test_blob_tree_crash_recovery Running 1 CTests, suite name 'splinterdb_quick', test case 'test_blob_tree_crash_recovery'. +TEST 1/1 splinterdb_quick:test_blob_tree_crash_recovery Running 1 CTests, suite name 'splinterdb_quick', test case 'test_blob_tree_crash_recovery'. +TEST 1/1 splinterdb_quick:test_blob_tree_crash_recovery [OK] +TEST 49/49 splinterdb_quick:test_get_config_returns_effective_config [OK] +RESULTS: 49 tests (49 ok, 0 failed, 0 skipped) ran in 11118 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-splinter-4gib.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-splinter-4gib.log new file mode 100644 index 000000000..c325c5ae0 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-splinter-4gib.log @@ -0,0 +1,89 @@ +Running 1 CTests, suite name 'splinter', test case 'all'. +TEST 1/38 splinter:test_inserts [OK] +TEST 2/38 splinter:test_btree_pack_abort_releases_reserved_extents btree_pack error status: 5 +[OK] +TEST 3/38 splinter:test_btree_pack_rejects_counter_overflow btree_pack: output statistics exceed the on-disk limit +btree_pack error status: 28 +[OK] +TEST 4/38 splinter:test_malformed_trunk_pivot_bundle [OK] +TEST 5/38 splinter:test_invalid_branch_reference_rejected Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +Unsupported branch format version 2 at 6553600; this build requires version 1. +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +[OK] +TEST 6/38 splinter:test_malformed_trunk_pivot_stats trunk_merge_lookup: ondisk_node_find_pivot failed: 22 +trunk_merge_lookup_async: ondisk_node_find_pivot_async failed: 22 +[OK] +TEST 7/38 splinter:test_malformed_trunk_child_height trunk_merge_lookup_async: ondisk_node_handle_init_async failed: 22 +[OK] +TEST 8/38 splinter:test_two_log_checkpoint [OK] +TEST 9/38 splinter:test_reopen_with_changed_filter_config [OK] +TEST 10/38 splinter:test_logged_large_insert_shares_blob_extents [OK] +TEST 11/38 splinter:test_durable_barrier_waits_for_visible_insert_log_write [OK] +TEST 12/38 splinter:test_durable_barrier_packs_concurrent_small_tails [OK] +TEST 13/38 splinter:test_shard_log_first_fit_decreasing_packing [OK] +TEST 14/38 splinter:test_shard_log_page_alloc_failure_retries_open_buffer shard_log_graduate_buffer: out of log space +shard_log_graduate_group: failed to terminate group 1: No space left on device +[OK] +TEST 15/38 splinter:test_durable_barrier_without_log_publishes_root [OK] +TEST 16/38 splinter:test_checkpoint_reports_terminal_incorporation_failure trunk_node_serialize():2802: cache_alloc() failedserialize_nodes():2986: node_serialize() failedserialize_nodes_and_save_contingent_compactions: serialize_nodes failed: 12 +restore_balance_leaf():6155: serialize_nodes() failed: Cannot allocate memorytrunk_flush_prepare: flush_then_compact failed: 12 +trunk_incorporate_prepare failed: Cannot allocate memory +Memtable incorporation failed: generation=0 state=INCORPORATION_ASSIGNED status=Cannot allocate memory memtable_root=262144 +memtable flush failed: Cannot allocate memory +Shutdown found memtable from failed incorporation: generation=0 state=INCORPORATION_FAILED status=Cannot allocate memory memtable_root=262144 compacted_root=0 +[OK] +TEST 17/38 splinter:test_durable_barrier_reopens_writers_before_device_barrier [OK] +TEST 18/38 splinter:test_concurrent_durable_barriers_coalesce [OK] +TEST 19/38 splinter:test_durable_barrier_waits_for_checkpoint_publication [OK] +TEST 20/38 splinter:test_log_prepare_failure_leaves_update_unapplied core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_unmount: failed to publish the unmount root: Input/output error +core_unmount: shutdown retained recovery state; the next mount must recover +[OK] +TEST 21/38 splinter:test_checkpoint_advance_retries_cut_publish_failure core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +[OK] +TEST 22/38 splinter:test_checkpoint_advance_retries_completion_failure core_checkpoint_advance: completion publish failed: Input/output error +core_checkpoint_advance: completion publish failed: Input/output error +[OK] +TEST 23/38 splinter:test_self_managed_checkpoints_reclaim_log_space [OK] +TEST 24/38 splinter:test_auto_checkpoint [OK] +TEST 25/38 splinter:test_auto_checkpoint_grace_allows_natural_rotation [OK] +TEST 26/38 splinter:test_auto_checkpoint_grace_forces_overwrite_rotation [OK] +TEST 27/38 splinter:test_auto_checkpoint_on_overwrites [OK] +TEST 28/38 splinter:test_recover_allocations_reproduces_persisted_map [OK] +TEST 29/38 splinter:test_unmount_skips_allocator_map_that_needs_rebuild core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 30/38 splinter:test_late_reclamation_failure_requires_recovery WARNING: abandoned reclamation of mini allocator at 6553600 (type misc): Connection timed out. Remaining space is unavailable until reopen; clean allocation state will not be written. Recovery is required. +WARNING: core_unmount: reclamation failed (Connection timed out); remaining space was retained. Shutdown requires recovery and the next open will rebuild allocation state. +core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 31/38 splinter:test_mount_active_checkpoint_stays_mountable [OK] +TEST 32/38 splinter:test_lookups [OK] +TEST 33/38 splinter:test_splinter_print_diags [OK] +TEST 34/38 splinter:test_cache_read_errors_reach_core_operations ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 110 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 110 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 110 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 110 +[OK] +TEST 35/38 splinter:test_blob_descriptor_sizes [OK] +TEST 36/38 splinter:test_blob_trees [OK] +TEST 37/38 splinter:test_blob_height_three_and_large_stats [FAIL] + ERR: tests/unit/splinter_test.c:5941 should be true +TEST 38/38 splinter:test_blob_checksums OS-pid=9, OS-tid=9, Thread-ID=0, Assertion failed at src/platform_linux/platform_threads.c:420:platform_register_thread(): "thread_tid == INVALID_TID". Attempt to register thread that is already registered as thread 0 + diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-splinter.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-splinter.log new file mode 100644 index 000000000..ca7effb79 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-splinter.log @@ -0,0 +1,88 @@ +Running 1 CTests, suite name 'splinter', test case 'all'. +TEST 1/38 splinter:test_inserts [OK] +TEST 2/38 splinter:test_btree_pack_abort_releases_reserved_extents btree_pack error status: 5 +[OK] +TEST 3/38 splinter:test_btree_pack_rejects_counter_overflow btree_pack: output statistics exceed the on-disk limit +btree_pack error status: 28 +[OK] +TEST 4/38 splinter:test_malformed_trunk_pivot_bundle [OK] +TEST 5/38 splinter:test_invalid_branch_reference_rejected Unsupported branch format version 2 at 8519680; this build requires version 1. +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +Unsupported branch format version 2 at 8519680; this build requires version 1. +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +Unsupported branch format version 2 at 8519680; this build requires version 1. +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 22 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 22 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 22 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 22 +trunk_collect_branches: trunk_collect_bundle_branches failed: 22 +[OK] +TEST 6/38 splinter:test_malformed_trunk_pivot_stats trunk_merge_lookup: ondisk_node_find_pivot failed: 22 +trunk_merge_lookup_async: ondisk_node_find_pivot_async failed: 22 +[OK] +TEST 7/38 splinter:test_malformed_trunk_child_height trunk_merge_lookup_async: ondisk_node_handle_init_async failed: 22 +[OK] +TEST 8/38 splinter:test_two_log_checkpoint [OK] +TEST 9/38 splinter:test_reopen_with_changed_filter_config [OK] +TEST 10/38 splinter:test_logged_large_insert_shares_blob_extents [OK] +TEST 11/38 splinter:test_durable_barrier_waits_for_visible_insert_log_write [OK] +TEST 12/38 splinter:test_durable_barrier_packs_concurrent_small_tails [OK] +TEST 13/38 splinter:test_shard_log_first_fit_decreasing_packing [OK] +TEST 14/38 splinter:test_shard_log_page_alloc_failure_retries_open_buffer shard_log_graduate_buffer: out of log space +shard_log_graduate_group: failed to terminate group 1: No space left on device +[OK] +TEST 15/38 splinter:test_durable_barrier_without_log_publishes_root [OK] +TEST 16/38 splinter:test_checkpoint_reports_terminal_incorporation_failure trunk_node_serialize():2802: cache_alloc() failedserialize_nodes():2986: node_serialize() failedserialize_nodes_and_save_contingent_compactions: serialize_nodes failed: 12 +restore_balance_leaf():6155: serialize_nodes() failed: Cannot allocate memorytrunk_flush_prepare: flush_then_compact failed: 12 +trunk_incorporate_prepare failed: Cannot allocate memory +Memtable incorporation failed: generation=0 state=INCORPORATION_ASSIGNED status=Cannot allocate memory memtable_root=2228224 +memtable flush failed: Cannot allocate memory +Shutdown found memtable from failed incorporation: generation=0 state=INCORPORATION_FAILED status=Cannot allocate memory memtable_root=2228224 compacted_root=0 +[OK] +TEST 17/38 splinter:test_durable_barrier_reopens_writers_before_device_barrier [OK] +TEST 18/38 splinter:test_concurrent_durable_barriers_coalesce [OK] +TEST 19/38 splinter:test_durable_barrier_waits_for_checkpoint_publication [OK] +TEST 20/38 splinter:test_log_prepare_failure_leaves_update_unapplied core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_unmount: failed to publish the unmount root: Input/output error +core_unmount: shutdown retained recovery state; the next mount must recover +[OK] +TEST 21/38 splinter:test_checkpoint_advance_retries_cut_publish_failure core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +core_checkpoint_advance: failed to publish the log cut; will retry: Input/output error +[OK] +TEST 22/38 splinter:test_checkpoint_advance_retries_completion_failure core_checkpoint_advance: completion publish failed: Input/output error +core_checkpoint_advance: completion publish failed: Input/output error +[OK] +TEST 23/38 splinter:test_self_managed_checkpoints_reclaim_log_space [OK] +TEST 24/38 splinter:test_auto_checkpoint [OK] +TEST 25/38 splinter:test_auto_checkpoint_grace_allows_natural_rotation [OK] +TEST 26/38 splinter:test_auto_checkpoint_grace_forces_overwrite_rotation [OK] +TEST 27/38 splinter:test_auto_checkpoint_on_overwrites [OK] +TEST 28/38 splinter:test_recover_allocations_reproduces_persisted_map [OK] +TEST 29/38 splinter:test_unmount_skips_allocator_map_that_needs_rebuild core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 30/38 splinter:test_late_reclamation_failure_requires_recovery WARNING: abandoned reclamation of mini allocator at 8519680 (type misc): Connection timed out. Remaining space is unavailable until reopen; clean allocation state will not be written. Recovery is required. +WARNING: core_unmount: reclamation failed (Connection timed out); remaining space was retained. Shutdown requires recovery and the next open will rebuild allocation state. +core_unmount: allocator reference accounting is incomplete; leaving allocation state invalid for rebuild on the next mount +[OK] +TEST 31/38 splinter:test_mount_active_checkpoint_stays_mountable [OK] +TEST 32/38 splinter:test_lookups [OK] +TEST 33/38 splinter:test_splinter_print_diags [OK] +TEST 34/38 splinter:test_cache_read_errors_reach_core_operations ondisk_bundle_merge_lookup: btree_lookup_and_merge failed: 110 +trunk_merge_lookup: ondisk_bundle_merge_lookup failed: 110 +ondisk_bundle_merge_lookup_async: btree_lookup_and_merge_async failed: 110 +trunk_merge_lookup_async: ondisk_bundle_merge_lookup_async failed: 110 +[OK] +TEST 35/38 splinter:test_blob_descriptor_sizes [OK] +TEST 36/38 splinter:test_blob_trees [OK] +TEST 37/38 splinter:test_blob_height_three_and_large_stats [OK] +TEST 38/38 splinter:test_blob_checksums [OK] +RESULTS: 38 tests (38 ok, 0 failed, 0 skipped) ran in 36303 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-superblock.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-superblock.log new file mode 100644 index 000000000..0ebdd7a10 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-superblock.log @@ -0,0 +1,43 @@ +Running 1 CTests, suite name 'superblock', test case 'all'. +TEST 1/16 superblock:test_snapshot_persists_state [OK] +TEST 2/16 superblock:test_two_log_checkpoint_transitions [OK] +TEST 3/16 superblock:test_snapshot_preserves_unincorporated_sealed_log [OK] +TEST 4/16 superblock:test_two_log_checkpoint_torn_begin [OK] +TEST 5/16 superblock:test_torn_write_falls_back_to_older_generation [OK] +TEST 6/16 superblock:test_generation_wrap_selects_newest_slot [OK] +TEST 7/16 superblock:test_both_slots_corrupt_is_not_found superblock_mount: no valid superblock found +[OK] +TEST 8/16 superblock:test_mount_rejects_invalid_metadata superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +superblock_mount: no valid superblock found +[OK] +TEST 9/16 superblock:test_slot_read_error_falls_back_to_peer [OK] +TEST 10/16 superblock:test_both_slot_read_errors_are_reported superblock_mount: no readable valid superblock found: Input/output error +[OK] +TEST 11/16 superblock:test_read_geometry superblock_read_bootstrap: no valid superblock found +[OK] +TEST 12/16 superblock:test_read_bootstrap_allocation_state superblock_read_bootstrap: no valid superblock found +[OK] +TEST 13/16 superblock:test_bootstrap_geometry_metadata_budget Device geometry needs 134217728 bytes of cache lookup metadata, which exceeds the configured cache budget of 67108864 bytes. +Failed to open existing SplinterDB device 'db' with specified configuration: No space left on device +[OK] +TEST 14/16 superblock:test_unknown_structure_version_rejected Unsupported trunk format version 2; this build requires version 1. +Unsupported trunk format version 2; this build requires version 1. +Unsupported allocator format version 2; this build requires version 1. +Unsupported allocator format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +Unsupported log format version 2; this build requires version 1. +[OK] +TEST 15/16 superblock:test_unsupported_superblock_version_rejected Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 7; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +Unsupported superblock format version 9; this build requires version 8. +[OK] +TEST 16/16 superblock:test_format_sets_fresh_state [OK] +RESULTS: 16 tests (16 ok, 0 failed, 0 skipped) ran in 42 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/release-trunk-async.log b/benchmarks/results-20261013/main-7bb5fab/tests/release-trunk-async.log new file mode 100644 index 000000000..a884d0daa --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/release-trunk-async.log @@ -0,0 +1,5 @@ +Running 1 CTests, suite name 'trunk_async', test case 'all'. +TEST 1/2 trunk_async:pivot_read_error_survives_cross_thread_yield trunk_ondisk_node_read_pivot_key_async():1462: ondisk_node_handle_setup_content_page() failed: Connection timed outtrunk_merge_lookup_async: ondisk_node_find_pivot_async failed: 110 +[OK] +TEST 2/2 trunk_async:pivot_header_survives_cross_thread_yield [OK] +RESULTS: 2 tests (2 ok, 0 failed, 0 skipped) ran in 0 ms diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/shared/before.json b/benchmarks/results-20261013/main-7bb5fab/tests/shared/before.json new file mode 100644 index 000000000..2143ea26f --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/shared/before.json @@ -0,0 +1,97 @@ +{ + "main": "db3047a812d9b193b87f92fc0be7b75e923b89ee", + "target": "7bb5fab305b9c6179e31f75998dabea833537a08", + "projects": [ + { + "name": "cache-errors", + "branch": "robj/cache-errors", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/cache-errors", + "head": "2907e97e90a7861e789192485e0892522a9feb3a", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/cache-errors" + }, + { + "name": "cache-handoff", + "branch": "robj/cache-handoff", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/cache-handoff", + "head": "c76b6923eb5b915792e1e8303d266fb072f9348f", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/cache-handoff" + }, + { + "name": "disk-format", + "branch": "robj/disk-format", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format", + "head": "fb878f134c245276629828203a186dd2f26df981", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/disk-format" + }, + { + "name": "filtered-ranges", + "branch": "robj/filtered-ranges", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges", + "head": "b8ce54bf9b793c3576a9f96c19d4a51846e83269", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/filtered-ranges" + }, + { + "name": "log-group-limit", + "branch": "robj/log-group-limit", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/log-group-limit", + "head": "f295866abe0cd5d16c7954c7748afc2ddcdc6843", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/log-group-limit" + }, + { + "name": "branch-pointers", + "branch": "robj/branch-pointers", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/branch-pointers", + "head": "37787ae95aadc84fca543af0592a24448576dcd9", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/branch-pointers" + }, + { + "name": "branch-prefix", + "branch": "robj/branch-prefix", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/branch-prefix", + "head": "919b61f04355248a26d054a0a7948d5fbb33832a", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/branch-prefix" + }, + { + "name": "lazy-merge", + "branch": "robj/lazy-merge", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/lazy-merge", + "head": "9754337703a754546bd2ab38028856df185f897a", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/lazy-merge" + }, + { + "name": "cache-directory", + "branch": "robj/cache-directory", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/cache-directory", + "head": "9ecae65d8f0dd114132adf06fa326cb2389f5dc0", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/cache-directory" + }, + { + "name": "async-durable-barrier", + "branch": "robj/async-durable-barrier", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/async-durable-barrier", + "head": "c99b1dcd66ed3648eb592f45182eb8021a94cd7e", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/async-durable-barrier" + }, + { + "name": "cache-errors-direct24", + "branch": "robj/cache-errors-direct24", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/cache-errors-direct24", + "head": "573ba9ef6a2fe8d1293017556c29036340ea98ff", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/cache-errors-direct24" + }, + { + "name": "cache-errors-null", + "branch": "robj/cache-errors-null", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/cache-errors-null", + "head": "7d1f0a56ebc253bccc1bd74799eed7bf6e0d7fb3", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/cache-errors-null" + }, + { + "name": "cache-invalid-addresses", + "branch": "robj/cache-invalid-addresses", + "worktree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/cache-invalid-addresses", + "head": "193ba3e485df4e7438ee0fda78bb162258e3f054", + "backup_ref": "refs/heads/backup/projects-before-main-7bb5fab/cache-invalid-addresses" + } + ] +} diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/shared/run_job.py b/benchmarks/results-20261013/main-7bb5fab/tests/shared/run_job.py new file mode 100644 index 000000000..299c91084 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/shared/run_job.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Coordinate this campaign's builds/tests and disk-backed benchmarks.""" +import argparse +import fcntl +import json +import os +from pathlib import Path +import subprocess +import tempfile +import time + +ROOT = Path(__file__).resolve().parent.parent +p = argparse.ArgumentParser() +p.add_argument('mode', choices=['build', 'test', 'benchmark']) +p.add_argument('--name', required=True) +p.add_argument('--db-dir', type=Path) +p.add_argument('command', nargs=argparse.REMAINDER) +a = p.parse_args() +command = a.command[1:] if a.command[:1] == ['--'] else a.command +if not command: + p.error('command required after --') +if a.mode == 'benchmark': + if a.db_dir is None: + p.error('benchmark requires --db-dir') + a.db_dir.mkdir(parents=True, exist_ok=True) + db_dir = a.db_dir.resolve() + if not db_dir.is_relative_to(ROOT / 'benchmarks'): + p.error('benchmark database directory must be under the campaign benchmarks directory') + fs = subprocess.check_output(['findmnt', '-n', '-o', 'FSTYPE,SOURCE', '-T', str(db_dir)], text=True).strip() + if fs.split()[0] in {'tmpfs', 'ramfs'} or not fs.split()[1].startswith('/dev/'): + p.error('benchmark storage must be a verified disk filesystem: ' + fs) + bench_lock = open(ROOT / 'coordination' / 'benchmark.lock', 'a+') + fcntl.flock(bench_lock, fcntl.LOCK_EX) + print('Benchmark database storage:', fs, str(db_dir), flush=True) +machine_lock = open(ROOT / 'coordination' / 'machine.lock', 'a+') +fcntl.flock(machine_lock, fcntl.LOCK_EX if a.mode == 'benchmark' else fcntl.LOCK_SH) +record = dict(name=a.name, mode=a.mode, cwd=os.getcwd(), command=command, + started=time.time(), pid=os.getpid()) +if a.db_dir is not None: + record['db_dir'] = str(a.db_dir.resolve()) +print('Starting', a.name, a.mode, flush=True) +try: + # Tool commands run in distinct PID namespaces; GNU make's FIFO names + # otherwise collide when several builds choose /tmp/GMfifo8. + with tempfile.TemporaryDirectory(prefix='splinter-' + a.name + '-', dir='/tmp') as temporary: + environment = dict(os.environ, TMPDIR=temporary) + result = subprocess.run(command, env=environment) + record['returncode'] = result.returncode +finally: + record['finished'] = time.time() + with open(ROOT / 'coordination' / 'jobs.jsonl', 'a') as journal: + fcntl.flock(journal, fcntl.LOCK_EX) + journal.write(json.dumps(record) + '\n') +raise SystemExit(result.returncode) diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/shared/test_root_branches.py b/benchmarks/results-20261013/main-7bb5fab/tests/shared/test_root_branches.py new file mode 100644 index 000000000..c83e1f546 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/shared/test_root_branches.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Focused merge regression checks; invoke inside run_job.py test.""" +import hashlib +import json +from pathlib import Path +import re +import subprocess +import sys +import time + +project, mode = sys.argv[1:] +campaign = Path(__file__).resolve().parents[2] +tree = campaign / 'worktrees' / project +out = Path(__file__).resolve().parent / project +scratch = out / ('scratch-' + mode) +scratch.mkdir(parents=True, exist_ok=True) +build = tree / 'build-main-7bb5fab' +link = scratch / build.name +if not link.exists(): + link.symlink_to(build, target_is_directory=True) +base = build / mode / 'bin' +commands = [] +unit = lambda name: str(base / 'unit' / (name + '_test')) +if project == 'cache-invalid-addresses': + commands.append(('writeback', [unit('writeback_set')])) +else: + commands.append(('merge', [unit('merge')])) +commands.append(('quick', [unit('splinterdb_quick')])) +core_options = ['--db-location', str(scratch / 'core.db'), + '--cache-capacity-mib', '128', '--db-capacity-gib', '64'] +commands += [('cache-core', [unit('splinter'), *core_options, 'test_cache_read_errors_']), + ('blob-core', [unit('splinter'), *core_options, 'test_blob_'])] +if project == 'cache-invalid-addresses' and mode == 'debug': + for flag in ['prefetch-admission', 'async']: + commands.append((flag, [str(base / 'driver_test'), 'cache_test', '--' + flag, + '--db-location', str(scratch / (flag + '.db')), + '--cache-capacity-mib', '32', '--db-capacity-gib', '1'])) +if project == 'filtered-ranges': + commands.append(('filter', [str(base / 'driver_test'), 'filter_test', + '--db-location', str(scratch / 'filter.db'), + '--cache-capacity-mib', '64', '--db-capacity-gib', '4'])) +validation = out / ('validation-' + mode + '.json') +old_records = json.loads(validation.read_text()) if validation.exists() else [] +records = [] +for name, cmd in commands: + previous = next((r for r in old_records if r['name'] == name), None) + if previous and previous['returncode'] == 0 and previous['command'] == cmd: + records.append(previous) + continue + print(project, mode, name, flush=True) + log = out / (mode + '-' + name + '.log') + if log.exists(): + log.rename(log.with_suffix('.initial.log')) + (out / (mode + '-' + name + '.initial.json')).write_text(json.dumps(previous, indent=2) + '\n') + record = dict(name=name, command=cmd, started=time.time()) + with log.open('w') as stream: + result = subprocess.run(cmd, cwd=scratch, stdout=stream, stderr=subprocess.STDOUT) + content = log.read_text() + record.update(returncode=result.returncode, finished=time.time(), + log_sha256=hashlib.sha256(log.read_bytes()).hexdigest(), + summaries=[line for line in content.splitlines() if 'RESULTS:' in line]) + records.append(record) + validation.write_text(json.dumps(records, indent=2) + '\n') + if result.returncode: + raise SystemExit(result.returncode) + if name not in ['prefetch-admission', 'async', 'filter']: + assert re.search(r'RESULTS: [1-9][0-9]* tests', content), (name, content) diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/test_integration.py b/benchmarks/results-20261013/main-7bb5fab/tests/test_integration.py new file mode 100644 index 000000000..b2749dbc8 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/test_integration.py @@ -0,0 +1,19 @@ +import hashlib,json,re,subprocess,sys,time +from pathlib import Path +out=Path(__file__).resolve().parent +mode=sys.argv[1] +base=Path('build-main-7bb5fab')/mode/'bin' +db=Path('.cache/format-tests/main-7bb5fab') +commands=[('btree',[str(base/'unit/btree_test')]),('superblock',[str(base/'unit/superblock_test')]),('splinter',[str(base/'unit/splinter_test'),'--db-location',str(db/(mode+'-splinter.db')),'--cache-capacity-mib','128','--db-capacity-gib','64']),('quick',[str(base/'unit/splinterdb_quick_test')]),('trunk-async',[str(base/'unit/trunk_async_test')]),('filter',[str(base/'driver_test'),'filter_test','--db-location',str(db/(mode+'-filter.db')),'--cache-capacity-mib','64','--db-capacity-gib','4']),('log',[str(base/'driver_test'),'log_test','--db-location',str(db/(mode+'-log.db')),'--cache-capacity-mib','64','--db-capacity-gib','4','--num-inserts','10000'])] +records=[] +for name,cmd in commands: + print(mode,name,flush=True) + log=out/f'{mode}-{name}.log' + record=dict(name=name,command=cmd,started=time.time());records.append(record) + with log.open('w') as f:r=subprocess.run(cmd,stdout=f,stderr=subprocess.STDOUT) + record.update(returncode=r.returncode,finished=time.time(),log_sha256=hashlib.sha256(log.read_bytes()).hexdigest()) + text=log.read_text() + record['summaries']=[line for line in text.splitlines() if 'RESULTS:' in line] + (out/f'validation-{mode}.json').write_text(json.dumps(records,indent=2)+'\n') + if r.returncode:raise SystemExit(r.returncode) + if name not in ['filter','log'] and not re.search(r'RESULTS: [1-9][0-9]* tests',text):raise SystemExit('No tests matched: '+name) diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/validation-debug-4gib.json b/benchmarks/results-20261013/main-7bb5fab/tests/validation-debug-4gib.json new file mode 100644 index 000000000..c6baa29c3 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/validation-debug-4gib.json @@ -0,0 +1,47 @@ +[ + { + "name": "btree", + "command": [ + "build-main-7bb5fab/debug/bin/unit/btree_test" + ], + "started": 1791904624.2629488, + "returncode": 0, + "finished": 1791904624.2786815, + "log_sha256": "cfa2c7b6cb343a72ab59f8bded1acf481a38cc9f921cd5972b82efc6d0273a14", + "summaries": [ + "RESULTS: 11 tests (11 ok, 0 failed, 0 skipped) ran in 14 ms" + ] + }, + { + "name": "superblock", + "command": [ + "build-main-7bb5fab/debug/bin/unit/superblock_test" + ], + "started": 1791904624.2789748, + "returncode": 0, + "finished": 1791904624.3262405, + "log_sha256": "b27e0169de564deb7ca1319a55ecca6b5e4a4b17ad41ad1a29df2b18f8394121", + "summaries": [ + "RESULTS: 16 tests (16 ok, 0 failed, 0 skipped) ran in 46 ms" + ] + }, + { + "name": "splinter", + "command": [ + "build-main-7bb5fab/debug/bin/unit/splinter_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/debug-splinter.db", + "--cache-capacity-mib", + "128", + "--db-capacity-gib", + "4" + ], + "started": 1791904624.326511, + "returncode": 1, + "finished": 1791904709.6916904, + "log_sha256": "19abfb216913e1d89c0d6ef7f7dc7cd0ff4275f8de44e5e2cd7a3ca094b90f1f", + "summaries": [ + "RESULTS: 38 tests (37 ok, 1 failed, 0 skipped) ran in 85363 ms" + ] + } +] diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/validation-debug.json b/benchmarks/results-20261013/main-7bb5fab/tests/validation-debug.json new file mode 100644 index 000000000..122e819fc --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/validation-debug.json @@ -0,0 +1,111 @@ +[ + { + "name": "btree", + "command": [ + "build-main-7bb5fab/debug/bin/unit/btree_test" + ], + "started": 1791904765.9605756, + "returncode": 0, + "finished": 1791904765.9727993, + "log_sha256": "f8ca9a5b8c682cd418b940a5afef55253c9dc165dced3c780c25b4b141e8d05e", + "summaries": [ + "RESULTS: 11 tests (11 ok, 0 failed, 0 skipped) ran in 11 ms" + ] + }, + { + "name": "superblock", + "command": [ + "build-main-7bb5fab/debug/bin/unit/superblock_test" + ], + "started": 1791904765.973101, + "returncode": 0, + "finished": 1791904766.0031965, + "log_sha256": "417133bc50d178e582c1bdcbff4182a07ced3994a81d29eb409f4b713300f7d2", + "summaries": [ + "RESULTS: 16 tests (16 ok, 0 failed, 0 skipped) ran in 28 ms" + ] + }, + { + "name": "splinter", + "command": [ + "build-main-7bb5fab/debug/bin/unit/splinter_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/debug-splinter.db", + "--cache-capacity-mib", + "128", + "--db-capacity-gib", + "64" + ], + "started": 1791904766.0033975, + "returncode": 0, + "finished": 1791904855.3279638, + "log_sha256": "20208a30e6dcf9076de207d0712849e24113084851af3813c3bdbd9d7e492a7f", + "summaries": [ + "RESULTS: 38 tests (38 ok, 0 failed, 0 skipped) ran in 89322 ms" + ] + }, + { + "name": "quick", + "command": [ + "build-main-7bb5fab/debug/bin/unit/splinterdb_quick_test" + ], + "started": 1791904855.328229, + "returncode": 0, + "finished": 1791904865.8075054, + "log_sha256": "2e348ee7750ccc4c95d7c839ff752e2bb23183c605af0d0df0efabcb46033421", + "summaries": [ + "RESULTS: 49 tests (49 ok, 0 failed, 0 skipped) ran in 10478 ms" + ] + }, + { + "name": "trunk-async", + "command": [ + "build-main-7bb5fab/debug/bin/unit/trunk_async_test" + ], + "started": 1791904865.8077319, + "returncode": 0, + "finished": 1791904865.8092134, + "log_sha256": "358786a8840bf748a73c6f3d58cb3eb2bac80a7107a6387bfa11a167fe5265ed", + "summaries": [ + "RESULTS: 2 tests (2 ok, 0 failed, 0 skipped) ran in 0 ms" + ] + }, + { + "name": "filter", + "command": [ + "build-main-7bb5fab/debug/bin/driver_test", + "filter_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/debug-filter.db", + "--cache-capacity-mib", + "64", + "--db-capacity-gib", + "4" + ], + "started": 1791904865.8093722, + "returncode": 0, + "finished": 1791904871.8719559, + "log_sha256": "49259f7e14bc368c5ba8d9410dd23b58d270333eaf6b13736bbd038ca9b1d924", + "summaries": [] + }, + { + "name": "log", + "command": [ + "build-main-7bb5fab/debug/bin/driver_test", + "log_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/debug-log.db", + "--cache-capacity-mib", + "64", + "--db-capacity-gib", + "4", + "--num-inserts", + "10000" + ], + "started": 1791904871.872412, + "returncode": 0, + "finished": 1791904872.9354644, + "log_sha256": "372a0c0e44520c2d299d25bde942a0dfdd13119b77cfa2b5e4cc032e3e4d8095", + "summaries": [] + } +] diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/validation-release-4gib.json b/benchmarks/results-20261013/main-7bb5fab/tests/validation-release-4gib.json new file mode 100644 index 000000000..2d02bcfe6 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/validation-release-4gib.json @@ -0,0 +1,45 @@ +[ + { + "name": "btree", + "command": [ + "build-main-7bb5fab/release/bin/unit/btree_test" + ], + "started": 1791904712.7533197, + "returncode": 0, + "finished": 1791904712.7562423, + "log_sha256": "943adfa860f10e33a67981a9a9192ccba6163c0b6dcc7c3ac4e99accad002c40", + "summaries": [ + "RESULTS: 11 tests (11 ok, 0 failed, 0 skipped) ran in 1 ms" + ] + }, + { + "name": "superblock", + "command": [ + "build-main-7bb5fab/release/bin/unit/superblock_test" + ], + "started": 1791904712.7564511, + "returncode": 0, + "finished": 1791904712.8161988, + "log_sha256": "1f0520268e04debc824b341569ccfc0d217c7e67593dd220645ad3d903d5873d", + "summaries": [ + "RESULTS: 16 tests (16 ok, 0 failed, 0 skipped) ran in 58 ms" + ] + }, + { + "name": "splinter", + "command": [ + "build-main-7bb5fab/release/bin/unit/splinter_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/release-splinter.db", + "--cache-capacity-mib", + "128", + "--db-capacity-gib", + "4" + ], + "started": 1791904712.8164175, + "returncode": -6, + "finished": 1791904747.5774693, + "log_sha256": "40f1c0c881534b1c7584bf5db4e324181858eb9668642ca7d83649627dd82664", + "summaries": [] + } +] diff --git a/benchmarks/results-20261013/main-7bb5fab/tests/validation-release.json b/benchmarks/results-20261013/main-7bb5fab/tests/validation-release.json new file mode 100644 index 000000000..96d498670 --- /dev/null +++ b/benchmarks/results-20261013/main-7bb5fab/tests/validation-release.json @@ -0,0 +1,111 @@ +[ + { + "name": "btree", + "command": [ + "build-main-7bb5fab/release/bin/unit/btree_test" + ], + "started": 1791904789.9906392, + "returncode": 0, + "finished": 1791904789.994565, + "log_sha256": "43c08095c908b960b7c40ce73c84804496b2538fd4e5d1e6d67cf78f14d311f5", + "summaries": [ + "RESULTS: 11 tests (11 ok, 0 failed, 0 skipped) ran in 2 ms" + ] + }, + { + "name": "superblock", + "command": [ + "build-main-7bb5fab/release/bin/unit/superblock_test" + ], + "started": 1791904789.9948416, + "returncode": 0, + "finished": 1791904790.038308, + "log_sha256": "e8f058f14fdc7b2ae6c78a4983c56e5ae3c3cee7cdc4b46a303efa3a18cb2b1c", + "summaries": [ + "RESULTS: 16 tests (16 ok, 0 failed, 0 skipped) ran in 42 ms" + ] + }, + { + "name": "splinter", + "command": [ + "build-main-7bb5fab/release/bin/unit/splinter_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/release-splinter.db", + "--cache-capacity-mib", + "128", + "--db-capacity-gib", + "64" + ], + "started": 1791904790.0385456, + "returncode": 0, + "finished": 1791904826.3436356, + "log_sha256": "64e3e5c3f4c796d2be3661ddbc06e3da816516e1497a2b98aafc2a84a6947de7", + "summaries": [ + "RESULTS: 38 tests (38 ok, 0 failed, 0 skipped) ran in 36303 ms" + ] + }, + { + "name": "quick", + "command": [ + "build-main-7bb5fab/release/bin/unit/splinterdb_quick_test" + ], + "started": 1791904826.3439152, + "returncode": 0, + "finished": 1791904837.463507, + "log_sha256": "fb9f96d901f9d6653e0295bcffe6220ded0a75f71db48ec3a5c763be57a50b0c", + "summaries": [ + "RESULTS: 49 tests (49 ok, 0 failed, 0 skipped) ran in 11118 ms" + ] + }, + { + "name": "trunk-async", + "command": [ + "build-main-7bb5fab/release/bin/unit/trunk_async_test" + ], + "started": 1791904837.4639704, + "returncode": 0, + "finished": 1791904837.465928, + "log_sha256": "58337bde38e57564caa2561b024f9e05bedf6d7c6d6f1b220fee48ef6c622449", + "summaries": [ + "RESULTS: 2 tests (2 ok, 0 failed, 0 skipped) ran in 0 ms" + ] + }, + { + "name": "filter", + "command": [ + "build-main-7bb5fab/release/bin/driver_test", + "filter_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/release-filter.db", + "--cache-capacity-mib", + "64", + "--db-capacity-gib", + "4" + ], + "started": 1791904837.4663038, + "returncode": 0, + "finished": 1791904841.3591342, + "log_sha256": "bcfd592c4f9020413d7751bbf1e69d0de0a8c21e9953b19768e35c17564cd7e5", + "summaries": [] + }, + { + "name": "log", + "command": [ + "build-main-7bb5fab/release/bin/driver_test", + "log_test", + "--db-location", + ".cache/format-tests/main-7bb5fab/release-log.db", + "--cache-capacity-mib", + "64", + "--db-capacity-gib", + "4", + "--num-inserts", + "10000" + ], + "started": 1791904841.3596354, + "returncode": 0, + "finished": 1791904842.011724, + "log_sha256": "94d99f7647332b2b2a3310da728d30fddc459ed2c722060f5d0fc1bc4003e55e", + "summaries": [] + } +] diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md index b0df0b135..6cebcfa68 100644 --- a/docs/on-disk-format.md +++ b/docs/on-disk-format.md @@ -206,3 +206,13 @@ Commands, logs, source and executable hashes are retained in the main checkout's `.cache/projects-20261011/coordination/main-7bb5fab/disk-format/` directory. The measurements above predate this integration; they do not establish current performance neutrality. + +## Fresh benchmarks after main integration (2026-10-13) + +Resident point throughput changed -1.21%/-1.92% at one/eight threads, and disk point throughput changed -0.81%. These sample ranges overlap main; substantial control drift prevents precise attribution of small differences. + +These are updated-feature comparisons against main `7bb5fab`, not an isolated +measurement of merge cost. Main controls drifted; observed range overlap is +descriptive, not a statistical confidence interval. Exact samples, build/source +hashes, commands, verification scope and fresh correctness evidence are in +[the integration archive](../benchmarks/results-20261013/main-7bb5fab/README.md). From 7a0c817165692b7af3cbec94ad34d4e8586841e1 Mon Sep 17 00:00:00 2001 From: Rob Johnson Date: Tue, 13 Oct 2026 13:31:17 -0700 Subject: [PATCH 7/7] Prepare disk-format review with current-main validation and performance limits --- .../disk-format-pr-0fa3bd6c/README.md | 5 +- .../disk-format-pr-0fa3bd6c/inventory.json | 2 +- .../.clang-format | 3 + .../.gitattributes | 2 + .../predicate-disk-main-0fa3bd6c/README.txt | 9 + .../predicate-disk-main-0fa3bd6c/build.py | 38 + .../control-drift.json | 122 + .../disk-format-build.json | 159 + .../disk-format-kvbench-build.log | 2 + .../disk-format-libs.log | 1 + .../disk/00-main.log | 21 + .../disk/01-filtered-ranges.log | 21 + .../disk/02-disk-format.log | 21 + .../disk/03-disk-format.log | 21 + .../disk/04-filtered-ranges.log | 21 + .../disk/05-main.log | 21 + .../disk/manifest.json | 480 +++ .../disk/results.csv | 25 + .../collisions0-00-main-application.log | 6 + ...llisions0-01-filtered-ranges-predicate.log | 6 + .../collisions0-02-filtered-ranges-pruned.log | 6 + .../collisions0-03-filtered-ranges-pruned.log | 6 + ...llisions0-04-filtered-ranges-predicate.log | 6 + .../collisions0-05-main-application.log | 6 + .../collisions1-00-main-application.log | 6 + ...llisions1-01-filtered-ranges-predicate.log | 6 + .../collisions1-02-filtered-ranges-pruned.log | 6 + .../collisions1-03-filtered-ranges-pruned.log | 6 + ...llisions1-04-filtered-ranges-predicate.log | 6 + .../collisions1-05-main-application.log | 6 + .../feature/manifest.json | 1167 ++++++ .../filtered-ranges-build.json | 195 + ...ered-ranges-hash_range_benchmark-build.log | 2 + .../filtered-ranges-kvbench-build.log | 2 + .../filtered-ranges-libs.log | 2 + .../final-provenance-check.json | 6 + .../fixtures/disk-disk-format.csv | 2 + .../fixtures/disk-disk-format.log | 7 + .../fixtures/disk-main.csv | 2 + .../fixtures/disk-main.log | 7 + .../fixtures/fixtures.json | 26 + .../fixtures/manifest.json | 290 ++ .../fixtures/resident-disk-format.csv | 2 + .../fixtures/resident-disk-format.log | 7 + .../fixtures/resident-main.csv | 2 + .../fixtures/resident-main.log | 7 + .../predicate-disk-main-0fa3bd6c/host.json | 11 + .../inventory.json | 310 ++ .../library-build-configs.json | 23 + .../main-build.json | 194 + .../main-hash_range_benchmark-build.log | 2 + .../main-kvbench-build.log | 2 + .../main-libs.log | 1 + .../predicate-disk-main-0fa3bd6c/package.py | 32 + .../profiles.json | 17 + .../00-disk-format.log | 12 + .../repeat-resident-disk-point8/01-main.log | 12 + .../repeat-resident-disk-point8/02-main.log | 12 + .../03-disk-format.log | 12 + .../repeat-resident-disk-point8/manifest.json | 319 ++ .../repeat-resident-disk-point8/results.csv | 9 + .../repeat_point.py | 26 + .../report-generation.log | 78 + .../predicate-disk-main-0fa3bd6c/report.py | 113 + .../predicate-disk-main-0fa3bd6c/report.txt | 78 + .../resident-summary.json | 114 + .../resident/00-main.log | 26 + .../resident/01-filtered-ranges.log | 26 + .../resident/02-disk-format.log | 26 + .../resident/03-disk-format.log | 26 + .../resident/04-filtered-ranges.log | 26 + .../resident/05-main.log | 26 + .../resident/manifest.json | 487 +++ .../resident/results.csv | 31 + .../predicate-disk-main-0fa3bd6c/run.py | 151 + .../sources/hash_range_benchmark.c | 262 ++ .../sources/kvbench.c | 1707 ++++++++ .../predicate-disk-main-0fa3bd6c/status.json | 13 + .../predicate-disk-main-0fa3bd6c/summary.json | 3493 +++++++++++++++++ .../timing-ready.json | 10 + docs/disk-format-pr.md | 11 + docs/on-disk-format.md | 55 +- 82 files changed, 10519 insertions(+), 6 deletions(-) create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.clang-format create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.gitattributes create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/README.txt create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/build.py create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/control-drift.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-build.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-kvbench-build.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-libs.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/00-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/01-filtered-ranges.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/02-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/03-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/04-filtered-ranges.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/05-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/manifest.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/results.csv create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-00-main-application.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-01-filtered-ranges-predicate.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-02-filtered-ranges-pruned.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-03-filtered-ranges-pruned.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-04-filtered-ranges-predicate.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-05-main-application.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-00-main-application.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-01-filtered-ranges-predicate.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-02-filtered-ranges-pruned.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-03-filtered-ranges-pruned.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-04-filtered-ranges-predicate.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-05-main-application.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/manifest.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-build.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-hash_range_benchmark-build.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-kvbench-build.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-libs.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/final-provenance-check.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.csv create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.csv create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/fixtures.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/manifest.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.csv create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.csv create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/host.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/inventory.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/library-build-configs.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-build.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-hash_range_benchmark-build.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-kvbench-build.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-libs.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/package.py create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/profiles.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/00-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/01-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/02-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/03-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/manifest.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/results.csv create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat_point.py create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report-generation.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.py create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.txt create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident-summary.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/00-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/01-filtered-ranges.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/02-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/03-disk-format.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/04-filtered-ranges.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/05-main.log create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/manifest.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/results.csv create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/run.py create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/hash_range_benchmark.c create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/kvbench.c create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/status.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/summary.json create mode 100644 benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/timing-ready.json create mode 100644 docs/disk-format-pr.md diff --git a/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/README.md b/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/README.md index 4dcc73a04..b578c0355 100644 --- a/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/README.md +++ b/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/README.md @@ -7,8 +7,9 @@ checkpoint and notification paths remain intact. Debug and release each passed 150 unit cases, with two expected skips of the allocation-failure case because these builds disable allocation fault injection. -Both also passed the functional routing-filter suite and the log suite with 10,000 -inserts. The initially strict validation runner stopped at that expected skip; +Both also passed the functional routing-filter suite and the complete log suite +(500,000-record round trip). The invocation requested 10,000 inserts, but this +functional driver hardcodes 500,000; raw logs confirm the actual count. The initially strict validation runner stopped at that expected skip; its log and original runner are retained. After recognizing only that named skip, the debug run resumed with the remaining suites; no database case failed. diff --git a/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/inventory.json b/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/inventory.json index de918928c..95355ccba 100644 --- a/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/inventory.json +++ b/benchmarks/results-20261013/disk-format-pr-0fa3bd6c/inventory.json @@ -11,7 +11,7 @@ "tests-debug.log": "da6adcf9e85029bbcd4258991af346434258e2de0991a47b38da81fe0d09a64e", "debug-inspect.log": "dd1820c2e40bb3dca65f6123121d9bec4bf946aa5ba247d5b95927a7aece6542", "debug-quick.log": "f1684a10b1685d3d25ad911be476047ef0e7ce7d189ca9ed2da754422d496d23", - "README.md": "1739cafffd945adcfaa05614cb8af0c33d285db6dab5dbab2f1b36b9ea057a67", + "README.md": "d0cb149d9223f08562fe0759d0952d2e3dec22928e79093343af9587a7083591", "validation-debug.json": "54578979090760f0c188d45671e1f6e8dc4c79ce5a644c7d2c0e9f2fdb5eae7f", "reject-format7.log": "c8d0972c1189aa5bd82b209db35b7a9f1d1f8d82c96f8114eb03f6f7b9964f3d", "debug-filter.log": "d7369d01bde3fd47424d5844da5c5a6264838e045dc4a97b78a47019e6ab6603", diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.clang-format b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.clang-format new file mode 100644 index 000000000..7bddb1b04 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.clang-format @@ -0,0 +1,3 @@ +# Preserve immutable measured source snapshots. +DisableFormat: true +SortIncludes: Never diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.gitattributes b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.gitattributes new file mode 100644 index 000000000..ef52682c2 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/.gitattributes @@ -0,0 +1,2 @@ +*.log -whitespace +*.txt -whitespace diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/README.txt b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/README.txt new file mode 100644 index 000000000..acd47fb8c --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/README.txt @@ -0,0 +1,9 @@ +Fresh filtered-predicate and disk-format screening against main 0fa3bd6c. +OPEN PERFORMANCE CONCERN: disk-format resident point8 throughput fell 4.39% in the broad screen and 7.75% in the longer repeat, with non-overlapping sample ranges and zero disk reads. +The cause is unresolved. Code review is appropriate; performance-neutrality and merge-ready claims are not supported. The broad and repeat samples remain separate. +See report.txt for methodology, sample ranges and limits; summary.json for CPU/I/O and query counts. +Each profile build JSON records source, library, executable hashes and compile commands. +Each workload manifest records exact invocation, storage, verification and process resource usage. +Sources are frozen measured bytes. Historical absolute paths identify the original workspace. +Databases, libraries and executables are intentionally not embedded. Fixture creation logs and +digests are retained; every copied fixture was checked and fsynced before use. diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/build.py b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/build.py new file mode 100644 index 000000000..90f018bd0 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/build.py @@ -0,0 +1,38 @@ +"""Run via run_job build after profiles.json pins the final source heads.""" +import hashlib, json, subprocess, sys +from pathlib import Path +HERE = Path(__file__).resolve().parent + +def sha(p): return hashlib.sha256(Path(p).read_bytes()).hexdigest() +def save(p, value): p.write_text(json.dumps(value, indent=2) + '\n') +profiles = json.loads((HERE/'profiles.json').read_text()) +flags = ['cc', '-D_GNU_SOURCE', '-ggdb', '-Wall', '-pthread', '-Werror', + '-DXXH_STATIC_LINKING_ONLY', '-fPIC', '-DSPLINTERDB_PLATFORM_DIR=platform_linux', + '-O3', '-ffast-math', '-flto', '-march=native'] +for name, config in profiles.items(): + if len(sys.argv)>1 and name not in sys.argv[1:]: continue + tree = Path(config['tree']); head = config['head'] + assert subprocess.check_output(['git','rev-parse','HEAD'],cwd=tree,text=True).strip()==head + assert subprocess.run(['git','rev-parse','--verify','-q','MERGE_HEAD'],cwd=tree,stdout=subprocess.DEVNULL).returncode!=0 + subprocess.run(['git','diff','--quiet','HEAD','--','src','include'],cwd=tree,check=True) + command=['make','--jobserver-style=pipe','-j4','BUILD_ROOT='+config['build_root'],'BUILD_MODE=release','libs'] + with (HERE/(name+'-libs.log')).open('w') as log: + subprocess.run(command,cwd=tree,stdout=log,stderr=subprocess.STDOUT,check=True) + lib=tree/config['build_root']/'release/lib/libsplinterdb.a' + record=dict(tree=str(tree),head=head,library=str(lib),library_sha256=sha(lib),library_build_command=command, + source_files_sha256={},binaries={}) + for source in subprocess.check_output(['git','ls-files','src','include'],cwd=tree,text=True).splitlines(): + record['source_files_sha256'][source]=sha(tree/source) + for binary in ['kvbench']+(['hash_range_benchmark'] if name!='disk-format' else []): + source=HERE/'sources'/(binary+'.c'); destination=HERE/'bin'/name/binary + destination.parent.mkdir(parents=True,exist_ok=True) + defines=['-DKVBENCH_CLOSE_HAS_FORCE'] if binary=='kvbench' else [] + if binary=='hash_range_benchmark' and name=='filtered-ranges': defines+=['-DPREDICATE_RANGE_FILTER_API'] + command=flags+defines+['-I',str(tree/'include'),'-I',str(tree/'src'),'-I',str(tree/'src/platform_linux'), + str(source),str(lib),'-laio','-lxxhash','-lm','-lpthread','-o',str(destination)] + with (HERE/(name+'-'+binary+'-build.log')).open('w') as log: + subprocess.run(command,stdout=log,stderr=subprocess.STDOUT,check=True) + record['binaries'][binary]=dict(command=command,binary=str(destination),binary_sha256=sha(destination), + source=str(source),source_sha256=sha(source)) + save(HERE/(name+'-build.json'),record) + print(name,head,lib,record['library_sha256'],flush=True) diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/control-drift.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/control-drift.json new file mode 100644 index 000000000..ec70e847c --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/control-drift.json @@ -0,0 +1,122 @@ +[ + { + "regime": "resident", + "phase": "point:uniform@1", + "samples": { + "mean": 690741.25, + "min": 688766.2, + "max": 692716.3, + "samples": 2, + "values": [ + 692716.3, + 688766.2 + ] + }, + "last_vs_first_pct": -0.5702334418866806 + }, + { + "regime": "resident", + "phase": "point:uniform@8", + "samples": { + "mean": 3755201.6, + "min": 3747424.5, + "max": 3762978.7, + "samples": 2, + "values": [ + 3747424.5, + 3762978.7 + ] + }, + "last_vs_first_pct": 0.41506373243811456 + }, + { + "regime": "resident", + "phase": "range:uniform:1@8", + "samples": { + "mean": 369257.55, + "min": 363033.5, + "max": 375481.6, + "samples": 2, + "values": [ + 363033.5, + 375481.6 + ] + }, + "last_vs_first_pct": 3.428912207826551 + }, + { + "regime": "resident", + "phase": "range:uniform:10000@8", + "samples": { + "mean": 2913.0, + "min": 2894.8, + "max": 2931.2, + "samples": 2, + "values": [ + 2894.8, + 2931.2 + ] + }, + "last_vs_first_pct": 1.2574271106812196 + }, + { + "regime": "disk", + "phase": "point:uniform@8", + "samples": { + "mean": 142173.84999999998, + "min": 140322.8, + "max": 144024.9, + "samples": 2, + "values": [ + 144024.9, + 140322.8 + ] + }, + "last_vs_first_pct": -2.570458302696277 + }, + { + "regime": "disk", + "phase": "range:uniform:1@8", + "samples": { + "mean": 61259.05, + "min": 60885.8, + "max": 61632.3, + "samples": 2, + "values": [ + 61632.3, + 60885.8 + ] + }, + "last_vs_first_pct": -1.211215547691713 + }, + { + "regime": "disk", + "phase": "range:uniform:10000@8", + "samples": { + "mean": 742.45, + "min": 741.3, + "max": 743.6, + "samples": 2, + "values": [ + 741.3, + 743.6 + ] + }, + "last_vs_first_pct": 0.3102657493592398 + }, + { + "regime": "repeat-resident-disk-point8", + "phase": "point:uniform@8", + "samples": { + "mean": 4070674.25, + "min": 4026197.4, + "max": 4115151.1, + "samples": 2, + "values": [ + 4026197.4, + 4115151.1 + ] + }, + "last_vs_first_pct": 2.2093725459163993 + } +] diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-build.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-build.json new file mode 100644 index 000000000..4f56eb153 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-build.json @@ -0,0 +1,159 @@ +{ + "tree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format", + "head": "3e302c04c60babb8afecb58d27f808848624f83c", + "library": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-pr-0fa3bd6c/release/lib/libsplinterdb.a", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "library_build_command": [ + "make", + "--jobserver-style=pipe", + "-j4", + "BUILD_ROOT=build-pr-0fa3bd6c", + "BUILD_MODE=release", + "libs" + ], + "source_files_sha256": { + "include/splinterdb/data.h": "57be61933a057f52f679dfcf21755a8645e6228cd1a7cdee983f5d57768923ab", + "include/splinterdb/default_data_config.h": "179386f3e224d322a9d7ac97afd929292994a1d890a3254705f9ebea525bce32", + "include/splinterdb/platform_linux/public_platform.h": "c5ded82c5b5bb5bbcedf06a66951eb98dc8892f7b3c4f8ce07c474db98ac68a0", + "include/splinterdb/public_platform.h": "4ce9da3fea9a54fb7bb613ec1458f11f043525efb7d0bc55182f7183dd4a4f6e", + "include/splinterdb/public_util.h": "4e8f6b30408fb4174894d4ba809acff4e96f559370c7749b65bb3724a853603f", + "include/splinterdb/splinterdb.h": "9266891f702fc4f0855b26e08d73240127bf64aa1b19913f73f2736bfc486969", + "src/PackedArray.c": "97b64eb0a4f550e55f3385d39d729a878fbc581909ea16ec3b8987a7f46604cc", + "src/PackedArray.h": "e46f854b487a8b130f0613efcab11836012d27c8aacf9ca9e76a2de1a7eb6874", + "src/allocator.c": "96c7f0a5e0808719cf114d267c84da1f057b5eed748b0672d60686d7f130b96f", + "src/allocator.h": "06da1d6a491a15ef5229e8e2e4b4ad6468cb3c9ba620e0d956032dc0a1b6aec1", + "src/batch_rwlock.c": "c043d9e5ead75cd9e1266a6f51fb555012269f3822d7f0233dc2abbd962c3aae", + "src/batch_rwlock.h": "0ff5f0bd7fdf5f9a63ca2f77c387d574d8009ad726b47283c93566cf06bc0927", + "src/blob.c": "75e01e92c06d775b1cb36ea63a7cddf191b5179b94c3e9cd6b5b23ab0480485b", + "src/blob.h": "8f68d2f67de8a9d81a422b28d47aa636cc7234fccc3dbabe59f2355bc361e93c", + "src/blob_build.c": "e484e9c14fd8a9b3c1d211485c35931c944fc66ac4ee78ea20ffc5b3f074b10b", + "src/blob_build.h": "38176881cf9159247e6f578bbc7abb725cfa969c493715df7d1388cb7cbfd119", + "src/btree.c": "8b059dfa240564a7fca21379b467afc7635b9128d2ca72555300333ae06dbc97", + "src/btree.h": "2dc405429d4d84f606412d3a16dc07dd2aee2877524d658e946cb1a8bf1d865e", + "src/btree_private.h": "43693dd3048ae050b3358ccfc9b91371a3d9dff4f3d6d5c8b07445897dee09f6", + "src/cache.h": "d4c1e133d6423fff324d0ebcf5c95e13647c0022a3458935156746b90d816d6d", + "src/clockcache.c": "e0ef2e19a8280849df4ee80d9b8a54a0faaf56a13faefe7a91d54f5b7269f933", + "src/clockcache.h": "68f4e7b019d0440f85929bfc0da1891ce941318ee7dfd14a3d4d923e3d4e72d8", + "src/core.c": "71fcda2ce6f6b084a72567a56774096b62f2c2ab689648161c69498f2c6bf93a", + "src/core.h": "1e5b6eb93e004320987698f9b71980f7e1876626de7db85411c972f9fb482784", + "src/data_blob_build.c": "13475faa56bba94ee26ef7fec4e4da8a5045e0efe510a63291fd5e2697cb5d22", + "src/data_blob_build.h": "dd7f263a0b1138325b211c561ba7289183a2cc8472978cba3ca430295e11e5f3", + "src/data_internal.c": "0a42dd477aaa6ad6c4910841045c4c768ad8d248c0126c64c51035f53e769cde", + "src/data_internal.h": "98b4ebe77576c4cbb7dd9984ca5e297851175494062073f1a4c137c5afb8b587", + "src/default_data_config.c": "161352f5247978f3d57033818d05a79d0a1a0d0ffac2bc9f25f4f33eabca2227", + "src/histogram.c": "24e09dfabd6085021d8a06ef9b10f6e85b24914d95831be359df09f8a9729aff", + "src/histogram.h": "7ccfde6d3ada7ead18fe477c06a6b021db83ca8c041ab0998ca023b945a3c1c7", + "src/iterator.h": "2fcddeb9fa6bd49f4e3da933224219bdb830f9e2f11638ea1ed13eba534fa27f", + "src/log.h": "3e7eb447259e7901b915b52b7545e2d5088475171af1716dd9c849dab4c3d1b9", + "src/log_data.h": "8b9198c49c95367f5c66cfa23ec1a3b6cab2abd875fe74c4afb4dafb5a5c44fe", + "src/lookup_result.h": "7528b988a28ef91b6bd7a4fa10dfa51b9ce2475e801b3b6a8bf30adf88f1ef16", + "src/memtable.c": "7db3b4754497c73b4071c7e7e591cc76fc226197fb70d98fb46caf139b3953f4", + "src/memtable.h": "244cac7cf304df85a25c8bede2df106d6f840d65bbd1a3551b3d9f2ed9129fc1", + "src/merge.c": "df98421e597d1edeac871755f0473055b590791af8efad495575d20260e46a12", + "src/merge.h": "c17e2a0b6272ec52e6d1e80acd9df7dc85221f186e6c60e323c37b467dc5c41a", + "src/mini_allocator.c": "d620cbdff9867ce64c752992ede58783010fb09366220ad3bd6d8518911205ca", + "src/mini_allocator.h": "39d245b58914d28b26dfe230b9de15582382cec1c2a90fa14db19f5f2c203632", + "src/notification.c": "54af1fe6af2e5fc84fbee21a74ff55069a8aa7e4ec7f9d8c9485e85e91669b58", + "src/notification.h": "f4df33131f407e6a957602a05c72f4bca7efed98b1a39134f94bf889a6ca5f29", + "src/ondisk_ref.c": "2ace8cbbce3f80a54ff7762ad03a79a564665fa660394f1d3446235a3e40ed7b", + "src/ondisk_ref.h": "643325c8b4f16fab41d8c884c97bdfcf6c86e5178398508fa0f81b6b0606926d", + "src/pcq.h": "862cd346b0097014ccae797dc1463e7b3fc387f409319d5adcef7613b6f82f84", + "src/platform_linux/async.h": "41383b57e21f0f51d737749a4ce7c2c5330ac9363e742330625c9aedd4f13a52", + "src/platform_linux/laio.c": "0ce628806e6896070359a9e39d069f326f460ab697c6c17ff17d171614290881", + "src/platform_linux/laio.h": "f3987e22f1fd7a9d6b46c79f84e75559aec5410239a13d52a7a5343ba792b28e", + "src/platform_linux/platform.h": "f1c3d7c5bcd089ed7a7df129d3b8d00d7617de9a7aa6f9d1f7deeb739281e768", + "src/platform_linux/platform_assert.c": "112e803b6db03dd8c14a3bfc3f1139bbcc4b81f1d5e24a232b7b99ce89146c2d", + "src/platform_linux/platform_assert.h": "751dccceac9ce4a97d25d441b5216ced5ee5f079a13c9ccd43551b0994aa7cc2", + "src/platform_linux/platform_backtrace.h": "ef13ccbf4ae4747b0de244a37c026ad8769018d84f9fc4d6493db43d9714358c", + "src/platform_linux/platform_buffer.c": "b790d645b3dc5f638da871bf9611865dc700927f3b594a1d645b0350bd1bd027", + "src/platform_linux/platform_buffer.h": "72e016dbb19b8250378b3a5c79fbebf9c3d72a31d07a232f36431e04f0e4afa6", + "src/platform_linux/platform_condvar.c": "e595f69194accb3f3460d6fc9a4a9144bd7f495dbeb77f11bf3616dc0e9b03ec", + "src/platform_linux/platform_condvar.h": "a4ee02ae5069637bce4cae07f3ed9869925f17a9e6e1b04d69baedfc148fd493", + "src/platform_linux/platform_hash.h": "f69f5ed6218bfe62c62718b387e1c110e9fca46c07603d4d51ccc4fc782b09ab", + "src/platform_linux/platform_heap.c": "72bb2753f525e56f8c398988a787feb38f02efaa749e5d57cb476f967269c479", + "src/platform_linux/platform_heap.h": "921f3113d20b44c36817c284702e965e01769596ff012edd6ceaa70cbdbbd10a", + "src/platform_linux/platform_io.c": "875a7331f8bfa46e9a6961f50acb40b107d40883839c94def52a289a521c66be", + "src/platform_linux/platform_io.h": "69ae9556daf1c7a69e007abd494bf9c86ec81c08bca1e032663009cef95b2fac", + "src/platform_linux/platform_log.c": "318bd17c858475a3a91c5e9641b52808f7c4194e1c6fc9a7d801392b04c49538", + "src/platform_linux/platform_log.h": "569484386150fcfbf5dde722c5b8e34d67d65be3454e3e009cb9df41f11efea6", + "src/platform_linux/platform_machine.h": "c02745313c7268dbe99efa2b93dd4d37640b5a7aa2e88c78fd5b73742f252c7d", + "src/platform_linux/platform_mutex.c": "7ec1dd6a5397217d28fd85c10648d29a70bf66649f59f76d7387236031b15bf3", + "src/platform_linux/platform_mutex.h": "688991d6a7dc57a7a88f565541561344cdfbb1733ca6073f1655d6c436ed6544", + "src/platform_linux/platform_random.h": "184ac9da9c38a6c10275891be61ccb91beaf2234ebb9ce046d059cb16824d68f", + "src/platform_linux/platform_semaphore.h": "9fdb7cfeaf36c8bf9aa346a16466ed6bc50bea76fdb5722c86001563d55bedfe", + "src/platform_linux/platform_sleep.h": "34aeaea35133bf33e47eb1d9ace429297644e13bc100d90ebbd205bceb58d3ab", + "src/platform_linux/platform_sort.h": "d374788510f94a8c3b86ff6bfa1f6a3ff8ac33168eb26f6f7774984252f6d01e", + "src/platform_linux/platform_spinlock.h": "7b3f0409109f3341f2287fe7cd8fb69faac4a74b009a44c1eb83220f71e94de0", + "src/platform_linux/platform_status.h": "2749285ea1322e5c1639019b728fde4b87b0cae6f71a7fc16e420bd592d38c00", + "src/platform_linux/platform_string.h": "fa41faf9c6661c62ece783740250232709656b0a8677f542e390a5f3b0d834a6", + "src/platform_linux/platform_threads.c": "789465a2c6ac7ac501e5878f429cc0dd9b163145a522e247a4833e3de7af8f2a", + "src/platform_linux/platform_threads.h": "653e070115131756e6aaffc3c2764a6437fb39ff96df4afc8a01e6e751c9d622", + "src/platform_linux/platform_time.h": "f212a08cf8432693711cecfd389f9ed3d0a70e0c056dc062321bd11c4a8cbec7", + "src/platform_linux/platform_trace.c": "4a68e89f214920dd2762d072a0ad8e7c80d9b1733a64e4a7e3dd4bb04b91c352", + "src/platform_linux/platform_trace.h": "345f7283b01e5a9b4d5da7bea66ad44ff61f53155417c97990e23bb8fedadf33", + "src/platform_linux/platform_typed_alloc.h": "42c1d2bbd56c183d11768a5f43d440cf3c1bfe329c4352afa6acd3bc2b191a8e", + "src/platform_linux/platform_units.h": "1cb22a07b33ec8b58f9381438f480f561d12b838f315a117cc1f979e95b1ef65", + "src/platform_linux/platform_util.h": "4fa2702b3919eb1a7efcf29457f5ee9a3d7a7b4c38fed4606264562022fa2638", + "src/platform_linux/poison.h": "a32dfa445439eb08913e519117e7b11f72b2c3550577a74769ed935a792c42c5", + "src/platform_linux/shmalloc.c": "e4631d63a9db1ad58472ad3d034d081f2bb9a97f34c5898f7c848dda923727f8", + "src/platform_linux/shmalloc.h": "081a82473f6021f8ed628e79f8ae5dc7104988981f8c3b25749ea42c8158113d", + "src/prefetch.h": "9d55d9dad64e3795c81856bc38688936e4e74e488dfe2d29a1f5438ad000df42", + "src/rc_allocator.c": "23ef0da6fb2d5114fe403403c8e535bf8ba19ed2c8bef4397212933cedfc8b9e", + "src/rc_allocator.h": "5f1fae7e32207a4e0cd5c4f799c9b802ce116dd381ee5c074c21d64a229dc3ad", + "src/routing_filter.c": "a913ee83ca39267b6da882bae0db05b1453f39c863f31ec6fea9844998f72f03", + "src/routing_filter.h": "3104de8135817e33b585ba489bc4821a458f6a9c2ce9e561ea8be2fde28e43ba", + "src/shard_log.c": "b601c29a5dbcbf5ad5c0859ca87a3e9dc2088a10cad77327ee4f1c7f46c02f91", + "src/shard_log.h": "bde00a5cbe72e72ac7133006796e61f046ae758e8e4667aa610e184ed6ef4fd7", + "src/splinterdb.c": "aae08a23d4ae9d93f93bc7d35ea85972c45127ccee6d3caeb74b8573da2dd56b", + "src/splinterdb_tests_private.h": "579031767b8869f49cf1e3fc94caeec13a1362dfd047782538df40b281c93414", + "src/superblock.c": "e3fbe387ac4fef197c11977e35e3aecf0d31c27c9d4932e4ec54b1bfc391620a", + "src/superblock.h": "cec63cb28426e2d23140b3ac310a86ae9635d6b3ec429abe7c469bc0de06aa73", + "src/task.c": "470418c0da14795c4697c51ef7b22e128b691c1c24a704791eea09a10b9373a4", + "src/task.h": "ebcc3bd29cef9ee8718c6ff0186863fff0dad03bcfaa43e5eacaf83c1933d01d", + "src/trunk.c": "cff7a5eef6945c36b7784c888bdf6ab2510d132d87d963ba2c909322c13faf36", + "src/trunk.h": "5d8674472258da1be0a84e20a20869c84dc846254d7604e5799297491e6f91e1", + "src/util.c": "969df408c4c393f73feb05dfa37ebe1ed8be627cef630061830e4992c4c1fd9c", + "src/util.h": "9fa455f5a6268fa4a6864d444849b0872391af00512932c1587363cfd38304ef", + "src/vector.h": "6855dd0430f1a868d098797cbd4f3453ab542b5f6fbe716c79871124bee200a9", + "src/writeback_set.c": "cb485716f57a3183ce82543848edaf61f6568c9ff0445ea5ff989f95c00848ff", + "src/writeback_set.h": "68d6f589d24719a09e2680b05a1eb7e36c22b1fab978768f5b54f958f667aec9" + }, + "binaries": { + "kvbench": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/include", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/src", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/kvbench.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-pr-0fa3bd6c/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/kvbench.c", + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9" + } + } +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-kvbench-build.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-kvbench-build.log new file mode 100644 index 000000000..91b66fe35 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-kvbench-build.log @@ -0,0 +1,2 @@ +lto-wrapper: warning: using serial compilation of 13 LTRANS jobs +lto-wrapper: note: see the ‘-flto’ option documentation for more information diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-libs.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-libs.log new file mode 100644 index 000000000..8548d79e3 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk-format-libs.log @@ -0,0 +1 @@ +Checking for mismatched config...No mismatched config found diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/00-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/00-main.log new file mode 100644 index 000000000..e7bb1ff5b --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/00-main.log @@ -0,0 +1,21 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=0 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.17 ops/s= 11942936 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.86s ops/cpu-s=2312465 rd=146.9 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=864198 notfound=0 +== point:uniform@8 T=8 ops=864198 secs= 6.00 ops/s= 144025 lat us p50=65.02 p99=101.38 p99.9=182.27 max=4064.6 mean=55.45 + cpu=12.54s ops/cpu-s=68915 rd=3228.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 238181 ops (+23816 ops/s) +== range:uniform:1@8 T=8 ops=369812 secs= 6.00 ops/s= 61632 lat us p50=125.95 p99=251.90 p99.9=339.97 max=4916.9 mean=129.69 + cpu=51.15s ops/cpu-s=7230 rd=19823.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=123.90 p99=251.90 mean=128.37 per-tuple=nan ns deinit mean=1.25 us tuples/s=61632 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 5025 ops (+1844542043771687424 ops/s) +close took 0.087 s +== range:uniform:10000@8 T=8 ops=4448 secs= 6.00 ops/s= 741 lat us p50=11141.12 p99=23330.82 p99.9=27525.12 max=29814.3 mean=10791.46 + cpu=24.19s ops/cpu-s=184 rd=2364085.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=274.43 p99=827.39 mean=305.93 per-tuple=1050.0 ns deinit mean=2.33 us tuples/s=7400527 + insert/run=6.00s drain(close)=0.09s total=6.09s drain_cpu=0.02s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=474714112 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/01-filtered-ranges.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/01-filtered-ranges.log new file mode 100644 index 000000000..f5c702927 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/01-filtered-ranges.log @@ -0,0 +1,21 @@ +kvbench version=filtered-ranges splinter=splinterdb_build_version cf7dc4f7 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=1 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.15 ops/s= 13687587 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.75s ops/cpu-s=2652949 rd=146.5 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=5 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=856638 notfound=0 +== point:uniform@8 T=8 ops=856638 secs= 6.00 ops/s= 142765 lat us p50=66.56 p99=101.38 p99.9=178.18 max=4127.8 mean=55.93 + cpu=12.67s ops/cpu-s=67612 rd=3228.3 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 231278 ops (+23126 ops/s) +== range:uniform:1@8 T=8 ops=372541 secs= 6.00 ops/s= 62085 lat us p50=123.90 p99=256.00 p99.9=446.46 max=6488.1 mean=128.74 + cpu=51.00s ops/cpu-s=7305 rd=19827.1 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=121.86 p99=256.00 mean=127.47 per-tuple=nan ns deinit mean=1.20 us tuples/s=62085 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 5037 ops (+1844542529958210560 ops/s) +close took 0.082 s +== range:uniform:10000@8 T=8 ops=4462 secs= 6.00 ops/s= 744 lat us p50=11141.12 p99=23855.10 p99.9=28049.41 max=29078.3 mean=10757.87 + cpu=23.71s ops/cpu-s=188 rd=2363148.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=274.43 p99=827.39 mean=306.26 per-tuple=1046.8 ns deinit mean=2.40 us tuples/s=7423944 + insert/run=6.00s drain(close)=0.08s total=6.08s drain_cpu=0.01s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=474714112 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/02-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/02-disk-format.log new file mode 100644 index 000000000..ee2a894d8 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/02-disk-format.log @@ -0,0 +1,21 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=2 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.16 ops/s= 12481186 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.86s ops/cpu-s=2314108 rd=146.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=1 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=939819 notfound=0 +== point:uniform@8 T=8 ops=939819 secs= 6.00 ops/s= 156630 lat us p50=58.88 p99=99.33 p99.9=198.66 max=4860.6 mean=51.00 + cpu=10.86s ops/cpu-s=86522 rd=3229.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 237640 ops (+23762 ops/s) +== range:uniform:1@8 T=8 ops=369913 secs= 6.00 ops/s= 61649 lat us p50=123.90 p99=260.10 p99.9=462.85 max=16127.7 mean=129.65 + cpu=50.92s ops/cpu-s=7265 rd=19802.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=123.90 p99=260.10 mean=128.36 per-tuple=nan ns deinit mean=1.23 us tuples/s=61649 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 5100 ops (+1844537381669098240 ops/s) +close took 0.049 s +== range:uniform:10000@8 T=8 ops=4468 secs= 6.00 ops/s= 745 lat us p50=11141.12 p99=23855.10 p99.9=29097.98 max=35568.4 mean=10746.20 + cpu=23.55s ops/cpu-s=190 rd=2368844.8 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=274.43 p99=827.39 mean=304.89 per-tuple=1045.7 ns deinit mean=2.20 us tuples/s=7433949 + insert/run=6.00s drain(close)=0.05s total=6.05s drain_cpu=0.01s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=475119616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/03-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/03-disk-format.log new file mode 100644 index 000000000..1c19c651a --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/03-disk-format.log @@ -0,0 +1,21 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=3 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.16 ops/s= 12786932 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.79s ops/cpu-s=2538094 rd=146.6 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=849415 notfound=0 +== point:uniform@8 T=8 ops=849415 secs= 6.00 ops/s= 141560 lat us p50=66.56 p99=101.38 p99.9=182.27 max=3575.4 mean=56.41 + cpu=12.83s ops/cpu-s=66222 rd=3229.1 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 240714 ops (+24070 ops/s) +== range:uniform:1@8 T=8 ops=375631 secs= 6.00 ops/s= 62601 lat us p50=123.90 p99=243.71 p99.9=331.78 max=4268.2 mean=127.68 + cpu=51.15s ops/cpu-s=7344 rd=19842.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=121.86 p99=243.71 mean=126.41 per-tuple=nan ns deinit mean=1.20 us tuples/s=62601 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 5000 ops (+1844544020612111104 ops/s) +close took 0.086 s +== range:uniform:10000@8 T=8 ops=4372 secs= 6.00 ops/s= 729 lat us p50=11403.26 p99=23855.10 p99.9=27525.12 max=29484.4 mean=10973.05 + cpu=23.81s ops/cpu-s=184 rd=2363294.6 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=274.43 p99=892.93 mean=313.03 per-tuple=1067.6 ns deinit mean=2.20 us tuples/s=7273862 + insert/run=6.00s drain(close)=0.09s total=6.09s drain_cpu=0.02s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=475119616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/04-filtered-ranges.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/04-filtered-ranges.log new file mode 100644 index 000000000..6fcdaceaa --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/04-filtered-ranges.log @@ -0,0 +1,21 @@ +kvbench version=filtered-ranges splinter=splinterdb_build_version cf7dc4f7 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=4 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.15 ops/s= 13407135 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.75s ops/cpu-s=2674862 rd=146.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=3 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=840455 notfound=0 +== point:uniform@8 T=8 ops=840455 secs= 6.00 ops/s= 140070 lat us p50=66.56 p99=103.42 p99.9=876.54 max=5710.7 mean=57.02 + cpu=12.16s ops/cpu-s=69125 rd=3228.5 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 234249 ops (+23423 ops/s) +== range:uniform:1@8 T=8 ops=360443 secs= 6.00 ops/s= 60071 lat us p50=125.95 p99=266.24 p99.9=1490.94 max=4006.4 mean=133.06 + cpu=51.05s ops/cpu-s=7060 rd=19829.1 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=123.90 p99=266.24 mean=131.77 per-tuple=nan ns deinit mean=1.22 us tuples/s=60071 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 4909 ops (+1844536998218342400 ops/s) +close took 0.081 s +== range:uniform:10000@8 T=8 ops=4315 secs= 6.00 ops/s= 719 lat us p50=11403.26 p99=26476.54 p99.9=43515.90 max=55284.5 mean=11126.13 + cpu=23.73s ops/cpu-s=182 rd=2379482.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=282.62 p99=1196.03 mean=324.09 per-tuple=1081.8 ns deinit mean=2.54 us tuples/s=7178956 + insert/run=6.00s drain(close)=0.08s total=6.08s drain_cpu=0.02s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=474714112 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/05-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/05-main.log new file mode 100644 index 000000000..5b303260c --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/05-main.log @@ -0,0 +1,21 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db mode=open n=2000000 cache_mib=64 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=5 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=2000000 secs= 0.15 ops/s= 13038337 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.80s ops/cpu-s=2501967 rd=146.7 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + found=841983 notfound=0 +== point:uniform@8 T=8 ops=841983 secs= 6.00 ops/s= 140323 lat us p50=68.61 p99=99.33 p99.9=186.37 max=3479.4 mean=56.91 + cpu=12.78s ops/cpu-s=65881 rd=3229.3 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 229438 ops (+22942 ops/s) +== range:uniform:1@8 T=8 ops=365332 secs= 6.00 ops/s= 60886 lat us p50=123.90 p99=274.43 p99.9=598.02 max=8581.1 mean=131.28 + cpu=50.96s ops/cpu-s=7169 rd=19817.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=123.90 p99=274.43 mean=129.99 per-tuple=nan ns deinit mean=1.23 us tuples/s=60886 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 5070 ops (+1844539094749750272 ops/s) +close took 0.091 s +== range:uniform:10000@8 T=8 ops=4462 secs= 6.00 ops/s= 744 lat us p50=11141.12 p99=23855.10 p99.9=28049.41 max=30488.3 mean=10761.00 + cpu=24.00s ops/cpu-s=186 rd=2349744.5 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=274.43 p99=827.39 mean=309.92 per-tuple=1046.6 ns deinit mean=2.31 us tuples/s=7423942 + insert/run=6.00s drain(close)=0.09s total=6.09s drain_cpu=0.02s drain_rd=0 drain_wr=274432 wamp=nan db_apparent=2513178624 db_alloc=474714112 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/manifest.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/manifest.json new file mode 100644 index 000000000..fd79c40f1 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/manifest.json @@ -0,0 +1,480 @@ +{ + "mode": "disk", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "runner_sha256": "ad1a38ada90f6f275d482205f75eb359fda6bea5d21a58547a4724472db64da1", + "runs": [ + { + "label": "00-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/results.csv", + "--vtag", + "main", + "--stag", + "disk", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791922980.3116603, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923001.637733, + "log_sha256": "9f61e4f6860d5580f8bec3e1a4361f6d4ca9cd2dd27b78fdc7d6267c9b1a5ab9", + "process_cpu_s": 103.114213, + "process_read_bytes": 24361345024, + "process_write_bytes": 319488, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db", + "fixture_sha256": "e3f6d22da99b2bea72245e9aa5a01748b0ba5f5ca507512ff311d37fd27b1b2b", + "allocated_bytes": 474849280, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 474714112 + }, + { + "label": "01-filtered-ranges", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/results.csv", + "--vtag", + "filtered-ranges", + "--stag", + "disk", + "--otag", + "1", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791923005.115866, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "c97dd13b1c6041adf9a8499e06ac158c25961d7a8f5fc05a97050f103e124ce0", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923026.4388733, + "log_sha256": "89e9d282938af6c1e4b53b0027977eee762ddc5cbb1be7e80de10e870c83c115", + "process_cpu_s": 102.435588, + "process_read_bytes": 24274481152, + "process_write_bytes": 319488, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db", + "fixture_sha256": "e3f6d22da99b2bea72245e9aa5a01748b0ba5f5ca507512ff311d37fd27b1b2b", + "allocated_bytes": 474849280, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 474714112 + }, + { + "label": "02-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/results.csv", + "--vtag", + "disk-format", + "--stag", + "disk", + "--otag", + "2", + "--rep", + "2", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791923029.9635625, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791923051.2394354, + "log_sha256": "7f54b7bb8c6185efbb89b163e0f99c645e4a42e553626ff5e93f1bf402e77c5c", + "process_cpu_s": 100.44504799999999, + "process_read_bytes": 24788250624, + "process_write_bytes": 319488, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-disk-format.db", + "fixture_sha256": "dc350c6481fdd12f4e8faff8642718487862d7dafa47129735196ad64422852b", + "allocated_bytes": 475258880, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 475119616 + }, + { + "label": "03-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/results.csv", + "--vtag", + "disk-format", + "--stag", + "disk", + "--otag", + "3", + "--rep", + "3", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791923054.7242315, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791923076.044462, + "log_sha256": "8ae38233dc62dc3925b241ecdddd254d0702f4801f0e9a9b23d594d9d699de32", + "process_cpu_s": 102.90033699999998, + "process_read_bytes": 24336293888, + "process_write_bytes": 319488, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-disk-format.db", + "fixture_sha256": "dc350c6481fdd12f4e8faff8642718487862d7dafa47129735196ad64422852b", + "allocated_bytes": 475258880, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 475119616 + }, + { + "label": "04-filtered-ranges", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/results.csv", + "--vtag", + "filtered-ranges", + "--stag", + "disk", + "--otag", + "4", + "--rep", + "4", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791923079.5899572, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "c97dd13b1c6041adf9a8499e06ac158c25961d7a8f5fc05a97050f103e124ce0", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923100.9186287, + "log_sha256": "42e56e3649c71c008d56d32ec120a20be23ca3a581db89a31ae71f0ac145ced1", + "process_cpu_s": 102.01357299999995, + "process_read_bytes": 23811653632, + "process_write_bytes": 319488, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db", + "fixture_sha256": "e3f6d22da99b2bea72245e9aa5a01748b0ba5f5ca507512ff311d37fd27b1b2b", + "allocated_bytes": 474849280, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 474714112 + }, + { + "label": "05-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/scratch.db", + "--open", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk/results.csv", + "--vtag", + "main", + "--stag", + "disk", + "--otag", + "5", + "--rep", + "5", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/disk", + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791923104.41394, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923125.7401714, + "log_sha256": "b135d58d2e445ca98364e6988afc035043450287f10df92436d4ce7709daae64", + "process_cpu_s": 102.8766590000001, + "process_read_bytes": 24187314176, + "process_write_bytes": 319488, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db", + "fixture_sha256": "e3f6d22da99b2bea72245e9aa5a01748b0ba5f5ca507512ff311d37fd27b1b2b", + "allocated_bytes": 474849280, + "apparent_bytes": 2513178624 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 474714112 + } + ], + "complete": true, + "order": [ + "main", + "filtered-ranges", + "disk-format", + "disk-format", + "filtered-ranges", + "main" + ], + "phases": [ + "warm", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ] +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/results.csv b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/results.csv new file mode 100644 index 000000000..b6866c166 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/disk/results.csv @@ -0,0 +1,25 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,disk,0,0,warm,8,0,2000000,2000000,0.167,11942936.4,nan,nan,nan,0.000,nan,0.865,2312464.9,293781504,32768,146.89,0.02,nan,nan,nan,nan,nan,2000000,11942936.4,0.167,0.000,0.167,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +main,disk,0,0,point:uniform@8,8,0,2000000,864198,6.000,144024.9,65.024,101.376,182.272,4064.646,55.449,12.540,68915.3,2790273024,0,3228.74,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +main,disk,0,0,range:uniform:1@8,8,0,2000000,369812,6.000,61632.3,125.952,251.904,339.968,4916.907,129.694,51.153,7229.6,7330930688,0,19823.40,0.00,123.904,251.904,128.370,nan,1.251,369812,61632.3,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +main,disk,0,0,range:uniform:10000@8,8,0,2000000,4448,6.000,741.3,11141.120,23330.816,27525.120,29814.345,10791.460,24.193,183.9,10515451904,0,2364085.41,0.00,274.432,827.392,305.928,1050.00,2.329,44405785,7400527.3,6.000,0.087,6.087,0.015,0,274432,nan,2513178624,474714112,0.008,0,0,0,0,0 +filtered-ranges,disk,1,0,warm,8,1,2000000,2000000,0.146,13687586.8,nan,nan,nan,0.000,nan,0.754,2652949.2,293031936,32768,146.52,0.02,nan,nan,nan,nan,nan,2000000,13687586.8,0.146,0.000,0.146,0.000,0,0,nan,0,0,0.008,0,0,0,0,5 +filtered-ranges,disk,1,0,point:uniform@8,8,1,2000000,856638,6.000,142764.8,66.560,101.376,178.176,4127.836,55.934,12.670,67611.6,2765520896,0,3228.34,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +filtered-ranges,disk,1,0,range:uniform:1@8,8,1,2000000,372541,6.000,62085.1,123.904,256.000,446.464,6488.125,128.740,50.996,7305.4,7386394624,0,19827.07,0.00,121.856,256.000,127.471,nan,1.197,372541,62085.1,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +filtered-ranges,disk,1,0,range:uniform:10000@8,8,1,2000000,4462,6.000,743.6,11141.120,23855.104,28049.408,29078.273,10757.872,23.713,188.2,10544369664,0,2363148.74,0.00,274.432,827.392,306.258,1046.77,2.396,44545785,7423943.6,6.000,0.082,6.082,0.010,0,274432,nan,2513178624,474714112,0.008,0,0,0,0,0 +disk-format,disk,2,0,warm,8,2,2000000,2000000,0.160,12481185.9,nan,nan,nan,0.000,nan,0.864,2314107.7,293445632,32768,146.72,0.02,nan,nan,nan,nan,nan,2000000,12481185.9,0.160,0.000,0.160,0.000,0,0,nan,0,0,0.008,0,0,0,0,1 +disk-format,disk,2,0,point:uniform@8,8,2,2000000,939819,6.000,156630.1,58.880,99.328,198.656,4860.602,50.998,10.862,86522.2,3034685440,0,3229.01,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +disk-format,disk,2,0,range:uniform:1@8,8,2,2000000,369913,6.000,61649.2,123.904,260.096,462.848,16127.652,129.655,50.920,7264.6,7325282304,0,19802.72,0.00,123.904,260.096,128.357,nan,1.226,369913,61649.2,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +disk-format,disk,2,0,range:uniform:10000@8,8,2,2000000,4468,6.000,744.6,11141.120,23855.104,29097.984,35568.358,10746.198,23.551,189.7,10583998464,0,2368844.78,0.00,274.432,827.392,304.891,1045.67,2.198,44605785,7433949.2,6.000,0.049,6.049,0.012,0,274432,nan,2513178624,475119616,0.008,0,0,0,0,0 +disk-format,disk,3,0,warm,8,3,2000000,2000000,0.156,12786931.7,nan,nan,nan,0.000,nan,0.788,2538093.6,293187584,32768,146.59,0.02,nan,nan,nan,nan,nan,2000000,12786931.7,0.156,0.000,0.156,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +disk-format,disk,3,0,point:uniform@8,8,3,2000000,849415,6.000,141560.3,66.560,101.376,182.272,3575.427,56.411,12.827,66222.3,2742870016,0,3229.13,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +disk-format,disk,3,0,range:uniform:1@8,8,3,2000000,375631,6.000,62601.5,123.904,243.712,331.776,4268.233,127.684,51.149,7343.8,7453417472,0,19842.39,0.00,121.856,243.712,126.409,nan,1.203,375631,62601.5,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +disk-format,disk,3,0,range:uniform:10000@8,8,3,2000000,4372,6.000,728.6,11403.264,23855.104,27525.120,29484.432,10973.049,23.810,183.6,10332323840,0,2363294.57,0.00,274.432,892.928,313.027,1067.56,2.204,43645785,7273862.0,6.000,0.086,6.087,0.016,0,274432,nan,2513178624,475119616,0.008,0,0,0,0,0 +filtered-ranges,disk,4,0,warm,8,4,2000000,2000000,0.149,13407134.5,nan,nan,nan,0.000,nan,0.748,2674862.4,293351424,32768,146.68,0.02,nan,nan,nan,nan,nan,2000000,13407134.5,0.149,0.000,0.149,0.000,0,0,nan,0,0,0.008,0,0,0,0,3 +filtered-ranges,disk,4,0,point:uniform@8,8,4,2000000,840455,6.000,140069.9,66.560,103.424,876.544,5710.658,57.016,12.158,69125.1,2713423872,0,3228.52,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +filtered-ranges,disk,4,0,range:uniform:1@8,8,4,2000000,360443,6.000,60071.0,125.952,266.240,1490.944,4006.354,133.064,51.053,7060.2,7147261952,0,19829.10,0.00,123.904,266.240,131.765,nan,1.225,360443,60071.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +filtered-ranges,disk,4,0,range:uniform:10000@8,8,4,2000000,4315,6.000,719.1,11403.264,26476.544,43515.904,55284.533,11126.126,23.733,181.8,10267467776,0,2379482.68,0.00,282.624,1196.032,324.092,1081.76,2.538,43075785,7178955.7,6.000,0.081,6.081,0.017,0,274432,nan,2513178624,474714112,0.008,0,0,0,0,0 +main,disk,5,0,warm,8,5,2000000,2000000,0.153,13038337.5,nan,nan,nan,0.000,nan,0.799,2501967.2,293335040,32768,146.67,0.02,nan,nan,nan,nan,nan,2000000,13038337.5,0.153,0.000,0.153,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +main,disk,5,0,point:uniform@8,8,5,2000000,841983,6.000,140322.8,68.608,99.328,186.368,3479.365,56.908,12.780,65881.0,2718982144,0,3229.26,0.00,nan,nan,nan,nan,nan,0,0.0,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +main,disk,5,0,range:uniform:1@8,8,5,2000000,365332,6.000,60885.8,123.904,274.432,598.016,8581.136,131.285,50.959,7169.2,7239770112,0,19816.96,0.00,123.904,274.432,129.986,nan,1.226,365332,60885.8,6.000,0.000,6.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +main,disk,5,0,range:uniform:10000@8,8,5,2000000,4462,6.000,743.6,11141.120,23855.104,28049.408,30488.307,10761.001,24.002,185.9,10484559872,0,2349744.48,0.00,274.432,827.392,309.923,1046.63,2.311,44545785,7423941.7,6.000,0.091,6.092,0.017,0,274432,nan,2513178624,474714112,0.008,0,0,0,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-00-main-application.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-00-main-application.log new file mode 100644 index 000000000..a4eb6aaab --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-00-main-application.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=754678852 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +application,0,16384,1,10000,17594.0,164872,164872,82092296,500,0.162358,8192000,69632 +application,0,16384,2,10000,13184.9,484872,484872,165340036,500,0.130411,315392,0 +application,0,16384,10,10000,31664.6,3044872,3044872,819143660,500,0.315386,4096,0 +application,0,16384,100,10000,224872.8,31844872,31844872,8268776600,500,2.248521,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-01-filtered-ranges-predicate.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-01-filtered-ranges-predicate.log new file mode 100644 index 000000000..21a24859e --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-01-filtered-ranges-predicate.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=738366896 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +predicate,0,16384,1,10000,17153.2,10000,164872,82092296,500,0.157945,9240576,770048 +predicate,0,16384,2,10000,11888.4,20000,484872,165340036,500,0.118098,372736,0 +predicate,0,16384,10,10000,26638.3,100000,3044872,819143660,500,0.265547,4096,0 +predicate,0,16384,100,10000,188615.8,1000000,31844872,8268776600,500,1.885794,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-02-filtered-ranges-pruned.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-02-filtered-ranges-pruned.log new file mode 100644 index 000000000..8b61f4b6e --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-02-filtered-ranges-pruned.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=746706665 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +pruned,0,16384,1,10000,14467.8,10000,129959,82092296,500,0.135416,8679424,126976 +pruned,0,16384,2,10000,10314.8,20000,389746,165340036,500,0.101891,499712,0 +pruned,0,16384,10,10000,22023.3,100000,2467878,819143660,500,0.219592,8192,0 +pruned,0,16384,100,10000,151945.1,1000000,25846811,8268776600,500,1.519258,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-03-filtered-ranges-pruned.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-03-filtered-ranges-pruned.log new file mode 100644 index 000000000..4ce9dc249 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-03-filtered-ranges-pruned.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=761327115 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +pruned,0,16384,1,10000,14880.5,10000,129959,82092296,500,0.136954,8622080,430080 +pruned,0,16384,2,10000,9743.7,20000,389746,165340036,500,0.097276,614400,0 +pruned,0,16384,10,10000,21175.9,100000,2467878,819143660,500,0.211409,8192,0 +pruned,0,16384,100,10000,149478.0,1000000,25846811,8268776600,500,1.494755,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-04-filtered-ranges-predicate.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-04-filtered-ranges-predicate.log new file mode 100644 index 000000000..ab25895b4 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-04-filtered-ranges-predicate.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=733262626 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +predicate,0,16384,1,10000,16617.1,10000,164872,82092296,500,0.154058,8835072,245760 +predicate,0,16384,2,10000,11947.7,20000,484872,165340036,500,0.119128,299008,0 +predicate,0,16384,10,10000,27579.1,100000,3044872,819143660,500,0.274914,4096,0 +predicate,0,16384,100,10000,192128.7,1000000,31844872,8268776600,500,1.921121,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-05-main-application.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-05-main-application.log new file mode 100644 index 000000000..2df8281a8 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions0-05-main-application.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=740585396 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +application,0,16384,1,10000,17470.1,164872,164872,82092296,500,0.162722,9465856,323584 +application,0,16384,2,10000,12844.0,484872,484872,165340036,500,0.126877,319488,0 +application,0,16384,10,10000,30789.8,3044872,3044872,819143660,500,0.306814,4096,0 +application,0,16384,100,10000,218968.3,31844872,31844872,8268776600,500,2.189401,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-00-main-application.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-00-main-application.log new file mode 100644 index 000000000..c3604d6c4 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-00-main-application.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=756651937 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +application,1,16384,1,10000,16848.7,164872,164872,82092296,500,0.155823,8704000,368640 +application,1,16384,2,10000,12844.5,484872,484872,165340036,500,0.127171,319488,0 +application,1,16384,10,10000,30457.9,3044872,3044872,819143660,500,0.303617,4096,0 +application,1,16384,100,10000,217525.3,31844872,31844872,8268776600,500,2.174846,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-01-filtered-ranges-predicate.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-01-filtered-ranges-predicate.log new file mode 100644 index 000000000..49b7abf85 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-01-filtered-ranges-predicate.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=728622381 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +predicate,1,16384,1,10000,16916.2,10000,164872,82092296,500,0.157408,8548352,122880 +predicate,1,16384,2,10000,11843.7,20000,484872,165340036,500,0.116943,290816,0 +predicate,1,16384,10,10000,26801.9,100000,3044872,819143660,500,0.267066,4096,0 +predicate,1,16384,100,10000,188514.2,1000000,31844872,8268776600,500,1.885154,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-02-filtered-ranges-pruned.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-02-filtered-ranges-pruned.log new file mode 100644 index 000000000..b1be645d5 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-02-filtered-ranges-pruned.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=720111486 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +pruned,1,16384,1,10000,15069.9,10000,130895,82092296,500,0.137219,9129984,929792 +pruned,1,16384,2,10000,10051.6,20000,392556,165340036,500,0.099355,643072,0 +pruned,1,16384,10,10000,22287.6,100000,2485662,819143660,500,0.222435,8192,0 +pruned,1,16384,100,10000,147735.9,1000000,26033223,8268776600,500,1.477217,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-03-filtered-ranges-pruned.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-03-filtered-ranges-pruned.log new file mode 100644 index 000000000..96581e88f --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-03-filtered-ranges-pruned.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=747424638 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +pruned,1,16384,1,10000,15435.3,10000,130895,82092296,500,0.144062,8540160,483328 +pruned,1,16384,2,10000,10265.4,20000,392556,165340036,500,0.101250,622592,0 +pruned,1,16384,10,10000,21833.9,100000,2485662,819143660,500,0.217797,8192,0 +pruned,1,16384,100,10000,154227.9,1000000,26033223,8268776600,500,1.542162,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-04-filtered-ranges-predicate.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-04-filtered-ranges-predicate.log new file mode 100644 index 000000000..e7f7f2b4f --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-04-filtered-ranges-predicate.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=730901239 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +predicate,1,16384,1,10000,15954.9,10000,164872,82092296,500,0.148736,8847360,409600 +predicate,1,16384,2,10000,11897.8,20000,484872,165340036,500,0.117757,372736,0 +predicate,1,16384,10,10000,27085.9,100000,3044872,819143660,500,0.269960,4096,0 +predicate,1,16384,100,10000,190338.6,1000000,31844872,8268776600,500,1.903239,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-05-main-application.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-05-main-application.log new file mode 100644 index 000000000..cb277055b --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/collisions1-05-main-application.log @@ -0,0 +1,6 @@ +loaded=524288 elapsed_ns=738824452 +mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited,predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes +application,1,16384,1,10000,18701.2,164872,164872,82092296,500,0.171626,9277440,839680 +application,1,16384,2,10000,15494.5,484872,484872,165340036,500,0.152748,319488,0 +application,1,16384,10,10000,32761.6,3044872,3044872,819143660,500,0.311000,4096,0 +application,1,16384,100,10000,218090.4,31844872,31844872,8268776600,500,2.180660,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/manifest.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/manifest.json new file mode 100644 index 000000000..739a7a933 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/feature/manifest.json @@ -0,0 +1,1167 @@ +{ + "mode": "feature", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "runner_sha256": "ad1a38ada90f6f275d482205f75eb359fda6bea5d21a58547a4724472db64da1", + "runs": [ + { + "label": "collisions0-00-main-application", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "application", + "FILTER_HASH_COLLISIONS": "0" + }, + "started": 1791923165.5414188, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7f5895c21143d36b2ec29e65421820637de953fb332340c1b0c63489bac15a9f", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923169.5206294, + "log_sha256": "10292d4bc75c45c673d702185fae0a305bc285a4eb02b10174ef328653ef0774", + "process_cpu_s": 3.7270290000000004, + "process_read_bytes": 24338432, + "process_write_bytes": 85606400, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "17594.0", + "visited": "164872", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.162358", + "read_bytes": "8192000", + "write_bytes": "69632" + }, + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "13184.9", + "visited": "484872", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.130411", + "read_bytes": "315392", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "31664.6", + "visited": "3044872", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.315386", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "224872.8", + "visited": "31844872", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "2.248521", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85712896 + }, + { + "label": "collisions0-01-filtered-ranges-predicate", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "predicate", + "FILTER_HASH_COLLISIONS": "0" + }, + "started": 1791923169.5871198, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923173.064557, + "log_sha256": "9aba8a7439cf3bdc77877e5eaca8ff6e4126415cef11d101d3b7e16d3a4167b0", + "process_cpu_s": 3.280649999999999, + "process_read_bytes": 25190400, + "process_write_bytes": 85491712, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "17153.2", + "visited": "10000", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.157945", + "read_bytes": "9240576", + "write_bytes": "770048" + }, + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "11888.4", + "visited": "20000", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.118098", + "read_bytes": "372736", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "26638.3", + "visited": "100000", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.265547", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "188615.8", + "visited": "1000000", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.885794", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85602304 + }, + { + "label": "collisions0-02-filtered-ranges-pruned", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "pruned", + "FILTER_HASH_COLLISIONS": "0" + }, + "started": 1791923173.1181376, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923176.1443448, + "log_sha256": "8043099796e133fcc3255aea7e458d3c1135ef37512c8bdb5992ef756c111f56", + "process_cpu_s": 2.8035410000000005, + "process_read_bytes": 25690112, + "process_write_bytes": 85688320, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "14467.8", + "visited": "10000", + "predicate_calls": "129959", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.135416", + "read_bytes": "8679424", + "write_bytes": "126976" + }, + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "10314.8", + "visited": "20000", + "predicate_calls": "389746", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.101891", + "read_bytes": "499712", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "22023.3", + "visited": "100000", + "predicate_calls": "2467878", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.219592", + "read_bytes": "8192", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "151945.1", + "visited": "1000000", + "predicate_calls": "25846811", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.519258", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85790720 + }, + { + "label": "collisions0-03-filtered-ranges-pruned", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "pruned", + "FILTER_HASH_COLLISIONS": "0" + }, + "started": 1791923176.197764, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923179.2208564, + "log_sha256": "de5d3f3af544626f18c0d02113071c8165a11a463fc36b586f9a2314f25a8143", + "process_cpu_s": 2.770753, + "process_read_bytes": 24727552, + "process_write_bytes": 85598208, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "14880.5", + "visited": "10000", + "predicate_calls": "129959", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.136954", + "read_bytes": "8622080", + "write_bytes": "430080" + }, + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "9743.7", + "visited": "20000", + "predicate_calls": "389746", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.097276", + "read_bytes": "614400", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "21175.9", + "visited": "100000", + "predicate_calls": "2467878", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.211409", + "read_bytes": "8192", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "149478.0", + "visited": "1000000", + "predicate_calls": "25846811", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.494755", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85704704 + }, + { + "label": "collisions0-04-filtered-ranges-predicate", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "predicate", + "FILTER_HASH_COLLISIONS": "0" + }, + "started": 1791923179.2738848, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923182.850894, + "log_sha256": "d68e891a8e23bcb9ee7f71fda245664250652e07082c5bb339ea5d2167955041", + "process_cpu_s": 3.3136840000000003, + "process_read_bytes": 24932352, + "process_write_bytes": 85544960, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "16617.1", + "visited": "10000", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.154058", + "read_bytes": "8835072", + "write_bytes": "245760" + }, + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "11947.7", + "visited": "20000", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.119128", + "read_bytes": "299008", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "27579.1", + "visited": "100000", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.274914", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "192128.7", + "visited": "1000000", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.921121", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85647360 + }, + { + "label": "collisions0-05-main-application", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "application", + "FILTER_HASH_COLLISIONS": "0" + }, + "started": 1791923182.908952, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7f5895c21143d36b2ec29e65421820637de953fb332340c1b0c63489bac15a9f", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923186.7882178, + "log_sha256": "53d9a4fd51c5a58c4c301bb351c6d14ba8c50c8367c6a119448e682b30155a07", + "process_cpu_s": 3.6521869999999996, + "process_read_bytes": 25645056, + "process_write_bytes": 85594112, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "17470.1", + "visited": "164872", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.162722", + "read_bytes": "9465856", + "write_bytes": "323584" + }, + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "12844.0", + "visited": "484872", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.126877", + "read_bytes": "319488", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "30789.8", + "visited": "3044872", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.306814", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "0", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "218968.3", + "visited": "31844872", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "2.189401", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85700608 + }, + { + "label": "collisions1-00-main-application", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "application", + "FILTER_HASH_COLLISIONS": "1" + }, + "started": 1791923186.8482296, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7f5895c21143d36b2ec29e65421820637de953fb332340c1b0c63489bac15a9f", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923190.67707, + "log_sha256": "c8bcc2b65f82264698d758553be23e1232314a7260d3347ff188a67b807a6a86", + "process_cpu_s": 3.6349080000000025, + "process_read_bytes": 24752128, + "process_write_bytes": 85700608, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "16848.7", + "visited": "164872", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.155823", + "read_bytes": "8704000", + "write_bytes": "368640" + }, + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "12844.5", + "visited": "484872", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.127171", + "read_bytes": "319488", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "30457.9", + "visited": "3044872", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.303617", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "217525.3", + "visited": "31844872", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "2.174846", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85807104 + }, + { + "label": "collisions1-01-filtered-ranges-predicate", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "predicate", + "FILTER_HASH_COLLISIONS": "1" + }, + "started": 1791923190.7301552, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923194.207426, + "log_sha256": "072a3c10cc9224692c08aff4e8caeee33fa7b382deaf79b06c0d4cf76ebbe73d", + "process_cpu_s": 3.2701670000000016, + "process_read_bytes": 24653824, + "process_write_bytes": 85508096, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "16916.2", + "visited": "10000", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.157408", + "read_bytes": "8548352", + "write_bytes": "122880" + }, + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "11843.7", + "visited": "20000", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.116943", + "read_bytes": "290816", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "26801.9", + "visited": "100000", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.267066", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "188514.2", + "visited": "1000000", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.885154", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85610496 + }, + { + "label": "collisions1-02-filtered-ranges-pruned", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "pruned", + "FILTER_HASH_COLLISIONS": "1" + }, + "started": 1791923194.271274, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923197.246301, + "log_sha256": "da3ef1b2ac9790b729aeb1617d2937e3e29620c2bdbe501636682a71d3f3c1f8", + "process_cpu_s": 2.7409719999999993, + "process_read_bytes": 25391104, + "process_write_bytes": 85446656, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "15069.9", + "visited": "10000", + "predicate_calls": "130895", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.137219", + "read_bytes": "9129984", + "write_bytes": "929792" + }, + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "10051.6", + "visited": "20000", + "predicate_calls": "392556", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.099355", + "read_bytes": "643072", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "22287.6", + "visited": "100000", + "predicate_calls": "2485662", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.222435", + "read_bytes": "8192", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "147735.9", + "visited": "1000000", + "predicate_calls": "26033223", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.477217", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85553152 + }, + { + "label": "collisions1-03-filtered-ranges-pruned", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "pruned", + "FILTER_HASH_COLLISIONS": "1" + }, + "started": 1791923197.3042665, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923200.3293488, + "log_sha256": "d9fac9cd7197430601b68c2d4c821b7b341e4b0fb3fd91614a812f7e16cadf82", + "process_cpu_s": 2.834286000000004, + "process_read_bytes": 24928256, + "process_write_bytes": 85471232, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "15435.3", + "visited": "10000", + "predicate_calls": "130895", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.144062", + "read_bytes": "8540160", + "write_bytes": "483328" + }, + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "10265.4", + "visited": "20000", + "predicate_calls": "392556", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.101250", + "read_bytes": "622592", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "21833.9", + "visited": "100000", + "predicate_calls": "2485662", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.217797", + "read_bytes": "8192", + "write_bytes": "0" + }, + { + "mode": "pruned", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "154227.9", + "visited": "1000000", + "predicate_calls": "26033223", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.542162", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85577728 + }, + { + "label": "collisions1-04-filtered-ranges-predicate", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "predicate", + "FILTER_HASH_COLLISIONS": "1" + }, + "started": 1791923200.3864868, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791923203.8634338, + "log_sha256": "c8c78b8cd8e1db7850eed6fd70e3287ed84d20d6ae6f40f4b7977feee6b1a249", + "process_cpu_s": 3.2742930000000037, + "process_read_bytes": 24838144, + "process_write_bytes": 85532672, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "15954.9", + "visited": "10000", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.148736", + "read_bytes": "8847360", + "write_bytes": "409600" + }, + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "11897.8", + "visited": "20000", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.117757", + "read_bytes": "372736", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "27085.9", + "visited": "100000", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.269960", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "predicate", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "190338.6", + "visited": "1000000", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "1.903239", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85639168 + }, + { + "label": "collisions1-05-main-application", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/hash_range_benchmark", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/feature/scratch.db", + "16384", + "10000" + ], + "environment": { + "FILTER_MODE": "application", + "FILTER_HASH_COLLISIONS": "1" + }, + "started": 1791923203.920536, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7f5895c21143d36b2ec29e65421820637de953fb332340c1b0c63489bac15a9f", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923207.7997932, + "log_sha256": "9c87d7d76d98c761ae9488e3115dd77b800145b2f473bf4dbc8d9b9b274913f2", + "process_cpu_s": 3.6827840000000003, + "process_read_bytes": 26042368, + "process_write_bytes": 85520384, + "verification_kind": "per-query expected key/value/count checks and independently computed checksums", + "rows": [ + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "1", + "queries": "10000", + "ns_per_query": "18701.2", + "visited": "164872", + "predicate_calls": "164872", + "checksum": "82092296", + "warmup_queries": "500", + "cpu_s": "0.171626", + "read_bytes": "9277440", + "write_bytes": "839680" + }, + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "2", + "queries": "10000", + "ns_per_query": "15494.5", + "visited": "484872", + "predicate_calls": "484872", + "checksum": "165340036", + "warmup_queries": "500", + "cpu_s": "0.152748", + "read_bytes": "319488", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "10", + "queries": "10000", + "ns_per_query": "32761.6", + "visited": "3044872", + "predicate_calls": "3044872", + "checksum": "819143660", + "warmup_queries": "500", + "cpu_s": "0.311000", + "read_bytes": "4096", + "write_bytes": "0" + }, + { + "mode": "application", + "collisions": "1", + "keys_per_tenant": "16384", + "limit": "100", + "queries": "10000", + "ns_per_query": "218090.4", + "visited": "31844872", + "predicate_calls": "31844872", + "checksum": "8268776600", + "warmup_queries": "500", + "cpu_s": "2.180660", + "read_bytes": "0", + "write_bytes": "0" + } + ], + "errors": 0, + "violations": 0, + "allocated_bytes": 85622784 + } + ], + "complete": true, + "queries_per_limit": 10000, + "rows_per_tenant": 16384, + "warmup_queries_per_limit": 500, + "order": [ + [ + "main", + "application" + ], + [ + "filtered-ranges", + "predicate" + ], + [ + "filtered-ranges", + "pruned" + ], + [ + "filtered-ranges", + "pruned" + ], + [ + "filtered-ranges", + "predicate" + ], + [ + "main", + "application" + ] + ] +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-build.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-build.json new file mode 100644 index 000000000..7e651dc7b --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-build.json @@ -0,0 +1,195 @@ +{ + "tree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges", + "head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "library": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/build-filter-predicate/release/lib/libsplinterdb.a", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "library_build_command": [ + "make", + "--jobserver-style=pipe", + "-j4", + "BUILD_ROOT=build-filter-predicate", + "BUILD_MODE=release", + "libs" + ], + "source_files_sha256": { + "include/splinterdb/data.h": "6d8984e702da446c2fe145eb2f6a576ca040cfd5740733c405d0d7387b243ec2", + "include/splinterdb/default_data_config.h": "179386f3e224d322a9d7ac97afd929292994a1d890a3254705f9ebea525bce32", + "include/splinterdb/platform_linux/public_platform.h": "c5ded82c5b5bb5bbcedf06a66951eb98dc8892f7b3c4f8ce07c474db98ac68a0", + "include/splinterdb/public_platform.h": "4ce9da3fea9a54fb7bb613ec1458f11f043525efb7d0bc55182f7183dd4a4f6e", + "include/splinterdb/public_util.h": "4e8f6b30408fb4174894d4ba809acff4e96f559370c7749b65bb3724a853603f", + "include/splinterdb/splinterdb.h": "8a6533879de25f1e448a866c4c03304037e40c6e39e20923c0557e9c87d4fda0", + "src/PackedArray.c": "97b64eb0a4f550e55f3385d39d729a878fbc581909ea16ec3b8987a7f46604cc", + "src/PackedArray.h": "e46f854b487a8b130f0613efcab11836012d27c8aacf9ca9e76a2de1a7eb6874", + "src/allocator.c": "96c7f0a5e0808719cf114d267c84da1f057b5eed748b0672d60686d7f130b96f", + "src/allocator.h": "bc0621be7903d64e25059140005dafeea3f2a4a4ae0e658686501e52e4be934f", + "src/batch_rwlock.c": "c043d9e5ead75cd9e1266a6f51fb555012269f3822d7f0233dc2abbd962c3aae", + "src/batch_rwlock.h": "0ff5f0bd7fdf5f9a63ca2f77c387d574d8009ad726b47283c93566cf06bc0927", + "src/blob.c": "7d46871423a15992f697cabb363316183cb842a11c26017f8566267174e5ae45", + "src/blob.h": "8f68d2f67de8a9d81a422b28d47aa636cc7234fccc3dbabe59f2355bc361e93c", + "src/blob_build.c": "e484e9c14fd8a9b3c1d211485c35931c944fc66ac4ee78ea20ffc5b3f074b10b", + "src/blob_build.h": "38176881cf9159247e6f578bbc7abb725cfa969c493715df7d1388cb7cbfd119", + "src/btree.c": "f637081a194302c584cccfd8d717253817794b34bbca754bd2c9e597acc534b9", + "src/btree.h": "6ed244b6731c490f224c7b5fea7b5a9dc27eca1aa808e6dab6b44a0c7b511638", + "src/btree_private.h": "43693dd3048ae050b3358ccfc9b91371a3d9dff4f3d6d5c8b07445897dee09f6", + "src/cache.h": "d4c1e133d6423fff324d0ebcf5c95e13647c0022a3458935156746b90d816d6d", + "src/clockcache.c": "e0ef2e19a8280849df4ee80d9b8a54a0faaf56a13faefe7a91d54f5b7269f933", + "src/clockcache.h": "68f4e7b019d0440f85929bfc0da1891ce941318ee7dfd14a3d4d923e3d4e72d8", + "src/core.c": "36ca8da5f4a8c9adfdb44dfac235b0c8a7c6f99a1e31614a605970f14577c4f9", + "src/core.h": "2d4141f4b351cafae8e71e21dc6f570a6d0a4bf3b8b91df86f83dd3e9c0a7809", + "src/data_blob_build.c": "13475faa56bba94ee26ef7fec4e4da8a5045e0efe510a63291fd5e2697cb5d22", + "src/data_blob_build.h": "dd7f263a0b1138325b211c561ba7289183a2cc8472978cba3ca430295e11e5f3", + "src/data_internal.c": "0a42dd477aaa6ad6c4910841045c4c768ad8d248c0126c64c51035f53e769cde", + "src/data_internal.h": "98b4ebe77576c4cbb7dd9984ca5e297851175494062073f1a4c137c5afb8b587", + "src/default_data_config.c": "161352f5247978f3d57033818d05a79d0a1a0d0ffac2bc9f25f4f33eabca2227", + "src/histogram.c": "24e09dfabd6085021d8a06ef9b10f6e85b24914d95831be359df09f8a9729aff", + "src/histogram.h": "7ccfde6d3ada7ead18fe477c06a6b021db83ca8c041ab0998ca023b945a3c1c7", + "src/iterator.h": "2fcddeb9fa6bd49f4e3da933224219bdb830f9e2f11638ea1ed13eba534fa27f", + "src/log.h": "3e7eb447259e7901b915b52b7545e2d5088475171af1716dd9c849dab4c3d1b9", + "src/log_data.h": "06ab1d39b77abb01ec66a901fa52354908be26db8daef52899f5c03a6db2462e", + "src/lookup_result.h": "7528b988a28ef91b6bd7a4fa10dfa51b9ce2475e801b3b6a8bf30adf88f1ef16", + "src/memtable.c": "7db3b4754497c73b4071c7e7e591cc76fc226197fb70d98fb46caf139b3953f4", + "src/memtable.h": "244cac7cf304df85a25c8bede2df106d6f840d65bbd1a3551b3d9f2ed9129fc1", + "src/merge.c": "0b9547090a291b88f957689d8786fbaab35a2e2a18a4ae72b532c11415f5a9a7", + "src/merge.h": "771c1c2b5c581901c67aefa07d9e8c9d4f730e1137633b5b5efe35d14a6bd0d1", + "src/mini_allocator.c": "d9444da13fca2000f4128500b84be2b778300a4befc93f72e52b071b83c6491e", + "src/mini_allocator.h": "8d5242f5e4df5443baeab7a494a3c3b2f1491216abfd914c6dbc273370e45683", + "src/notification.c": "54af1fe6af2e5fc84fbee21a74ff55069a8aa7e4ec7f9d8c9485e85e91669b58", + "src/notification.h": "f4df33131f407e6a957602a05c72f4bca7efed98b1a39134f94bf889a6ca5f29", + "src/ondisk_ref.c": "2ace8cbbce3f80a54ff7762ad03a79a564665fa660394f1d3446235a3e40ed7b", + "src/ondisk_ref.h": "643325c8b4f16fab41d8c884c97bdfcf6c86e5178398508fa0f81b6b0606926d", + "src/pcq.h": "862cd346b0097014ccae797dc1463e7b3fc387f409319d5adcef7613b6f82f84", + "src/platform_linux/async.h": "41383b57e21f0f51d737749a4ce7c2c5330ac9363e742330625c9aedd4f13a52", + "src/platform_linux/laio.c": "0ce628806e6896070359a9e39d069f326f460ab697c6c17ff17d171614290881", + "src/platform_linux/laio.h": "f3987e22f1fd7a9d6b46c79f84e75559aec5410239a13d52a7a5343ba792b28e", + "src/platform_linux/platform.h": "f1c3d7c5bcd089ed7a7df129d3b8d00d7617de9a7aa6f9d1f7deeb739281e768", + "src/platform_linux/platform_assert.c": "112e803b6db03dd8c14a3bfc3f1139bbcc4b81f1d5e24a232b7b99ce89146c2d", + "src/platform_linux/platform_assert.h": "751dccceac9ce4a97d25d441b5216ced5ee5f079a13c9ccd43551b0994aa7cc2", + "src/platform_linux/platform_backtrace.h": "ef13ccbf4ae4747b0de244a37c026ad8769018d84f9fc4d6493db43d9714358c", + "src/platform_linux/platform_buffer.c": "b790d645b3dc5f638da871bf9611865dc700927f3b594a1d645b0350bd1bd027", + "src/platform_linux/platform_buffer.h": "72e016dbb19b8250378b3a5c79fbebf9c3d72a31d07a232f36431e04f0e4afa6", + "src/platform_linux/platform_condvar.c": "e595f69194accb3f3460d6fc9a4a9144bd7f495dbeb77f11bf3616dc0e9b03ec", + "src/platform_linux/platform_condvar.h": "a4ee02ae5069637bce4cae07f3ed9869925f17a9e6e1b04d69baedfc148fd493", + "src/platform_linux/platform_hash.h": "f69f5ed6218bfe62c62718b387e1c110e9fca46c07603d4d51ccc4fc782b09ab", + "src/platform_linux/platform_heap.c": "72bb2753f525e56f8c398988a787feb38f02efaa749e5d57cb476f967269c479", + "src/platform_linux/platform_heap.h": "921f3113d20b44c36817c284702e965e01769596ff012edd6ceaa70cbdbbd10a", + "src/platform_linux/platform_io.c": "875a7331f8bfa46e9a6961f50acb40b107d40883839c94def52a289a521c66be", + "src/platform_linux/platform_io.h": "69ae9556daf1c7a69e007abd494bf9c86ec81c08bca1e032663009cef95b2fac", + "src/platform_linux/platform_log.c": "318bd17c858475a3a91c5e9641b52808f7c4194e1c6fc9a7d801392b04c49538", + "src/platform_linux/platform_log.h": "569484386150fcfbf5dde722c5b8e34d67d65be3454e3e009cb9df41f11efea6", + "src/platform_linux/platform_machine.h": "c02745313c7268dbe99efa2b93dd4d37640b5a7aa2e88c78fd5b73742f252c7d", + "src/platform_linux/platform_mutex.c": "7ec1dd6a5397217d28fd85c10648d29a70bf66649f59f76d7387236031b15bf3", + "src/platform_linux/platform_mutex.h": "688991d6a7dc57a7a88f565541561344cdfbb1733ca6073f1655d6c436ed6544", + "src/platform_linux/platform_random.h": "184ac9da9c38a6c10275891be61ccb91beaf2234ebb9ce046d059cb16824d68f", + "src/platform_linux/platform_semaphore.h": "9fdb7cfeaf36c8bf9aa346a16466ed6bc50bea76fdb5722c86001563d55bedfe", + "src/platform_linux/platform_sleep.h": "34aeaea35133bf33e47eb1d9ace429297644e13bc100d90ebbd205bceb58d3ab", + "src/platform_linux/platform_sort.h": "d374788510f94a8c3b86ff6bfa1f6a3ff8ac33168eb26f6f7774984252f6d01e", + "src/platform_linux/platform_spinlock.h": "7b3f0409109f3341f2287fe7cd8fb69faac4a74b009a44c1eb83220f71e94de0", + "src/platform_linux/platform_status.h": "2749285ea1322e5c1639019b728fde4b87b0cae6f71a7fc16e420bd592d38c00", + "src/platform_linux/platform_string.h": "fa41faf9c6661c62ece783740250232709656b0a8677f542e390a5f3b0d834a6", + "src/platform_linux/platform_threads.c": "789465a2c6ac7ac501e5878f429cc0dd9b163145a522e247a4833e3de7af8f2a", + "src/platform_linux/platform_threads.h": "653e070115131756e6aaffc3c2764a6437fb39ff96df4afc8a01e6e751c9d622", + "src/platform_linux/platform_time.h": "f212a08cf8432693711cecfd389f9ed3d0a70e0c056dc062321bd11c4a8cbec7", + "src/platform_linux/platform_trace.c": "4a68e89f214920dd2762d072a0ad8e7c80d9b1733a64e4a7e3dd4bb04b91c352", + "src/platform_linux/platform_trace.h": "345f7283b01e5a9b4d5da7bea66ad44ff61f53155417c97990e23bb8fedadf33", + "src/platform_linux/platform_typed_alloc.h": "42c1d2bbd56c183d11768a5f43d440cf3c1bfe329c4352afa6acd3bc2b191a8e", + "src/platform_linux/platform_units.h": "1cb22a07b33ec8b58f9381438f480f561d12b838f315a117cc1f979e95b1ef65", + "src/platform_linux/platform_util.h": "4fa2702b3919eb1a7efcf29457f5ee9a3d7a7b4c38fed4606264562022fa2638", + "src/platform_linux/poison.h": "a32dfa445439eb08913e519117e7b11f72b2c3550577a74769ed935a792c42c5", + "src/platform_linux/shmalloc.c": "e4631d63a9db1ad58472ad3d034d081f2bb9a97f34c5898f7c848dda923727f8", + "src/platform_linux/shmalloc.h": "081a82473f6021f8ed628e79f8ae5dc7104988981f8c3b25749ea42c8158113d", + "src/prefetch.h": "9d55d9dad64e3795c81856bc38688936e4e74e488dfe2d29a1f5438ad000df42", + "src/rc_allocator.c": "a145d39af39caf9dec6f1e1e24b6606ed615587d8dee52a23e0df410b2d07545", + "src/rc_allocator.h": "65f01decb7244b8f6a6c9e7acb47d4ccb4f2f3f369461e70aff33211f03ee39b", + "src/routing_filter.c": "6d3409d06eeac3bbb09e7764a7df3b5697e2ca211852b68c19ce43936911c4ff", + "src/routing_filter.h": "f0621e4cbf199fc9c65f410e112f8392365ac0c3538ecc3e526663aee3ba04bd", + "src/shard_log.c": "31449c056fb90448f08954acd7049e8ce7f2a9a6e349f5b78ec1c134e5f9465d", + "src/shard_log.h": "07c5867b959eefd35e3e9f08e6c015a594f6331850844590e9c2b1126cd2716b", + "src/splinterdb.c": "2ba3a6ccfaa07ae2dca5cde2c9fe43c070bae9458c8f95bfbab28697e700382d", + "src/splinterdb_tests_private.h": "579031767b8869f49cf1e3fc94caeec13a1362dfd047782538df40b281c93414", + "src/superblock.c": "34dd4c724e53c98f170c024c70dd22dae0a466806c167134200f92128420dfba", + "src/superblock.h": "72cd00ce0a7c43d1c25905d6bd39a400fa3245c133a3f559c58ae34a962157f2", + "src/task.c": "470418c0da14795c4697c51ef7b22e128b691c1c24a704791eea09a10b9373a4", + "src/task.h": "ebcc3bd29cef9ee8718c6ff0186863fff0dad03bcfaa43e5eacaf83c1933d01d", + "src/trunk.c": "25be4f541b2fde0862d2940254fbfbaab9ecf7da38438af11fdd87c40d9369b1", + "src/trunk.h": "b2797275df4eed3546be3bb3d20d465836a79ae72bba8ce254b86a327439c079", + "src/util.c": "969df408c4c393f73feb05dfa37ebe1ed8be627cef630061830e4992c4c1fd9c", + "src/util.h": "9fa455f5a6268fa4a6864d444849b0872391af00512932c1587363cfd38304ef", + "src/vector.h": "6855dd0430f1a868d098797cbd4f3453ab542b5f6fbe716c79871124bee200a9", + "src/writeback_set.c": "cb485716f57a3183ce82543848edaf61f6568c9ff0445ea5ff989f95c00848ff", + "src/writeback_set.h": "68d6f589d24719a09e2680b05a1eb7e36c22b1fab978768f5b54f958f667aec9" + }, + "binaries": { + "kvbench": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/include", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/src", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/kvbench.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/build-filter-predicate/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/kvbench" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/kvbench", + "binary_sha256": "c97dd13b1c6041adf9a8499e06ac158c25961d7a8f5fc05a97050f103e124ce0", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/kvbench.c", + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9" + }, + "hash_range_benchmark": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DPREDICATE_RANGE_FILTER_API", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/include", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/src", + "-I", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/hash_range_benchmark.c", + "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/build-filter-predicate/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/hash_range_benchmark", + "binary_sha256": "3a708ee583c2009b665a690120024e199ef76e060d6a0d165b4d7c511d9f07f4", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/hash_range_benchmark.c", + "source_sha256": "1353f2f2acbff19dc7551a385ec4aa3b6cf6f2eba39ce19ea9cbc53c20a61add" + } + } +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-hash_range_benchmark-build.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-hash_range_benchmark-build.log new file mode 100644 index 000000000..1cda486c9 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-hash_range_benchmark-build.log @@ -0,0 +1,2 @@ +lto-wrapper: warning: using serial compilation of 12 LTRANS jobs +lto-wrapper: note: see the ‘-flto’ option documentation for more information diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-kvbench-build.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-kvbench-build.log new file mode 100644 index 000000000..91b66fe35 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-kvbench-build.log @@ -0,0 +1,2 @@ +lto-wrapper: warning: using serial compilation of 13 LTRANS jobs +lto-wrapper: note: see the ‘-flto’ option documentation for more information diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-libs.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-libs.log new file mode 100644 index 000000000..fe9902bc3 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/filtered-ranges-libs.log @@ -0,0 +1,2 @@ +Checking for mismatched config...No mismatched config found +Building archive build-filter-predicate/release/lib/libsplinterdb.a diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/final-provenance-check.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/final-provenance-check.json new file mode 100644 index 000000000..bdfc3cbb8 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/final-provenance-check.json @@ -0,0 +1,6 @@ +{ + "all_pinned_heads_runtime_sources_libraries_binaries_harness_sources_match": true, + "all_completed_run_log_hashes_match": true, + "runs": 32, + "timing_finished": true +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.csv b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.csv new file mode 100644 index 000000000..e3ab3fcde --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +disk-format,disk,0,0,load:random,1,0,2000000,2000000,5.099,392249.9,0.984,2.720,8.320,108571.776,2.452,4.214,474616.4,158998528,437063680,79.50,218.53,nan,nan,nan,nan,nan,0,0.0,5.099,0.093,5.192,0.024,0,37994496,1.9156,2513178624,475258880,0.014,0,0,0,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.log new file mode 100644 index 000000000..615144505 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-disk-format.log @@ -0,0 +1,7 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-disk-format.db mode=create n=2000000 cache_mib=64 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=0 sabotage=0 +create took 0.014 s +phase load:random: T=1 dist=random (fixed count) +close took 0.093 s +== load:random T=1 ops=2000000 secs= 5.10 ops/s= 392250 lat us p50=0.98 p99=2.72 p99.9=8.32 max=108571.8 mean=2.45 + cpu=4.21s ops/cpu-s=474616 rd=79.5 B/op wr=218.5 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=5.10s drain(close)=0.09s total=5.19s drain_cpu=0.02s drain_rd=0 drain_wr=37994496 wamp=1.916 db_apparent=2513178624 db_alloc=475258880 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.csv b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.csv new file mode 100644 index 000000000..3c167243b --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,disk,0,0,load:random,1,0,2000000,2000000,5.286,378349.4,0.984,2.976,8.320,114075.828,2.545,4.304,464736.9,162500608,437792768,81.25,218.90,nan,nan,nan,nan,nan,0,0.0,5.286,0.097,5.383,0.025,0,36859904,1.9139,2513178624,474849280,0.015,0,0,0,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.log new file mode 100644 index 000000000..dfc0f758b --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/disk-main.log @@ -0,0 +1,7 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db mode=create n=2000000 cache_mib=64 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=6.0 warmup=1.0 rep=0 sabotage=0 +create took 0.015 s +phase load:random: T=1 dist=random (fixed count) +close took 0.097 s +== load:random T=1 ops=2000000 secs= 5.29 ops/s= 378349 lat us p50=0.98 p99=2.98 p99.9=8.32 max=114075.8 mean=2.54 + cpu=4.30s ops/cpu-s=464737 rd=81.3 B/op wr=218.9 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=5.29s drain(close)=0.10s total=5.38s drain_cpu=0.02s drain_rd=0 drain_wr=36859904 wamp=1.914 db_apparent=2513178624 db_alloc=474849280 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/fixtures.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/fixtures.json new file mode 100644 index 000000000..a7af8c38e --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/fixtures.json @@ -0,0 +1,26 @@ +{ + "resident-main": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "resident-disk-format": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "disk-main": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db", + "fixture_sha256": "e3f6d22da99b2bea72245e9aa5a01748b0ba5f5ca507512ff311d37fd27b1b2b", + "allocated_bytes": 474849280, + "apparent_bytes": 2513178624 + }, + "disk-disk-format": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-disk-format.db", + "fixture_sha256": "dc350c6481fdd12f4e8faff8642718487862d7dafa47129735196ad64422852b", + "allocated_bytes": 475258880, + "apparent_bytes": 2513178624 + } +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/manifest.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/manifest.json new file mode 100644 index 000000000..34b5cdd98 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/manifest.json @@ -0,0 +1,290 @@ +{ + "mode": "fixtures", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "runner_sha256": "ad1a38ada90f6f275d482205f75eb359fda6bea5d21a58547a4724472db64da1", + "runs": [ + { + "label": "resident-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures", + "load:random" + ], + "environment": {}, + "started": 1791922361.0406556, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791922361.5570598, + "log_sha256": "368e3edc4cf82a0853dfd119f38e8bc8d71b99b2bf4a605eb8de4aea0d2f016d", + "process_cpu_s": 0.390232, + "process_read_bytes": 0, + "process_write_bytes": 28803072, + "verification_kind": "load count/API errors/close; warm scans later validate every key/value", + "errors": 0, + "violations": 0, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "fixture_fsynced": true, + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + { + "label": "resident-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db", + "--create", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures", + "load:random" + ], + "environment": {}, + "started": 1791922361.8386126, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791922362.3550184, + "log_sha256": "33392dac5f25db63688655b4b1c55f5aafd389906075f6fde17b139f10b9f087", + "process_cpu_s": 0.38215400000000005, + "process_read_bytes": 0, + "process_write_bytes": 28803072, + "verification_kind": "load count/API errors/close; warm scans later validate every key/value", + "errors": 0, + "violations": 0, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "fixture_fsynced": true, + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + { + "label": "disk-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.csv", + "--vtag", + "main", + "--stag", + "disk", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures", + "load:random" + ], + "environment": {}, + "started": 1791922362.6429803, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791922368.1246145, + "log_sha256": "bab82f1f8713052eae6adc0cc1eb5443fa5549f7564f954df17bb56960dd4adb", + "process_cpu_s": 4.372297, + "process_read_bytes": 162500608, + "process_write_bytes": 474705920, + "verification_kind": "load count/API errors/close; warm scans later validate every key/value", + "errors": 0, + "violations": 0, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-main.db", + "fixture_sha256": "e3f6d22da99b2bea72245e9aa5a01748b0ba5f5ca507512ff311d37fd27b1b2b", + "fixture_fsynced": true, + "allocated_bytes": 474849280, + "apparent_bytes": 2513178624 + }, + { + "label": "disk-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-disk-format.db", + "--create", + "--n", + "2000000", + "--cache-mib", + "64", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "1", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "6", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-disk-format.csv", + "--vtag", + "disk-format", + "--stag", + "disk", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures", + "load:random" + ], + "environment": {}, + "started": 1791922372.068808, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791922377.3509493, + "log_sha256": "5f84764c215fc98079a2d169731bb5977deff08a11d90dbde6acbedc99a058cd", + "process_cpu_s": 4.283090000000001, + "process_read_bytes": 158998528, + "process_write_bytes": 475111424, + "verification_kind": "load count/API errors/close; warm scans later validate every key/value", + "errors": 0, + "violations": 0, + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/disk-disk-format.db", + "fixture_sha256": "dc350c6481fdd12f4e8faff8642718487862d7dafa47129735196ad64422852b", + "fixture_fsynced": true, + "allocated_bytes": 475258880, + "apparent_bytes": 2513178624 + } + ], + "complete": true +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.csv b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.csv new file mode 100644 index 000000000..e0ee454d9 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +disk-format,resident,0,0,load:random,1,0,200000,200000,0.338,591931.7,1.000,4.544,8.320,5479.930,1.623,0.338,592055.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.338,0.135,0.473,0.030,0,28749824,1.1593,162664448,28905472,0.016,0,0,0,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.log new file mode 100644 index 000000000..126d3022e --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-disk-format.log @@ -0,0 +1,7 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db mode=create n=200000 cache_mib=128 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=0 sabotage=0 +create took 0.016 s +phase load:random: T=1 dist=random (fixed count) +close took 0.135 s +== load:random T=1 ops=200000 secs= 0.34 ops/s= 591932 lat us p50=1.00 p99=4.54 p99.9=8.32 max=5479.9 mean=1.62 + cpu=0.34s ops/cpu-s=592056 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=0.34s drain(close)=0.13s total=0.47s drain_cpu=0.03s drain_rd=0 drain_wr=28749824 wamp=1.159 db_apparent=162664448 db_alloc=28905472 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.csv b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.csv new file mode 100644 index 000000000..dd76516ed --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.csv @@ -0,0 +1,2 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,resident,0,0,load:random,1,0,200000,200000,0.338,592410.1,0.984,4.544,8.832,5685.420,1.621,0.337,593021.3,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,0.338,0.152,0.490,0.039,0,28749824,1.1593,162664448,28905472,0.017,0,0,0,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.log new file mode 100644 index 000000000..58d6f875c --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/fixtures/resident-main.log @@ -0,0 +1,7 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db mode=create n=200000 cache_mib=128 disk_gib=8 log=0 threads=1 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=0 sabotage=0 +create took 0.017 s +phase load:random: T=1 dist=random (fixed count) +close took 0.152 s +== load:random T=1 ops=200000 secs= 0.34 ops/s= 592410 lat us p50=0.98 p99=4.54 p99.9=8.83 max=5685.4 mean=1.62 + cpu=0.34s ops/cpu-s=593021 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=0.34s drain(close)=0.15s total=0.49s drain_cpu=0.04s drain_rd=0 drain_wr=28749824 wamp=1.159 db_apparent=162664448 db_alloc=28905472 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/host.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/host.json new file mode 100644 index 000000000..db3305a8e --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/host.json @@ -0,0 +1,11 @@ +{ + "cpu": "{\n \"lscpu\": [\n {\n \"field\": \"Architecture:\",\n \"data\": \"x86_64\"\n },{\n \"field\": \"CPU op-mode(s):\",\n \"data\": \"32-bit, 64-bit\"\n },{\n \"field\": \"Address sizes:\",\n \"data\": \"46 bits physical, 48 bits virtual\"\n },{\n \"field\": \"Byte Order:\",\n \"data\": \"Little Endian\"\n },{\n \"field\": \"CPU(s):\",\n \"data\": \"20\"\n },{\n \"field\": \"On-line CPU(s) list:\",\n \"data\": \"0-19\"\n },{\n \"field\": \"Vendor ID:\",\n \"data\": \"GenuineIntel\"\n },{\n \"field\": \"Model name:\",\n \"data\": \"12th Gen Intel(R) Core(TM) i7-12800H\"\n },{\n \"field\": \"CPU family:\",\n \"data\": \"6\"\n },{\n \"field\": \"Model:\",\n \"data\": \"154\"\n },{\n \"field\": \"Thread(s) per core:\",\n \"data\": \"2\"\n },{\n \"field\": \"Core(s) per socket:\",\n \"data\": \"14\"\n },{\n \"field\": \"Socket(s):\",\n \"data\": \"1\"\n },{\n \"field\": \"Stepping:\",\n \"data\": \"3\"\n },{\n \"field\": \"CPU(s) scaling MHz:\",\n \"data\": \"20%\"\n },{\n \"field\": \"CPU max MHz:\",\n \"data\": \"4800.0000\"\n },{\n \"field\": \"CPU min MHz:\",\n \"data\": \"400.0000\"\n },{\n \"field\": \"BogoMIPS:\",\n \"data\": \"5606.40\"\n },{\n \"field\": \"Flags:\",\n \"data\": \"fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities\"\n },{\n \"field\": \"Virtualization:\",\n \"data\": \"VT-x\"\n },{\n \"field\": \"L1d cache:\",\n \"data\": \"544 KiB (14 instances)\"\n },{\n \"field\": \"L1i cache:\",\n \"data\": \"704 KiB (14 instances)\"\n },{\n \"field\": \"L2 cache:\",\n \"data\": \"11.5 MiB (8 instances)\"\n },{\n \"field\": \"L3 cache:\",\n \"data\": \"24 MiB (1 instance)\"\n },{\n \"field\": \"NUMA node(s):\",\n \"data\": \"1\"\n },{\n \"field\": \"NUMA node0 CPU(s):\",\n \"data\": \"0-19\"\n },{\n \"field\": \"Vulnerability Gather data sampling:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Indirect target selection:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Itlb multihit:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability L1tf:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Mds:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Meltdown:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Mmio stale data:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Reg file data sampling:\",\n \"data\": \"Mitigation; Clear Register File\"\n },{\n \"field\": \"Vulnerability Retbleed:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Spec rstack overflow:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Spec store bypass:\",\n \"data\": \"Mitigation; Speculative Store Bypass disabled via prctl\"\n },{\n \"field\": \"Vulnerability Spectre v1:\",\n \"data\": \"Mitigation; usercopy/swapgs barriers and __user pointer sanitization\"\n },{\n \"field\": \"Vulnerability Spectre v2:\",\n \"data\": \"Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\"\n },{\n \"field\": \"Vulnerability Srbds:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Tsa:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Tsx async abort:\",\n \"data\": \"Not affected\"\n },{\n \"field\": \"Vulnerability Vmscape:\",\n \"data\": \"Mitigation; IBPB before exit to userspace\"\n }\n ]\n}", + "kernel": "Linux snafu 6.12.107+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.107-1 (2026-08-29) x86_64 GNU/Linux", + "compiler": "cc (Debian 14.2.0-19) 14.2.0\nCopyright (C) 2024 Free Software Foundation, Inc.\nThis is free software; see the source for copying conditions. There is NO\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb] /home/rob/projects/splinterdb", + "space": "Filesystem 1B-blocks Used Available Use% Mounted on\n/dev/mapper/snafu-root_crypt 830432124928 714661253120 73512001536 91% /home/rob/projects/splinterdb", + "sources_sha256": { + "sources/hash_range_benchmark.c": "1353f2f2acbff19dc7551a385ec4aa3b6cf6f2eba39ce19ea9cbc53c20a61add", + "sources/kvbench.c": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9" + } +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/inventory.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/inventory.json new file mode 100644 index 000000000..7a34755c9 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/inventory.json @@ -0,0 +1,310 @@ +{ + ".clang-format": { + "sha256": "53261471487a168ba8e76c97b8be1fd4c1401d1574546c42c0e3129e6450e1f3", + "bytes": 88 + }, + ".gitattributes": { + "sha256": "42feaff44a7ea6670986543a06e6343020f20eef449644e4c17836a0b090e5ca", + "bytes": 36 + }, + "README.txt": { + "sha256": "d6dc9a6df5ad440d008e56d09f95c4f6f6daa52d322dee950774b6891d9d282e", + "bytes": 979 + }, + "build.py": { + "sha256": "57dd56263b31384b4d5836fe62a11bf926c336a04052c787d320a0f81d225e36", + "bytes": 2821 + }, + "control-drift.json": { + "sha256": "fb4352d0127f51fdcf0ebdc8e6624e3dd67320589793013e34e3dfddac242fbe", + "bytes": 2310 + }, + "disk/00-main.log": { + "sha256": "9f61e4f6860d5580f8bec3e1a4361f6d4ca9cd2dd27b78fdc7d6267c9b1a5ab9", + "bytes": 2072 + }, + "disk/01-filtered-ranges.log": { + "sha256": "89e9d282938af6c1e4b53b0027977eee762ddc5cbb1be7e80de10e870c83c115", + "bytes": 2083 + }, + "disk/02-disk-format.log": { + "sha256": "7f54b7bb8c6185efbb89b163e0f99c645e4a42e553626ff5e93f1bf402e77c5c", + "bytes": 2085 + }, + "disk/03-disk-format.log": { + "sha256": "8ae38233dc62dc3925b241ecdddd254d0702f4801f0e9a9b23d594d9d699de32", + "bytes": 2085 + }, + "disk/04-filtered-ranges.log": { + "sha256": "42e56e3649c71c008d56d32ec120a20be23ca3a581db89a31ae71f0ac145ced1", + "bytes": 2085 + }, + "disk/05-main.log": { + "sha256": "b135d58d2e445ca98364e6988afc035043450287f10df92436d4ce7709daae64", + "bytes": 2071 + }, + "disk/manifest.json": { + "sha256": "c8b85267f5b682ce04ab5a3d17a2746dcbe6ed7e7676d3222b10f13925f1b6c8", + "bytes": 15537 + }, + "disk/results.csv": { + "sha256": "556da1f3fed946fe450b74df68483877882df9a7f6f051c6eeee64a2a5e981c5", + "bytes": 6344 + }, + "disk-format-build.json": { + "sha256": "49a32d03202f3def8a447a360808639bc5727c93234c01190ae85f800e8250cb", + "bytes": 12721 + }, + "disk-format-kvbench-build.log": { + "sha256": "031d29ed8c303df98848d42071f1e943bd1dbb5c146b1d0ad197b09946a84f51", + "bytes": 146 + }, + "disk-format-libs.log": { + "sha256": "044d872dbaf683cf11414b179ac193d2956152e77ff4ae935361728111be4469", + "bytes": 60 + }, + "feature/collisions0-00-main-application.log": { + "sha256": "10292d4bc75c45c673d702185fae0a305bc285a4eb02b10174ef328653ef0774", + "bytes": 507 + }, + "feature/collisions0-01-filtered-ranges-predicate.log": { + "sha256": "9aba8a7439cf3bdc77877e5eaca8ff6e4126415cef11d101d3b7e16d3a4167b0", + "bytes": 496 + }, + "feature/collisions0-02-filtered-ranges-pruned.log": { + "sha256": "8043099796e133fcc3255aea7e458d3c1135ef37512c8bdb5992ef756c111f56", + "bytes": 484 + }, + "feature/collisions0-03-filtered-ranges-pruned.log": { + "sha256": "de5d3f3af544626f18c0d02113071c8165a11a463fc36b586f9a2314f25a8143", + "bytes": 483 + }, + "feature/collisions0-04-filtered-ranges-predicate.log": { + "sha256": "d68e891a8e23bcb9ee7f71fda245664250652e07082c5bb339ea5d2167955041", + "bytes": 496 + }, + "feature/collisions0-05-main-application.log": { + "sha256": "53d9a4fd51c5a58c4c301bb351c6d14ba8c50c8367c6a119448e682b30155a07", + "bytes": 508 + }, + "feature/collisions1-00-main-application.log": { + "sha256": "c8bcc2b65f82264698d758553be23e1232314a7260d3347ff188a67b807a6a86", + "bytes": 508 + }, + "feature/collisions1-01-filtered-ranges-predicate.log": { + "sha256": "072a3c10cc9224692c08aff4e8caeee33fa7b382deaf79b06c0d4cf76ebbe73d", + "bytes": 496 + }, + "feature/collisions1-02-filtered-ranges-pruned.log": { + "sha256": "da3ef1b2ac9790b729aeb1617d2937e3e29620c2bdbe501636682a71d3f3c1f8", + "bytes": 484 + }, + "feature/collisions1-03-filtered-ranges-pruned.log": { + "sha256": "d9fac9cd7197430601b68c2d4c821b7b341e4b0fb3fd91614a812f7e16cadf82", + "bytes": 484 + }, + "feature/collisions1-04-filtered-ranges-predicate.log": { + "sha256": "c8c78b8cd8e1db7850eed6fd70e3287ed84d20d6ae6f40f4b7977feee6b1a249", + "bytes": 496 + }, + "feature/collisions1-05-main-application.log": { + "sha256": "9c87d7d76d98c761ae9488e3115dd77b800145b2f473bf4dbc8d9b9b274913f2", + "bytes": 508 + }, + "feature/manifest.json": { + "sha256": "36363b92ebbd243298ab2761d652e75a638c2d08ce83c8743e906c2d14c10f8a", + "bytes": 38584 + }, + "filtered-ranges-build.json": { + "sha256": "6b8f0526e37bd80eaa65707dc2c8f9ed7990222d42c6ac238d2228829cf3d06e", + "bytes": 14554 + }, + "filtered-ranges-hash_range_benchmark-build.log": { + "sha256": "76caa1cfc19a040cca84cc4cb969c62285b5edebd75eec3ee9843f786499fbd0", + "bytes": 146 + }, + "filtered-ranges-kvbench-build.log": { + "sha256": "031d29ed8c303df98848d42071f1e943bd1dbb5c146b1d0ad197b09946a84f51", + "bytes": 146 + }, + "filtered-ranges-libs.log": { + "sha256": "15502645730b7b255f14f03bbfb7506eb3c1a941bc63bb83aec5048374b1c146", + "bytes": 132 + }, + "final-provenance-check.json": { + "sha256": "6b005ad291867b61e9663a3719c3d2658f559651661b00e81abdb5bfcaa4a6ae", + "bytes": 175 + }, + "fixtures/disk-disk-format.csv": { + "sha256": "a73dc3538ea8116fd2ed387682695a0f9687a81af2bb403a6b9f6f3cace40aa4", + "bytes": 623 + }, + "fixtures/disk-disk-format.log": { + "sha256": "5f84764c215fc98079a2d169731bb5977deff08a11d90dbde6acbedc99a058cd", + "bytes": 816 + }, + "fixtures/disk-main.csv": { + "sha256": "27e258c195279665af362352c936388727a41d57d9f838daf87e1d938b81e694", + "bytes": 616 + }, + "fixtures/disk-main.log": { + "sha256": "bab82f1f8713052eae6adc0cc1eb5443fa5549f7564f954df17bb56960dd4adb", + "bytes": 796 + }, + "fixtures/fixtures.json": { + "sha256": "8575413c9697c1c5cdede23da9dbd1504fb5ab82efa0263af7f19a819c83cd61", + "bytes": 1287 + }, + "fixtures/manifest.json": { + "sha256": "8859bc475492cca33e07a6f3a09183c8d3cc5bc35e491e8a115adebdaadc4b3b", + "bytes": 9774 + }, + "fixtures/resident-disk-format.csv": { + "sha256": "ab34e8caa57509111da48d5ee6fce132d301cf21d2a66132ce74f69b20059caa", + "bytes": 602 + }, + "fixtures/resident-disk-format.log": { + "sha256": "33392dac5f25db63688655b4b1c55f5aafd389906075f6fde17b139f10b9f087", + "bytes": 813 + }, + "fixtures/resident-main.csv": { + "sha256": "4e03b793ddf7e54e9505029c2fafd5666f0bdb1a3f9667500927c08a56ccac52", + "bytes": 595 + }, + "fixtures/resident-main.log": { + "sha256": "368e3edc4cf82a0853dfd119f38e8bc8d71b99b2bf4a605eb8de4aea0d2f016d", + "bytes": 793 + }, + "host.json": { + "sha256": "e0c14ac420cd3b5fbf421e0c63c8d9057da51af2482bf2e8949c57e9c2b35c58", + "bytes": 6057 + }, + "library-build-configs.json": { + "sha256": "cb43e591f0d73569e6bc43a06311b180fca13bed85c963c08038b8b68fd57a8d", + "bytes": 3037 + }, + "main-build.json": { + "sha256": "dedc407e542854ba09e209a33cced704339e56028e7b293c6622d7455c4faa8c", + "bytes": 13948 + }, + "main-hash_range_benchmark-build.log": { + "sha256": "031d29ed8c303df98848d42071f1e943bd1dbb5c146b1d0ad197b09946a84f51", + "bytes": 146 + }, + "main-kvbench-build.log": { + "sha256": "8ec756edaefc86649d0dc3e4ee61fa5d8b155cd88d67daae04e9befcdf25020f", + "bytes": 146 + }, + "main-libs.log": { + "sha256": "044d872dbaf683cf11414b179ac193d2956152e77ff4ae935361728111be4469", + "bytes": 60 + }, + "package.py": { + "sha256": "17ffce499143d0ddd446f6b30850ceef765e308726c9d8dccead11341abe3b3a", + "bytes": 2524 + }, + "profiles.json": { + "sha256": "cf3209f2ffcec1d7693e89ebb87d157c7487f96c6e013cffeb77089c23f36fc6", + "bytes": 594 + }, + "repeat-resident-disk-point8/00-disk-format.log": { + "sha256": "686d48f5ff9785b98e8a208cf385687f83560a8e040cfb2432c918f985a83102", + "bytes": 1190 + }, + "repeat-resident-disk-point8/01-main.log": { + "sha256": "8a5b0e650f191e1f5eb78daefde09b9367092243204f9bbfd4a9e43322f78b08", + "bytes": 1176 + }, + "repeat-resident-disk-point8/02-main.log": { + "sha256": "e7b311aa54d9a7c06c80061901258d86e0a64c1a0dee4e53bd6ddd415d803240", + "bytes": 1176 + }, + "repeat-resident-disk-point8/03-disk-format.log": { + "sha256": "16d97b1e08eb089ea70f86720649920d278250e7155c706ce9478f886afa8af8", + "bytes": 1189 + }, + "repeat-resident-disk-point8/manifest.json": { + "sha256": "ef33dae2c3b7fbd453b2e29251984a2b8488c97edd5254feb5b0c80fcffd9bc3", + "bytes": 10693 + }, + "repeat-resident-disk-point8/results.csv": { + "sha256": "ea56418788e2b65e40e6435cac79402fdc885e37062db72eb7fa707a4e8478b2", + "bytes": 2160 + }, + "repeat_point.py": { + "sha256": "1baecd91c684a9296201ec9786c901bd00db22cad34d3dc7e8c43d73f26daf80", + "bytes": 1736 + }, + "report-generation.log": { + "sha256": "7aa25116b6bd318044a6dd0940e2bd3edd4b07e5d5bb3d946aae7a16dd0915a8", + "bytes": 5753 + }, + "report.py": { + "sha256": "c0fbdbf3a3a4c1abc45c236eedbc3ba20942e50a8bc1d886a14f5f1b66a26401", + "bytes": 9629 + }, + "report.txt": { + "sha256": "7aa25116b6bd318044a6dd0940e2bd3edd4b07e5d5bb3d946aae7a16dd0915a8", + "bytes": 5753 + }, + "resident/00-main.log": { + "sha256": "94b181b0dea1e7faf167567fda59b8f7b5f49d322e0a15f881b2aea75af05274", + "bytes": 2440 + }, + "resident/01-filtered-ranges.log": { + "sha256": "4dd228f28c7d054085fb4ad8271b3525dce900507706c43b3ca3d479c49fb572", + "bytes": 2453 + }, + "resident/02-disk-format.log": { + "sha256": "70fa3a858470944e9ecc26ab90fa085b308ad6f316156e2eee0be2776d5856ae", + "bytes": 2453 + }, + "resident/03-disk-format.log": { + "sha256": "1ead91833e3cfbdfd752896ca690912a741e0a86085c25f70273074f019d17e0", + "bytes": 2454 + }, + "resident/04-filtered-ranges.log": { + "sha256": "0f4b57dee27b33f0abfd8d8439c07b8b890821d920a7ac355536b50c2d9f12b0", + "bytes": 2451 + }, + "resident/05-main.log": { + "sha256": "14870934fcc0ac5f3b2dc9a9133bb128b705c6c7e45abbdb0338db7704233e77", + "bytes": 2441 + }, + "resident/manifest.json": { + "sha256": "ee9dd59cc11383acbb34f79424c2c2b69924b6fb91c5d844a3ba50b90a854d0a", + "bytes": 15823 + }, + "resident/results.csv": { + "sha256": "1d956aef17a9b03cd6571d59b24609dfd4ead4260a9463e47dc55cb7500f05f0", + "bytes": 7353 + }, + "resident-summary.json": { + "sha256": "861168a517b770edfb764fe6e64b709834aca8b9a6fe0354b5621379a62dc818", + "bytes": 1929 + }, + "run.py": { + "sha256": "ad1a38ada90f6f275d482205f75eb359fda6bea5d21a58547a4724472db64da1", + "bytes": 9967 + }, + "sources/hash_range_benchmark.c": { + "sha256": "1353f2f2acbff19dc7551a385ec4aa3b6cf6f2eba39ce19ea9cbc53c20a61add", + "bytes": 8467 + }, + "sources/kvbench.c": { + "sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9", + "bytes": 49348 + }, + "status.json": { + "sha256": "0f4ee3e3a2cb8029d28a74ae7f8555abae62825d9e63c0b3e324966156e28b1e", + "bytes": 305 + }, + "summary.json": { + "sha256": "40c1fc452e10118aff491e3c6f0bd553d818fe19efca69f6e6ca44da08c6b780", + "bytes": 70895 + }, + "timing-ready.json": { + "sha256": "38124e74cb284c48384ca208e361d4d55fe4fb30c8057a25db2d00c1feaffee0", + "bytes": 348 + } +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/library-build-configs.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/library-build-configs.json new file mode 100644 index 000000000..061e6a5cf --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/library-build-configs.json @@ -0,0 +1,23 @@ +{ + "main": [ + { + "path": "/home/rob/projects/splinterdb/build-main-0fa3bd6c/release/build-config.fb6bc8eeb8390429253883c09491f5bd", + "sha256": "c6553d71650096982bd788f5082929c46d759fda3de270f4deb9b10646715008", + "contents": "BUILD_ROOT=build-main-0fa3bd6c\nMAKEFLAGS= -j4 --jobserver-auth=3,4 --jobserver-auth=-2,-2 -- BUILD_MODE=release BUILD_ROOT=build-main-0fa3bd6c\nBUILD_MODE=release\nQT_ACCESSIBILITY=1\nCC = cc\nDEPFLAGS = -MMD -MP\nCFLAGS = -D_GNU_SOURCE -ggdb -Wall -pthread -Wfatal-errors -Werror -Wvla -DXXH_STATIC_LINKING_ONLY -fPIC -DSPLINTERDB_PLATFORM_DIR=platform_linux -O3 -ffast-math -flto -DPLATFORM_MEMORY_FAULT_INJECTION=0\nTEST_MEMORY_FAULT_DISABLED_CFLAGS = -DPLATFORM_MEMORY_FAULT_INJECTION_DISABLED_IN_THIS_FILE\nINCLUDE = -I include -I src -I src/platform_linux -I tests\nTARGET_ARCH =\nLD = cc\nLDFLAGS = -ggdb -pthread -O3 -ffast-math -flto\nLIBS = -lm -lpthread -laio -lxxhash\nAR = ar\n" + } + ], + "filtered-ranges": [ + { + "path": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges/build-filter-predicate/release/build-config.fb6bc8eeb8390429253883c09491f5bd", + "sha256": "121e482967423b5268cfb4b392e4e64ff3690ec2e07b6ad46de5a7a251ec1c2a", + "contents": "BUILD_ROOT=build-filter-predicate\nMAKEFLAGS= -j6 --jobserver-auth=fifo:/tmp/splinter-filter-predicate-libs-release-tghehp69/GMfifo6 -- BUILD_MODE=release BUILD_ROOT=build-filter-predicate\nBUILD_MODE=release\nQT_ACCESSIBILITY=1\nCC = cc\nDEPFLAGS = -MMD -MP\nCFLAGS = -D_GNU_SOURCE -ggdb -Wall -pthread -Wfatal-errors -Werror -Wvla -DXXH_STATIC_LINKING_ONLY -fPIC -DSPLINTERDB_PLATFORM_DIR=platform_linux -O3 -ffast-math -flto -DPLATFORM_MEMORY_FAULT_INJECTION=0\nTEST_MEMORY_FAULT_DISABLED_CFLAGS = -DPLATFORM_MEMORY_FAULT_INJECTION_DISABLED_IN_THIS_FILE\nINCLUDE = -I include -I src -I src/platform_linux -I tests\nTARGET_ARCH =\nLD = cc\nLDFLAGS = -ggdb -pthread -O3 -ffast-math -flto\nLIBS = -lm -lpthread -laio -lxxhash\nAR = ar\n" + } + ], + "disk-format": [ + { + "path": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format/build-pr-0fa3bd6c/release/build-config.fb6bc8eeb8390429253883c09491f5bd", + "sha256": "a13a9ce817a53a672d771eaec5db5b020cafbdea4812af5640341a01f7361aee", + "contents": "BUILD_ROOT=build-pr-0fa3bd6c\nMAKEFLAGS= -j6 --jobserver-auth=fifo:/tmp/splinter-disk-format-pr-release-build-cf558ann/GMfifo4 -- BUILD_MODE=release BUILD_ROOT=build-pr-0fa3bd6c\nBUILD_MODE=release\nQT_ACCESSIBILITY=1\nCC = cc\nDEPFLAGS = -MMD -MP\nCFLAGS = -D_GNU_SOURCE -ggdb -Wall -pthread -Wfatal-errors -Werror -Wvla -DXXH_STATIC_LINKING_ONLY -fPIC -DSPLINTERDB_PLATFORM_DIR=platform_linux -O3 -ffast-math -flto -DPLATFORM_MEMORY_FAULT_INJECTION=0\nTEST_MEMORY_FAULT_DISABLED_CFLAGS = -DPLATFORM_MEMORY_FAULT_INJECTION_DISABLED_IN_THIS_FILE\nINCLUDE = -I include -I src -I src/platform_linux -I tests\nTARGET_ARCH =\nLD = cc\nLDFLAGS = -ggdb -pthread -O3 -ffast-math -flto\nLIBS = -lm -lpthread -laio -lxxhash\nAR = ar\n" + } + ] +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-build.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-build.json new file mode 100644 index 000000000..66f864f79 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-build.json @@ -0,0 +1,194 @@ +{ + "tree": "/home/rob/projects/splinterdb", + "head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "library": "/home/rob/projects/splinterdb/build-main-0fa3bd6c/release/lib/libsplinterdb.a", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "library_build_command": [ + "make", + "--jobserver-style=pipe", + "-j4", + "BUILD_ROOT=build-main-0fa3bd6c", + "BUILD_MODE=release", + "libs" + ], + "source_files_sha256": { + "include/splinterdb/data.h": "6d8984e702da446c2fe145eb2f6a576ca040cfd5740733c405d0d7387b243ec2", + "include/splinterdb/default_data_config.h": "179386f3e224d322a9d7ac97afd929292994a1d890a3254705f9ebea525bce32", + "include/splinterdb/platform_linux/public_platform.h": "c5ded82c5b5bb5bbcedf06a66951eb98dc8892f7b3c4f8ce07c474db98ac68a0", + "include/splinterdb/public_platform.h": "4ce9da3fea9a54fb7bb613ec1458f11f043525efb7d0bc55182f7183dd4a4f6e", + "include/splinterdb/public_util.h": "4e8f6b30408fb4174894d4ba809acff4e96f559370c7749b65bb3724a853603f", + "include/splinterdb/splinterdb.h": "9266891f702fc4f0855b26e08d73240127bf64aa1b19913f73f2736bfc486969", + "src/PackedArray.c": "97b64eb0a4f550e55f3385d39d729a878fbc581909ea16ec3b8987a7f46604cc", + "src/PackedArray.h": "e46f854b487a8b130f0613efcab11836012d27c8aacf9ca9e76a2de1a7eb6874", + "src/allocator.c": "96c7f0a5e0808719cf114d267c84da1f057b5eed748b0672d60686d7f130b96f", + "src/allocator.h": "bc0621be7903d64e25059140005dafeea3f2a4a4ae0e658686501e52e4be934f", + "src/batch_rwlock.c": "c043d9e5ead75cd9e1266a6f51fb555012269f3822d7f0233dc2abbd962c3aae", + "src/batch_rwlock.h": "0ff5f0bd7fdf5f9a63ca2f77c387d574d8009ad726b47283c93566cf06bc0927", + "src/blob.c": "7d46871423a15992f697cabb363316183cb842a11c26017f8566267174e5ae45", + "src/blob.h": "8f68d2f67de8a9d81a422b28d47aa636cc7234fccc3dbabe59f2355bc361e93c", + "src/blob_build.c": "e484e9c14fd8a9b3c1d211485c35931c944fc66ac4ee78ea20ffc5b3f074b10b", + "src/blob_build.h": "38176881cf9159247e6f578bbc7abb725cfa969c493715df7d1388cb7cbfd119", + "src/btree.c": "f637081a194302c584cccfd8d717253817794b34bbca754bd2c9e597acc534b9", + "src/btree.h": "6ed244b6731c490f224c7b5fea7b5a9dc27eca1aa808e6dab6b44a0c7b511638", + "src/btree_private.h": "43693dd3048ae050b3358ccfc9b91371a3d9dff4f3d6d5c8b07445897dee09f6", + "src/cache.h": "d4c1e133d6423fff324d0ebcf5c95e13647c0022a3458935156746b90d816d6d", + "src/clockcache.c": "e0ef2e19a8280849df4ee80d9b8a54a0faaf56a13faefe7a91d54f5b7269f933", + "src/clockcache.h": "68f4e7b019d0440f85929bfc0da1891ce941318ee7dfd14a3d4d923e3d4e72d8", + "src/core.c": "64fce490235c4c1885978eb7f840efe26f99ecea858763ded2d3453a82ce41de", + "src/core.h": "1e5b6eb93e004320987698f9b71980f7e1876626de7db85411c972f9fb482784", + "src/data_blob_build.c": "13475faa56bba94ee26ef7fec4e4da8a5045e0efe510a63291fd5e2697cb5d22", + "src/data_blob_build.h": "dd7f263a0b1138325b211c561ba7289183a2cc8472978cba3ca430295e11e5f3", + "src/data_internal.c": "0a42dd477aaa6ad6c4910841045c4c768ad8d248c0126c64c51035f53e769cde", + "src/data_internal.h": "98b4ebe77576c4cbb7dd9984ca5e297851175494062073f1a4c137c5afb8b587", + "src/default_data_config.c": "161352f5247978f3d57033818d05a79d0a1a0d0ffac2bc9f25f4f33eabca2227", + "src/histogram.c": "24e09dfabd6085021d8a06ef9b10f6e85b24914d95831be359df09f8a9729aff", + "src/histogram.h": "7ccfde6d3ada7ead18fe477c06a6b021db83ca8c041ab0998ca023b945a3c1c7", + "src/iterator.h": "2fcddeb9fa6bd49f4e3da933224219bdb830f9e2f11638ea1ed13eba534fa27f", + "src/log.h": "3e7eb447259e7901b915b52b7545e2d5088475171af1716dd9c849dab4c3d1b9", + "src/log_data.h": "06ab1d39b77abb01ec66a901fa52354908be26db8daef52899f5c03a6db2462e", + "src/lookup_result.h": "7528b988a28ef91b6bd7a4fa10dfa51b9ce2475e801b3b6a8bf30adf88f1ef16", + "src/memtable.c": "7db3b4754497c73b4071c7e7e591cc76fc226197fb70d98fb46caf139b3953f4", + "src/memtable.h": "244cac7cf304df85a25c8bede2df106d6f840d65bbd1a3551b3d9f2ed9129fc1", + "src/merge.c": "df98421e597d1edeac871755f0473055b590791af8efad495575d20260e46a12", + "src/merge.h": "c17e2a0b6272ec52e6d1e80acd9df7dc85221f186e6c60e323c37b467dc5c41a", + "src/mini_allocator.c": "d9444da13fca2000f4128500b84be2b778300a4befc93f72e52b071b83c6491e", + "src/mini_allocator.h": "8d5242f5e4df5443baeab7a494a3c3b2f1491216abfd914c6dbc273370e45683", + "src/notification.c": "54af1fe6af2e5fc84fbee21a74ff55069a8aa7e4ec7f9d8c9485e85e91669b58", + "src/notification.h": "f4df33131f407e6a957602a05c72f4bca7efed98b1a39134f94bf889a6ca5f29", + "src/ondisk_ref.c": "2ace8cbbce3f80a54ff7762ad03a79a564665fa660394f1d3446235a3e40ed7b", + "src/ondisk_ref.h": "643325c8b4f16fab41d8c884c97bdfcf6c86e5178398508fa0f81b6b0606926d", + "src/pcq.h": "862cd346b0097014ccae797dc1463e7b3fc387f409319d5adcef7613b6f82f84", + "src/platform_linux/async.h": "41383b57e21f0f51d737749a4ce7c2c5330ac9363e742330625c9aedd4f13a52", + "src/platform_linux/laio.c": "0ce628806e6896070359a9e39d069f326f460ab697c6c17ff17d171614290881", + "src/platform_linux/laio.h": "f3987e22f1fd7a9d6b46c79f84e75559aec5410239a13d52a7a5343ba792b28e", + "src/platform_linux/platform.h": "f1c3d7c5bcd089ed7a7df129d3b8d00d7617de9a7aa6f9d1f7deeb739281e768", + "src/platform_linux/platform_assert.c": "112e803b6db03dd8c14a3bfc3f1139bbcc4b81f1d5e24a232b7b99ce89146c2d", + "src/platform_linux/platform_assert.h": "751dccceac9ce4a97d25d441b5216ced5ee5f079a13c9ccd43551b0994aa7cc2", + "src/platform_linux/platform_backtrace.h": "ef13ccbf4ae4747b0de244a37c026ad8769018d84f9fc4d6493db43d9714358c", + "src/platform_linux/platform_buffer.c": "b790d645b3dc5f638da871bf9611865dc700927f3b594a1d645b0350bd1bd027", + "src/platform_linux/platform_buffer.h": "72e016dbb19b8250378b3a5c79fbebf9c3d72a31d07a232f36431e04f0e4afa6", + "src/platform_linux/platform_condvar.c": "e595f69194accb3f3460d6fc9a4a9144bd7f495dbeb77f11bf3616dc0e9b03ec", + "src/platform_linux/platform_condvar.h": "a4ee02ae5069637bce4cae07f3ed9869925f17a9e6e1b04d69baedfc148fd493", + "src/platform_linux/platform_hash.h": "f69f5ed6218bfe62c62718b387e1c110e9fca46c07603d4d51ccc4fc782b09ab", + "src/platform_linux/platform_heap.c": "72bb2753f525e56f8c398988a787feb38f02efaa749e5d57cb476f967269c479", + "src/platform_linux/platform_heap.h": "921f3113d20b44c36817c284702e965e01769596ff012edd6ceaa70cbdbbd10a", + "src/platform_linux/platform_io.c": "875a7331f8bfa46e9a6961f50acb40b107d40883839c94def52a289a521c66be", + "src/platform_linux/platform_io.h": "69ae9556daf1c7a69e007abd494bf9c86ec81c08bca1e032663009cef95b2fac", + "src/platform_linux/platform_log.c": "318bd17c858475a3a91c5e9641b52808f7c4194e1c6fc9a7d801392b04c49538", + "src/platform_linux/platform_log.h": "569484386150fcfbf5dde722c5b8e34d67d65be3454e3e009cb9df41f11efea6", + "src/platform_linux/platform_machine.h": "c02745313c7268dbe99efa2b93dd4d37640b5a7aa2e88c78fd5b73742f252c7d", + "src/platform_linux/platform_mutex.c": "7ec1dd6a5397217d28fd85c10648d29a70bf66649f59f76d7387236031b15bf3", + "src/platform_linux/platform_mutex.h": "688991d6a7dc57a7a88f565541561344cdfbb1733ca6073f1655d6c436ed6544", + "src/platform_linux/platform_random.h": "184ac9da9c38a6c10275891be61ccb91beaf2234ebb9ce046d059cb16824d68f", + "src/platform_linux/platform_semaphore.h": "9fdb7cfeaf36c8bf9aa346a16466ed6bc50bea76fdb5722c86001563d55bedfe", + "src/platform_linux/platform_sleep.h": "34aeaea35133bf33e47eb1d9ace429297644e13bc100d90ebbd205bceb58d3ab", + "src/platform_linux/platform_sort.h": "d374788510f94a8c3b86ff6bfa1f6a3ff8ac33168eb26f6f7774984252f6d01e", + "src/platform_linux/platform_spinlock.h": "7b3f0409109f3341f2287fe7cd8fb69faac4a74b009a44c1eb83220f71e94de0", + "src/platform_linux/platform_status.h": "2749285ea1322e5c1639019b728fde4b87b0cae6f71a7fc16e420bd592d38c00", + "src/platform_linux/platform_string.h": "fa41faf9c6661c62ece783740250232709656b0a8677f542e390a5f3b0d834a6", + "src/platform_linux/platform_threads.c": "789465a2c6ac7ac501e5878f429cc0dd9b163145a522e247a4833e3de7af8f2a", + "src/platform_linux/platform_threads.h": "653e070115131756e6aaffc3c2764a6437fb39ff96df4afc8a01e6e751c9d622", + "src/platform_linux/platform_time.h": "f212a08cf8432693711cecfd389f9ed3d0a70e0c056dc062321bd11c4a8cbec7", + "src/platform_linux/platform_trace.c": "4a68e89f214920dd2762d072a0ad8e7c80d9b1733a64e4a7e3dd4bb04b91c352", + "src/platform_linux/platform_trace.h": "345f7283b01e5a9b4d5da7bea66ad44ff61f53155417c97990e23bb8fedadf33", + "src/platform_linux/platform_typed_alloc.h": "42c1d2bbd56c183d11768a5f43d440cf3c1bfe329c4352afa6acd3bc2b191a8e", + "src/platform_linux/platform_units.h": "1cb22a07b33ec8b58f9381438f480f561d12b838f315a117cc1f979e95b1ef65", + "src/platform_linux/platform_util.h": "4fa2702b3919eb1a7efcf29457f5ee9a3d7a7b4c38fed4606264562022fa2638", + "src/platform_linux/poison.h": "a32dfa445439eb08913e519117e7b11f72b2c3550577a74769ed935a792c42c5", + "src/platform_linux/shmalloc.c": "e4631d63a9db1ad58472ad3d034d081f2bb9a97f34c5898f7c848dda923727f8", + "src/platform_linux/shmalloc.h": "081a82473f6021f8ed628e79f8ae5dc7104988981f8c3b25749ea42c8158113d", + "src/prefetch.h": "9d55d9dad64e3795c81856bc38688936e4e74e488dfe2d29a1f5438ad000df42", + "src/rc_allocator.c": "a145d39af39caf9dec6f1e1e24b6606ed615587d8dee52a23e0df410b2d07545", + "src/rc_allocator.h": "65f01decb7244b8f6a6c9e7acb47d4ccb4f2f3f369461e70aff33211f03ee39b", + "src/routing_filter.c": "3e722a880217cc4f5831f162dd6b3853a5d0d3f67ca3d3421b2144650657f73a", + "src/routing_filter.h": "579a3a4cc00b9cb125479b23eaa93e4a8309a195fe4082d988b629d4b95a2c91", + "src/shard_log.c": "31449c056fb90448f08954acd7049e8ce7f2a9a6e349f5b78ec1c134e5f9465d", + "src/shard_log.h": "07c5867b959eefd35e3e9f08e6c015a594f6331850844590e9c2b1126cd2716b", + "src/splinterdb.c": "40aa799ce9defb7d731e4ab3ba2ed8f4860bcc89a0d9641a4a004db063055dc8", + "src/splinterdb_tests_private.h": "579031767b8869f49cf1e3fc94caeec13a1362dfd047782538df40b281c93414", + "src/superblock.c": "34dd4c724e53c98f170c024c70dd22dae0a466806c167134200f92128420dfba", + "src/superblock.h": "72cd00ce0a7c43d1c25905d6bd39a400fa3245c133a3f559c58ae34a962157f2", + "src/task.c": "470418c0da14795c4697c51ef7b22e128b691c1c24a704791eea09a10b9373a4", + "src/task.h": "ebcc3bd29cef9ee8718c6ff0186863fff0dad03bcfaa43e5eacaf83c1933d01d", + "src/trunk.c": "892373ec383717289a6220b4351b670637e7c41877a97eb0edcb27a9e4e270f2", + "src/trunk.h": "05c5c5313f5a42b68482d03b58d2f17081c6da468a0ce888b0b0b38cc069baf2", + "src/util.c": "969df408c4c393f73feb05dfa37ebe1ed8be627cef630061830e4992c4c1fd9c", + "src/util.h": "9fa455f5a6268fa4a6864d444849b0872391af00512932c1587363cfd38304ef", + "src/vector.h": "6855dd0430f1a868d098797cbd4f3453ab542b5f6fbe716c79871124bee200a9", + "src/writeback_set.c": "cb485716f57a3183ce82543848edaf61f6568c9ff0445ea5ff989f95c00848ff", + "src/writeback_set.h": "68d6f589d24719a09e2680b05a1eb7e36c22b1fab978768f5b54f958f667aec9" + }, + "binaries": { + "kvbench": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-DKVBENCH_CLOSE_HAS_FORCE", + "-I", + "/home/rob/projects/splinterdb/include", + "-I", + "/home/rob/projects/splinterdb/src", + "-I", + "/home/rob/projects/splinterdb/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/kvbench.c", + "/home/rob/projects/splinterdb/build-main-0fa3bd6c/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/kvbench.c", + "source_sha256": "45cf9d2e1f74e90719885955d5a78651cc5c1d97cda2daed534a3c0cb07f48a9" + }, + "hash_range_benchmark": { + "command": [ + "cc", + "-D_GNU_SOURCE", + "-ggdb", + "-Wall", + "-pthread", + "-Werror", + "-DXXH_STATIC_LINKING_ONLY", + "-fPIC", + "-DSPLINTERDB_PLATFORM_DIR=platform_linux", + "-O3", + "-ffast-math", + "-flto", + "-march=native", + "-I", + "/home/rob/projects/splinterdb/include", + "-I", + "/home/rob/projects/splinterdb/src", + "-I", + "/home/rob/projects/splinterdb/src/platform_linux", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/hash_range_benchmark.c", + "/home/rob/projects/splinterdb/build-main-0fa3bd6c/release/lib/libsplinterdb.a", + "-laio", + "-lxxhash", + "-lm", + "-lpthread", + "-o", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/hash_range_benchmark" + ], + "binary": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/hash_range_benchmark", + "binary_sha256": "7f5895c21143d36b2ec29e65421820637de953fb332340c1b0c63489bac15a9f", + "source": "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/sources/hash_range_benchmark.c", + "source_sha256": "1353f2f2acbff19dc7551a385ec4aa3b6cf6f2eba39ce19ea9cbc53c20a61add" + } + } +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-hash_range_benchmark-build.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-hash_range_benchmark-build.log new file mode 100644 index 000000000..91b66fe35 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-hash_range_benchmark-build.log @@ -0,0 +1,2 @@ +lto-wrapper: warning: using serial compilation of 13 LTRANS jobs +lto-wrapper: note: see the ‘-flto’ option documentation for more information diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-kvbench-build.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-kvbench-build.log new file mode 100644 index 000000000..fb5b92627 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-kvbench-build.log @@ -0,0 +1,2 @@ +lto-wrapper: warning: using serial compilation of 14 LTRANS jobs +lto-wrapper: note: see the ‘-flto’ option documentation for more information diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-libs.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-libs.log new file mode 100644 index 000000000..8548d79e3 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/main-libs.log @@ -0,0 +1 @@ +Checking for mismatched config...No mismatched config found diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/package.py b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/package.py new file mode 100644 index 000000000..303e225ec --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/package.py @@ -0,0 +1,32 @@ +"""Prepare reviewable evidence without databases, executables or libraries.""" +import hashlib, json, shutil +from pathlib import Path +HERE=Path(__file__).resolve().parent +OUT=Path('/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013') +status=json.loads((HERE/'status.json').read_text()) +assert status['complete'] and status['errors']==status['violations']==0 +package=HERE/'package';package.mkdir(exist_ok=False) +for p in HERE.iterdir(): + if p.is_file() and p.suffix in ('.py','.json','.txt','.log'): + shutil.copy2(p,package/p.name) +shutil.copytree(HERE/'sources',package/'sources') +for mode in ['fixtures','resident','disk','feature','repeat-resident-disk-point8']: + target=package/mode;target.mkdir() + for p in (OUT/mode).iterdir(): + if p.is_file() and p.suffix in ('.csv','.json','.log','.txt'): + shutil.copy2(p,target/p.name) +(package/'README.txt').write_text('Fresh filtered-predicate and disk-format screening against main 0fa3bd6c.\n' + 'OPEN PERFORMANCE CONCERN: disk-format resident point8 throughput fell 4.39% in the broad screen and 7.75% in the longer repeat, with non-overlapping sample ranges and zero disk reads.\n' + 'The cause is unresolved. Code review is appropriate; performance-neutrality and merge-ready claims are not supported. The broad and repeat samples remain separate.\n' + 'See report.txt for methodology, sample ranges and limits; summary.json for CPU/I/O and query counts.\n' + 'Each profile build JSON records source, library, executable hashes and compile commands.\n' + 'Each workload manifest records exact invocation, storage, verification and process resource usage.\n' + 'Sources are frozen measured bytes. Historical absolute paths identify the original workspace.\n' + 'Databases, libraries and executables are intentionally not embedded. Fixture creation logs and\n' + 'digests are retained; every copied fixture was checked and fsynced before use.\n') +(package/'.clang-format').write_text('# Preserve immutable measured source snapshots.\nDisableFormat: true\nSortIncludes: Never\n') +(package/'.gitattributes').write_text('*.log -whitespace\n*.txt -whitespace\n') +files={str(p.relative_to(package)):{'sha256':hashlib.sha256(p.read_bytes()).hexdigest(),'bytes':p.stat().st_size} + for p in sorted(package.rglob('*')) if p.is_file()} +(package/'inventory.json').write_text(json.dumps(files,indent=2)+'\n') +print(json.dumps({'path':str(package),'files':len(files),'bytes':sum(r['bytes'] for r in files.values())})) diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/profiles.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/profiles.json new file mode 100644 index 000000000..92e84d341 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/profiles.json @@ -0,0 +1,17 @@ +{ + "main": { + "tree": "/home/rob/projects/splinterdb", + "build_root": "build-main-0fa3bd6c", + "head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2" + }, + "filtered-ranges": { + "tree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/filtered-ranges", + "build_root": "build-filter-predicate", + "head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5" + }, + "disk-format": { + "tree": "/home/rob/projects/splinterdb/.cache/projects-20261011/worktrees/disk-format", + "build_root": "build-pr-0fa3bd6c", + "head": "3e302c04c60babb8afecb58d27f808848624f83c" + } +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/00-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/00-disk-format.log new file mode 100644 index 000000000..9556287aa --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/00-disk-format.log @@ -0,0 +1,12 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=15.0 warmup=1.0 rep=0 sabotage=0 +open took 0.011 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 6507049 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.13s ops/cpu-s=1486646 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 42066867 ops (+4206375 ops/s) + found=57895937 notfound=0 +close took 0.122 s +== point:uniform@8 T=8 ops=57895937 secs= 15.00 ops/s= 3859664 lat us p50=1.90 p99=4.06 p99.9=9.09 max=1075.4 mean=2.02 + cpu=119.97s ops/cpu-s=482599 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=15.00s drain(close)=0.12s total=15.12s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/01-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/01-main.log new file mode 100644 index 000000000..8bafb857f --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/01-main.log @@ -0,0 +1,12 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=15.0 warmup=1.0 rep=1 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 5901494 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.12s ops/cpu-s=1723247 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=1 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 43493821 ops (+4349071 ops/s) + found=60394092 notfound=0 +close took 0.088 s +== point:uniform@8 T=8 ops=60394092 secs= 15.00 ops/s= 4026197 lat us p50=1.84 p99=3.68 p99.9=8.32 max=988.8 mean=1.93 + cpu=119.98s ops/cpu-s=503373 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=15.00s drain(close)=0.09s total=15.09s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/02-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/02-main.log new file mode 100644 index 000000000..4a48b07f6 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/02-main.log @@ -0,0 +1,12 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=15.0 warmup=1.0 rep=2 sabotage=0 +open took 0.011 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.04 ops/s= 5566068 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.15s ops/cpu-s=1291414 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=6 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 45027977 ops (+4502461 ops/s) + found=61728685 notfound=0 +close took 0.076 s +== point:uniform@8 T=8 ops=61728685 secs= 15.00 ops/s= 4115151 lat us p50=1.84 p99=3.30 p99.9=7.62 max=464.9 mean=1.89 + cpu=119.97s ops/cpu-s=514521 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=15.00s drain(close)=0.08s total=15.08s drain_cpu=0.02s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/03-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/03-disk-format.log new file mode 100644 index 000000000..a5b4a0c88 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/03-disk-format.log @@ -0,0 +1,12 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=15.0 warmup=1.0 rep=3 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 6670963 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.12s ops/cpu-s=1690646 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 39481677 ops (+3947531 ops/s) + found=54767940 notfound=0 +close took 0.090 s +== point:uniform@8 T=8 ops=54767940 secs= 15.00 ops/s= 3651129 lat us p50=2.00 p99=4.42 p99.9=9.86 max=510.1 mean=2.13 + cpu=119.97s ops/cpu-s=456507 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + insert/run=15.00s drain(close)=0.09s total=15.09s drain_cpu=0.01s drain_rd=0 drain_wr=307200 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/manifest.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/manifest.json new file mode 100644 index 000000000..125888233 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/manifest.json @@ -0,0 +1,319 @@ +{ + "mode": "repeat-resident-disk-point8", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "runner_sha256": "ad1a38ada90f6f275d482205f75eb359fda6bea5d21a58547a4724472db64da1", + "runs": [ + { + "label": "00-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/results.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8", + "warm", + "point:uniform@8" + ], + "environment": {}, + "started": 1791923236.4197931, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791923252.6456418, + "log_sha256": "686d48f5ff9785b98e8a208cf385687f83560a8e040cfb2432c918f985a83102", + "process_cpu_s": 128.13298099999997, + "process_read_bytes": 28295168, + "process_write_bytes": 319488, + "errors": 0, + "violations": 0, + "verification_kind": "full warm scan and point key/value validation; successful close", + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "01-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/results.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "1", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8", + "warm", + "point:uniform@8" + ], + "environment": {}, + "started": 1791923252.9325798, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923269.0923226, + "log_sha256": "8a5b0e650f191e1f5eb78daefde09b9367092243204f9bbfd4a9e43322f78b08", + "process_cpu_s": 128.080198, + "process_read_bytes": 28295168, + "process_write_bytes": 315392, + "errors": 0, + "violations": 0, + "verification_kind": "full warm scan and point key/value validation; successful close", + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "02-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/results.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "2", + "--rep", + "2", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8", + "warm", + "point:uniform@8" + ], + "environment": {}, + "started": 1791923269.3683746, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791923285.5448258, + "log_sha256": "e7b311aa54d9a7c06c80061901258d86e0a64c1a0dee4e53bd6ddd415d803240", + "process_cpu_s": 128.15200000000004, + "process_read_bytes": 28295168, + "process_write_bytes": 319488, + "errors": 0, + "violations": 0, + "verification_kind": "full warm scan and point key/value validation; successful close", + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "03-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "15", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8/results.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "3", + "--rep", + "3", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/repeat-resident-disk-point8", + "warm", + "point:uniform@8" + ], + "environment": {}, + "started": 1791923285.7939076, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791923301.970146, + "log_sha256": "16d97b1e08eb089ea70f86720649920d278250e7155c706ce9478f886afa8af8", + "process_cpu_s": 128.110204, + "process_read_bytes": 28295168, + "process_write_bytes": 315392, + "errors": 0, + "violations": 0, + "verification_kind": "full warm scan and point key/value validation; successful close", + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + } + ], + "complete": true, + "order": [ + "disk-format", + "main", + "main", + "disk-format" + ], + "phases": [ + "warm", + "point:uniform@8" + ], + "seconds_per_phase": 15, + "repeat_runner_sha256": "1baecd91c684a9296201ec9786c901bd00db22cad34d3dc7e8c43d73f26daf80", + "rationale": "Reverse the broad-screen profile placement and lengthen the one point8 phase; retain broad samples separately." +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/results.csv b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/results.csv new file mode 100644 index 000000000..e4eb2bd1c --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat-resident-disk-point8/results.csv @@ -0,0 +1,9 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +disk-format,resident,0,0,warm,8,0,200000,200000,0.031,6507049.2,nan,nan,nan,0.000,nan,0.135,1486646.2,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,6507049.2,0.031,0.000,0.031,0.000,0,0,nan,0,0,0.011,0,0,0,0,0 +disk-format,resident,0,0,point:uniform@8,8,0,200000,57895937,15.000,3859663.8,1.904,4.064,9.088,1075.427,2.017,119.967,482599.1,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.122,15.123,0.014,0,307200,nan,162664448,28655616,0.011,0,0,0,0,0 +main,resident,1,0,warm,8,1,200000,200000,0.034,5901493.7,nan,nan,nan,0.000,nan,0.116,1723246.6,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,5901493.7,0.034,0.000,0.034,0.000,0,0,nan,0,0,0.008,0,0,0,0,1 +main,resident,1,0,point:uniform@8,8,1,200000,60394092,15.000,4026197.4,1.840,3.680,8.320,988.799,1.933,119.979,503373.0,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.088,15.088,0.014,0,307200,nan,162664448,28655616,0.008,0,0,0,0,0 +main,resident,2,0,warm,8,2,200000,200000,0.036,5566068.2,nan,nan,nan,0.000,nan,0.155,1291414.0,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,5566068.2,0.036,0.000,0.036,0.000,0,0,nan,0,0,0.011,0,0,0,0,6 +main,resident,2,0,point:uniform@8,8,2,200000,61728685,15.000,4115151.1,1.840,3.296,7.616,464.919,1.892,119.973,514520.6,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.076,15.076,0.018,0,307200,nan,162664448,28655616,0.011,0,0,0,0,0 +disk-format,resident,3,0,warm,8,3,200000,200000,0.030,6670963.2,nan,nan,nan,0.000,nan,0.118,1690645.7,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,6670963.2,0.030,0.000,0.030,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +disk-format,resident,3,0,point:uniform@8,8,3,200000,54767940,15.000,3651128.5,2.000,4.416,9.856,510.115,2.133,119.972,456506.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,15.000,0.090,15.090,0.015,0,307200,nan,162664448,28655616,0.008,0,0,0,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat_point.py b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat_point.py new file mode 100644 index 000000000..4a579ab29 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/repeat_point.py @@ -0,0 +1,26 @@ +"""A separate longer BAAB repeat of the disk-format resident point8 concern.""" +import json, os, subprocess +from pathlib import Path +import run as common +folder,m=common.mode_start('repeat-resident-disk-point8') +order=['disk-format','main','main','disk-format'];phases=['warm','point:uniform@8'] +m.update(order=order,phases=phases,seconds_per_phase=15,repeat_runner_sha256=common.sha(__file__), + rationale='Reverse the broad-screen profile placement and lengthen the one point8 phase; retain broad samples separately.') +fixtures=json.loads((common.OUT/'fixtures/fixtures.json').read_text()) +for i,profile in enumerate(order): + fixture=fixtures['resident-'+profile];src=Path(fixture['fixture']);meta=src.stat() + assert common.sha(src)==fixture['fixture_sha256'] + db=folder/'scratch.db';assert not db.exists() + subprocess.run(['cp','--reflink=auto','--sparse=always',str(src),str(db)],check=True) + with db.open('rb') as image:os.fsync(image.fileno()) + csvfile=folder/'results.csv' + args=common.args(db,csvfile,profile,'resident',i) + args[args.index('--secs')+1]=15 + rec=common.run(folder,m,f'{i:02d}-{profile}',profile,'kvbench',args+phases) + rows=common.verify_csv(csvfile,i,phases,profile) + assert all(int(r['rd_bytes'])==0 for r in rows if r['phase']!='warm') + rec.update(verified=True,errors=0,violations=0,verification_kind='full warm scan and point key/value validation; successful close', + fixture=fixture,fixture_copy_fsynced=True,allocated_bytes=db.stat().st_blocks*512) + assert src.stat().st_mtime_ns==meta.st_mtime_ns and src.stat().st_size==meta.st_size + common.save(folder/'manifest.json',m);db.unlink() +m['complete']=True;common.save(folder/'manifest.json',m) diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report-generation.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report-generation.log new file mode 100644 index 000000000..57079fc24 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report-generation.log @@ -0,0 +1,78 @@ +Filtered predicates and disk-format screening against main 0fa3bd6c + +OPEN PERFORMANCE CONCERN: disk-format resident point8 throughput fell 4.39% in the broad screen and 7.75% in the separate longer repeat, both with non-overlapping sample ranges and zero physical reads. +The cause remains unassigned. Disk-format is ready for code review; these results do not support performance-neutrality or merge-ready claims. Broad and repeat samples are retained separately. + +All databases were on verified real disk using O_DIRECT. Timing was serialized with the campaign benchmark lock. +Ordinary controls: main / filtered / disk-format / disk-format / filtered / main, 8-second resident and 6-second disk phases. +Fresh deterministic fixtures were created serially, closed and fsynced; disk-format uses equivalent data in its own format. +A full warm scan validates every key/value before timed phases. Timed operations also validate their results. +Feature runs: application / predicate-only / pruned / pruned / predicate-only / application for each collision setting. +Each feature run ingests 524288 keys grouped by tenant, then queries before close so selective immutable branches remain. +Each limit has 500 untimed warmup queries and 10000 measured queries; warmup counters are excluded. +Feature checksums are independently recomputed in Python; every matched tuple is checked for row, tenant, 64-byte value length and its 16-byte identity prefix. + +These compare each whole feature branch with new main, not the isolated cost of its main merge. +These are screening samples, not confidence intervals or proof of performance neutrality. Two samples per profile. +Sample range overlap is descriptive. Device and CPU drift can obscure small differences; all controls and raw samples are retained. +A separate 15-second resident point8 repeat uses disk-format/main/main/disk-format to investigate the first disk sample; it does not replace the broad-screen results. +Feature CPU/I/O is per measured limit; process resource usage in manifests also includes ingestion, warmup and close. +Predicate callbacks increment a call counter in both variants; the candidate also checks returned tenant identity externally. +Collision mode maps each tenant pair to the same full 32-bit hash under every seed; exact predicates still distinguish tenants. + +RESIDENT +phase | profile | throughput change | main ops/s range | profile ops/s range | overlaps +point:uniform@1 | filtered-ranges | -2.53% | 688766..692716 | 654933..691544 | True +point:uniform@1 | disk-format | -1.13% | 688766..692716 | 667909..697962 | True +point:uniform@8 | filtered-ranges | +0.69% | 3747424..3762979 | 3714776..3847725 | True +point:uniform@8 | disk-format | -4.39% | 3747424..3762979 | 3435059..3745575 | False +range:uniform:1@8 | filtered-ranges | -0.18% | 363034..375482 | 364020..373202 | True +range:uniform:1@8 | disk-format | +0.78% | 363034..375482 | 367890..376370 | True +range:uniform:10000@8 | filtered-ranges | -0.74% | 2895..2931 | 2834..2948 | True +range:uniform:10000@8 | disk-format | +0.13% | 2895..2931 | 2912..2921 | True + +DISK +phase | profile | throughput change | main ops/s range | profile ops/s range | overlaps +point:uniform@8 | filtered-ranges | -0.53% | 140323..144025 | 140070..142765 | True +point:uniform@8 | disk-format | +4.87% | 140323..144025 | 141560..156630 | True +range:uniform:1@8 | filtered-ranges | -0.30% | 60886..61632 | 60071..62085 | True +range:uniform:1@8 | disk-format | +1.41% | 60886..61632 | 61649..62602 | False +range:uniform:10000@8 | filtered-ranges | -1.50% | 741..744 | 719..744 | True +range:uniform:10000@8 | disk-format | -0.79% | 741..744 | 729..745 | True + +REPEAT-RESIDENT-DISK-POINT8 +phase | profile | throughput change | main ops/s range | profile ops/s range | overlaps +point:uniform@8 | disk-format | -7.75% | 4026197..4115151 | 3651128..3859664 | False + +FEATURE LATENCY (negative change is faster) +collisions | limit | mode | latency change | main ns/query range | mode ns/query range | overlaps +0 | 1 | predicate | -3.69% | 17470..17594 | 16617..17153 | False +0 | 1 | pruned | -16.30% | 17470..17594 | 14468..14880 | False +0 | 2 | predicate | -8.42% | 12844..13185 | 11888..11948 | False +0 | 2 | pruned | -22.94% | 12844..13185 | 9744..10315 | False +0 | 10 | predicate | -13.19% | 30790..31665 | 26638..27579 | False +0 | 10 | pruned | -30.83% | 30790..31665 | 21176..22023 | False +0 | 100 | predicate | -14.22% | 218968..224873 | 188616..192129 | False +0 | 100 | pruned | -32.09% | 218968..224873 | 149478..151945 | False +1 | 1 | predicate | -7.54% | 16849..18701 | 15955..16916 | True +1 | 1 | pruned | -14.19% | 16849..18701 | 15070..15435 | False +1 | 2 | predicate | -16.22% | 12844..15494 | 11844..11898 | False +1 | 2 | pruned | -28.31% | 12844..15494 | 10052..10265 | False +1 | 10 | predicate | -14.76% | 30458..32762 | 26802..27086 | False +1 | 10 | pruned | -30.21% | 30458..32762 | 21834..22288 | False +1 | 100 | predicate | -13.03% | 217525..218090 | 188514..190339 | False +1 | 100 | pruned | -30.68% | 217525..218090 | 147736..154228 | False + +MAIN CONTROL DRIFT +resident point:uniform@1: last versus first -0.57% +resident point:uniform@8: last versus first +0.42% +resident range:uniform:1@8: last versus first +3.43% +resident range:uniform:10000@8: last versus first +1.26% +disk point:uniform@8: last versus first -2.57% +disk range:uniform:1@8: last versus first -1.21% +disk range:uniform:10000@8: last versus first +0.31% +repeat-resident-disk-point8 point:uniform@8: last versus first +2.21% + +Resident measured physical read bytes: 0 +All runs verified: {"fixtures": 4, "resident": 6, "disk": 6, "feature": 12, "repeat-resident-disk-point8": 4} +Detailed CPU/op, I/O/op, p99, sample rates and predicate/visited counts are in summary.json. diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.py b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.py new file mode 100644 index 000000000..7dad4a358 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.py @@ -0,0 +1,113 @@ +"""Validate complete manifests before aggregating; retain raw sample ranges.""" +import csv, json, statistics +from pathlib import Path +HERE=Path(__file__).resolve().parent +OUT=Path('/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013') + +def save(p,x):p.write_text(json.dumps(x,indent=2)+'\n') +def valid(mode): + m=json.loads((OUT/mode/'manifest.json').read_text()) + assert m['complete'] and m['runs'] and all(r['verified'] and r['returncode']==0 for r in m['runs']) + return m + +def describe(values):return {'mean':statistics.fmean(values),'min':min(values),'max':max(values),'samples':len(values),'values':values} +summary={};controls=[] +for regime in ['resident','disk','repeat-resident-disk-point8']: + m=valid(regime) + expected_order=['disk-format','main','main','disk-format'] if regime.startswith('repeat-') else ['main','filtered-ranges','disk-format','disk-format','filtered-ranges','main'] + assert [r['profile'] for r in m['runs']]==expected_order + with (OUT/regime/'results.csv').open() as f:rows=list(csv.DictReader(f)) + assert len(rows)==len(m['runs'])*len(m['phases']) + for run in m['runs']: + cmd=run['command']; index=cmd[cmd.index('--otag')+1] + rr=[r for r in rows if r['order']==index] + assert [r['phase'] for r in rr]==m['phases'] and all(r['version']==run['profile'] for r in rr) + assert all(int(r['ops'])>0 and int(r['errors'])==int(r['violations'])==0 for r in rr) + assert all(int(r['ops'])==int(r['n']) for r in rr if r['phase']=='warm') + results=[] + for phase in m['phases'][1:]: + base=[r for r in rows if r['phase']==phase and r['version']=='main'] + baseline=describe([float(r['ops_s']) for r in base]) + controls.append({'regime':regime,'phase':phase,'samples':baseline,'last_vs_first_pct':100*(float(base[-1]['ops_s'])/float(base[0]['ops_s'])-1)}) + for profile in dict.fromkeys(r['profile'] for r in m['runs']): + rr=[r for r in rows if r['phase']==phase and r['version']==profile] + throughput=describe([float(r['ops_s']) for r in rr]) + results.append(dict(profile=profile,phase=phase,throughput=throughput,baseline=baseline, + throughput_delta_pct=100*(throughput['mean']/baseline['mean']-1), + ranges_overlap=max(throughput['min'],baseline['min'])<=min(throughput['max'],baseline['max']), + cpu_us_per_op=describe([float(r['cpu_s'])*1e6/int(r['ops']) for r in rr]), + read_B_per_op=describe([float(r['rd_B_op']) for r in rr]), + write_B_per_op=describe([float(r['wr_B_op']) for r in rr]), + p99_us=describe([float(r['p99_us']) for r in rr]), + physical_read_bytes=sum(int(r['rd_bytes']) for r in rr),errors=0,violations=0)) + summary[regime]=results +m=valid('feature');results=[] +assert len(m['runs'])==12 +for run in m['runs']: + for row in run['rows']: + count=int(row['queries']); limit=int(row['limit']) + if row['mode']=='application': + expected_visited=count*((limit-1)*32+1)+sum(q%32 for q in range(count)) + assert int(row['visited'])==int(row['predicate_calls'])==expected_visited + else: + assert int(row['visited'])==count*limit +for collision in [0,1]: + for limit in [1,2,10,100]: + groups={mode:[r for run in m['runs'] for r in run['rows'] if r['mode']==mode and int(r['collisions'])==collision and int(r['limit'])==limit] for mode in ['application','predicate','pruned']} + assert all(len(rows)==2 for rows in groups.values()) + checks={r['checksum'] for rows in groups.values() for r in rows};assert len(checks)==1 + base=describe([float(r['ns_per_query']) for r in groups['application']]) + for mode,rows in groups.items(): + costs=describe([float(r['ns_per_query']) for r in rows]) + results.append(dict(collisions=collision,limit=limit,mode=mode,latency_ns=costs,baseline=base, + latency_delta_pct=100*(costs['mean']/base['mean']-1), + ranges_overlap=max(costs['min'],base['min'])<=min(costs['max'],base['max']), + cpu_us_per_query=describe([float(r['cpu_s'])*1e6/int(r['queries']) for r in rows]), + read_B_per_query=describe([int(r['read_bytes'])/int(r['queries']) for r in rows]), + write_B_per_query=describe([int(r['write_bytes'])/int(r['queries']) for r in rows]), + visited_per_query=describe([int(r['visited'])/int(r['queries']) for r in rows]), + predicate_calls_per_query=describe([int(r['predicate_calls'])/int(r['queries']) for r in rows]), + checksum=next(iter(checks)),verified=True)) +summary['feature']=results +save(HERE/'summary.json',summary);save(HERE/'control-drift.json',controls) +counts={mode:len(valid(mode)['runs']) for mode in ['fixtures','resident','disk','feature','repeat-resident-disk-point8']} +assert counts=={'fixtures':4,'resident':6,'disk':6,'feature':12,'repeat-resident-disk-point8':4} +save(HERE/'status.json',{'complete':True,'runs':counts,'errors':sum(r.get('errors',0) for mode in counts for r in valid(mode)['runs']), + 'violations':sum(r.get('violations',0) for mode in counts for r in valid(mode)['runs']), + 'verification':'All processes succeeded; all manifests verified; exact warm/load counts and output checks validated.'}) +lines=['Filtered predicates and disk-format screening against main 0fa3bd6c','', + 'OPEN PERFORMANCE CONCERN: disk-format resident point8 throughput fell 4.39% in the broad screen and 7.75% in the separate longer repeat, both with non-overlapping sample ranges and zero physical reads.', + 'The cause remains unassigned. Disk-format is ready for code review; these results do not support performance-neutrality or merge-ready claims. Broad and repeat samples are retained separately.', '', + 'All databases were on verified real disk using O_DIRECT. Timing was serialized with the campaign benchmark lock.', + 'Ordinary controls: main / filtered / disk-format / disk-format / filtered / main, 8-second resident and 6-second disk phases.', + 'Fresh deterministic fixtures were created serially, closed and fsynced; disk-format uses equivalent data in its own format.', + 'A full warm scan validates every key/value before timed phases. Timed operations also validate their results.', + 'Feature runs: application / predicate-only / pruned / pruned / predicate-only / application for each collision setting.', + 'Each feature run ingests 524288 keys grouped by tenant, then queries before close so selective immutable branches remain.', + 'Each limit has 500 untimed warmup queries and 10000 measured queries; warmup counters are excluded.', + 'Feature checksums are independently recomputed in Python; every matched tuple is checked for row, tenant, 64-byte value length and its 16-byte identity prefix.', + '', 'These compare each whole feature branch with new main, not the isolated cost of its main merge.', + 'These are screening samples, not confidence intervals or proof of performance neutrality. Two samples per profile.', + 'Sample range overlap is descriptive. Device and CPU drift can obscure small differences; all controls and raw samples are retained.', + 'A separate 15-second resident point8 repeat uses disk-format/main/main/disk-format to investigate the first disk sample; it does not replace the broad-screen results.', + 'Feature CPU/I/O is per measured limit; process resource usage in manifests also includes ingestion, warmup and close.', + 'Predicate callbacks increment a call counter in both variants; the candidate also checks returned tenant identity externally.', + 'Collision mode maps each tenant pair to the same full 32-bit hash under every seed; exact predicates still distinguish tenants.', ''] +for regime in ['resident','disk','repeat-resident-disk-point8']: + lines += [regime.upper(), 'phase | profile | throughput change | main ops/s range | profile ops/s range | overlaps'] + for r in summary[regime]: + if r['profile']=='main':continue + a=r['baseline'];b=r['throughput'] + lines.append(f"{r['phase']} | {r['profile']} | {r['throughput_delta_pct']:+.2f}% | {a['min']:.0f}..{a['max']:.0f} | {b['min']:.0f}..{b['max']:.0f} | {r['ranges_overlap']}") + lines.append('') +lines+=['FEATURE LATENCY (negative change is faster)','collisions | limit | mode | latency change | main ns/query range | mode ns/query range | overlaps'] +for r in results: + if r['mode']=='application':continue + a=r['baseline'];b=r['latency_ns'] + lines.append(f"{r['collisions']} | {r['limit']} | {r['mode']} | {r['latency_delta_pct']:+.2f}% | {a['min']:.0f}..{a['max']:.0f} | {b['min']:.0f}..{b['max']:.0f} | {r['ranges_overlap']}") +lines+=['','MAIN CONTROL DRIFT'] +for r in controls:lines.append(f"{r['regime']} {r['phase']}: last versus first {r['last_vs_first_pct']:+.2f}%") +lines+=['','Resident measured physical read bytes: '+str(sum(r['physical_read_bytes'] for r in summary['resident'])), + 'All runs verified: '+json.dumps(counts), + 'Detailed CPU/op, I/O/op, p99, sample rates and predicate/visited counts are in summary.json.'] +(HERE/'report.txt').write_text('\n'.join(lines)+'\n');print('\n'.join(lines)) diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.txt b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.txt new file mode 100644 index 000000000..57079fc24 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.txt @@ -0,0 +1,78 @@ +Filtered predicates and disk-format screening against main 0fa3bd6c + +OPEN PERFORMANCE CONCERN: disk-format resident point8 throughput fell 4.39% in the broad screen and 7.75% in the separate longer repeat, both with non-overlapping sample ranges and zero physical reads. +The cause remains unassigned. Disk-format is ready for code review; these results do not support performance-neutrality or merge-ready claims. Broad and repeat samples are retained separately. + +All databases were on verified real disk using O_DIRECT. Timing was serialized with the campaign benchmark lock. +Ordinary controls: main / filtered / disk-format / disk-format / filtered / main, 8-second resident and 6-second disk phases. +Fresh deterministic fixtures were created serially, closed and fsynced; disk-format uses equivalent data in its own format. +A full warm scan validates every key/value before timed phases. Timed operations also validate their results. +Feature runs: application / predicate-only / pruned / pruned / predicate-only / application for each collision setting. +Each feature run ingests 524288 keys grouped by tenant, then queries before close so selective immutable branches remain. +Each limit has 500 untimed warmup queries and 10000 measured queries; warmup counters are excluded. +Feature checksums are independently recomputed in Python; every matched tuple is checked for row, tenant, 64-byte value length and its 16-byte identity prefix. + +These compare each whole feature branch with new main, not the isolated cost of its main merge. +These are screening samples, not confidence intervals or proof of performance neutrality. Two samples per profile. +Sample range overlap is descriptive. Device and CPU drift can obscure small differences; all controls and raw samples are retained. +A separate 15-second resident point8 repeat uses disk-format/main/main/disk-format to investigate the first disk sample; it does not replace the broad-screen results. +Feature CPU/I/O is per measured limit; process resource usage in manifests also includes ingestion, warmup and close. +Predicate callbacks increment a call counter in both variants; the candidate also checks returned tenant identity externally. +Collision mode maps each tenant pair to the same full 32-bit hash under every seed; exact predicates still distinguish tenants. + +RESIDENT +phase | profile | throughput change | main ops/s range | profile ops/s range | overlaps +point:uniform@1 | filtered-ranges | -2.53% | 688766..692716 | 654933..691544 | True +point:uniform@1 | disk-format | -1.13% | 688766..692716 | 667909..697962 | True +point:uniform@8 | filtered-ranges | +0.69% | 3747424..3762979 | 3714776..3847725 | True +point:uniform@8 | disk-format | -4.39% | 3747424..3762979 | 3435059..3745575 | False +range:uniform:1@8 | filtered-ranges | -0.18% | 363034..375482 | 364020..373202 | True +range:uniform:1@8 | disk-format | +0.78% | 363034..375482 | 367890..376370 | True +range:uniform:10000@8 | filtered-ranges | -0.74% | 2895..2931 | 2834..2948 | True +range:uniform:10000@8 | disk-format | +0.13% | 2895..2931 | 2912..2921 | True + +DISK +phase | profile | throughput change | main ops/s range | profile ops/s range | overlaps +point:uniform@8 | filtered-ranges | -0.53% | 140323..144025 | 140070..142765 | True +point:uniform@8 | disk-format | +4.87% | 140323..144025 | 141560..156630 | True +range:uniform:1@8 | filtered-ranges | -0.30% | 60886..61632 | 60071..62085 | True +range:uniform:1@8 | disk-format | +1.41% | 60886..61632 | 61649..62602 | False +range:uniform:10000@8 | filtered-ranges | -1.50% | 741..744 | 719..744 | True +range:uniform:10000@8 | disk-format | -0.79% | 741..744 | 729..745 | True + +REPEAT-RESIDENT-DISK-POINT8 +phase | profile | throughput change | main ops/s range | profile ops/s range | overlaps +point:uniform@8 | disk-format | -7.75% | 4026197..4115151 | 3651128..3859664 | False + +FEATURE LATENCY (negative change is faster) +collisions | limit | mode | latency change | main ns/query range | mode ns/query range | overlaps +0 | 1 | predicate | -3.69% | 17470..17594 | 16617..17153 | False +0 | 1 | pruned | -16.30% | 17470..17594 | 14468..14880 | False +0 | 2 | predicate | -8.42% | 12844..13185 | 11888..11948 | False +0 | 2 | pruned | -22.94% | 12844..13185 | 9744..10315 | False +0 | 10 | predicate | -13.19% | 30790..31665 | 26638..27579 | False +0 | 10 | pruned | -30.83% | 30790..31665 | 21176..22023 | False +0 | 100 | predicate | -14.22% | 218968..224873 | 188616..192129 | False +0 | 100 | pruned | -32.09% | 218968..224873 | 149478..151945 | False +1 | 1 | predicate | -7.54% | 16849..18701 | 15955..16916 | True +1 | 1 | pruned | -14.19% | 16849..18701 | 15070..15435 | False +1 | 2 | predicate | -16.22% | 12844..15494 | 11844..11898 | False +1 | 2 | pruned | -28.31% | 12844..15494 | 10052..10265 | False +1 | 10 | predicate | -14.76% | 30458..32762 | 26802..27086 | False +1 | 10 | pruned | -30.21% | 30458..32762 | 21834..22288 | False +1 | 100 | predicate | -13.03% | 217525..218090 | 188514..190339 | False +1 | 100 | pruned | -30.68% | 217525..218090 | 147736..154228 | False + +MAIN CONTROL DRIFT +resident point:uniform@1: last versus first -0.57% +resident point:uniform@8: last versus first +0.42% +resident range:uniform:1@8: last versus first +3.43% +resident range:uniform:10000@8: last versus first +1.26% +disk point:uniform@8: last versus first -2.57% +disk range:uniform:1@8: last versus first -1.21% +disk range:uniform:10000@8: last versus first +0.31% +repeat-resident-disk-point8 point:uniform@8: last versus first +2.21% + +Resident measured physical read bytes: 0 +All runs verified: {"fixtures": 4, "resident": 6, "disk": 6, "feature": 12, "repeat-resident-disk-point8": 4} +Detailed CPU/op, I/O/op, p99, sample rates and predicate/visited counts are in summary.json. diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident-summary.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident-summary.json new file mode 100644 index 000000000..6d3d97abc --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident-summary.json @@ -0,0 +1,114 @@ +[ + { + "phase": "point:uniform@1", + "profile": "filtered-ranges", + "main": [ + 692716.3, + 688766.2 + ], + "candidate": [ + 654932.6, + 691544.1 + ], + "delta_pct": -2.5339300353062755, + "overlap": true + }, + { + "phase": "point:uniform@1", + "profile": "disk-format", + "main": [ + 692716.3, + 688766.2 + ], + "candidate": [ + 667909.4, + 697961.6 + ], + "delta_pct": -1.1300541266356934, + "overlap": true + }, + { + "phase": "point:uniform@8", + "profile": "filtered-ranges", + "main": [ + 3747424.5, + 3762978.7 + ], + "candidate": [ + 3714776.4, + 3847725.4 + ], + "delta_pct": 0.6936857930610163, + "overlap": true + }, + { + "phase": "point:uniform@8", + "profile": "disk-format", + "main": [ + 3747424.5, + 3762978.7 + ], + "candidate": [ + 3435058.8, + 3745574.9 + ], + "delta_pct": -4.39083616709155, + "overlap": false + }, + { + "phase": "range:uniform:1@8", + "profile": "filtered-ranges", + "main": [ + 363033.5, + 375481.6 + ], + "candidate": [ + 364020.2, + 373202.3 + ], + "delta_pct": -0.17502688841433134, + "overlap": true + }, + { + "phase": "range:uniform:1@8", + "profile": "disk-format", + "main": [ + 363033.5, + 375481.6 + ], + "candidate": [ + 367889.5, + 376369.6 + ], + "delta_pct": 0.7777769202010898, + "overlap": true + }, + { + "phase": "range:uniform:10000@8", + "profile": "filtered-ranges", + "main": [ + 2894.8, + 2931.2 + ], + "candidate": [ + 2834.4, + 2948.5 + ], + "delta_pct": -0.7397871610024143, + "overlap": true + }, + { + "phase": "range:uniform:10000@8", + "profile": "disk-format", + "main": [ + 2894.8, + 2931.2 + ], + "candidate": [ + 2912.4, + 2921.4 + ], + "delta_pct": 0.13388259526261326, + "overlap": true + } +] diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/00-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/00-main.log new file mode 100644 index 000000000..5ffb1ca04 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/00-main.log @@ -0,0 +1,26 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=0 sabotage=0 +open took 0.009 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 6357250 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.13s ops/cpu-s=1561805 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=1 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=5541952 notfound=0 +== point:uniform@1 T=1 ops=5541952 secs= 8.00 ops/s= 692716 lat us p50=1.36 p99=1.87 p99.9=4.80 max=68.0 mean=1.40 + cpu=8.00s ops/cpu-s=692709 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 7778162 ops (+777706 ops/s) + found=29981074 notfound=0 +== point:uniform@8 T=8 ops=29981074 secs= 8.00 ops/s= 3747424 lat us p50=1.97 p99=4.29 p99.9=10.62 max=4039.1 mean=2.08 + cpu=63.91s ops/cpu-s=469113 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1062485 ops (+1844528540946548224 ops/s) +== range:uniform:1@8 T=8 ops=2904386 secs= 8.00 ops/s= 363034 lat us p50=21.25 p99=40.45 p99.9=60.93 max=901.0 mean=21.97 + cpu=63.97s ops/cpu-s=45405 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=20.22 p99=38.40 mean=21.01 per-tuple=nan ns deinit mean=0.90 us tuples/s=363034 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11529 ops (+1844542589532594176 ops/s) +close took 0.063 s +== range:uniform:10000@8 T=8 ops=23159 secs= 8.00 ops/s= 2895 lat us p50=2785.28 p99=4521.98 p99.9=4915.20 max=5893.1 mean=2763.22 + cpu=63.98s ops/cpu-s=362 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=22.78 p99=53.76 mean=25.14 per-tuple=280.6 ns deinit mean=1.22 us tuples/s=28229845 + insert/run=8.00s drain(close)=0.06s total=8.06s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/01-filtered-ranges.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/01-filtered-ranges.log new file mode 100644 index 000000000..4b22c21d6 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/01-filtered-ranges.log @@ -0,0 +1,26 @@ +kvbench version=filtered-ranges splinter=splinterdb_build_version cf7dc4f7 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=1 sabotage=0 +open took 0.009 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 5780390 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.13s ops/cpu-s=1485178 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=5239678 notfound=0 +== point:uniform@1 T=1 ops=5239678 secs= 8.00 ops/s= 654933 lat us p50=1.42 p99=2.59 p99.9=8.32 max=535.4 mean=1.48 + cpu=8.00s ops/cpu-s=655095 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=90 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 8265450 ops (+826432 ops/s) + found=29719807 notfound=0 +== point:uniform@8 T=8 ops=29719807 secs= 8.00 ops/s= 3714776 lat us p50=1.94 p99=4.29 p99.9=15.49 max=3081.4 mean=2.10 + cpu=63.93s ops/cpu-s=464890 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=1 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 989349 ops (+1844535670250882304 ops/s) +== range:uniform:1@8 T=8 ops=2912265 secs= 8.00 ops/s= 364020 lat us p50=21.25 p99=39.42 p99.9=53.76 max=1466.2 mean=21.91 + cpu=63.98s ops/cpu-s=45518 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=2 swapout=0 majflt=0 + startup us p50=20.22 p99=37.38 mean=20.93 per-tuple=nan ns deinit mean=0.92 us tuples/s=364020 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11563 ops (+1844524787820430848 ops/s) +close took 0.077 s +== range:uniform:10000@8 T=8 ops=22676 secs= 8.00 ops/s= 2834 lat us p50=2850.82 p99=4521.98 p99.9=5177.34 max=6821.0 mean=2822.21 + cpu=63.95s ops/cpu-s=355 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=7 swapout=0 majflt=0 + startup us p50=23.81 p99=52.74 mean=26.30 per-tuple=286.4 ns deinit mean=1.38 us tuples/s=27651276 + insert/run=8.00s drain(close)=0.08s total=8.08s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/02-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/02-disk-format.log new file mode 100644 index 000000000..fa23a4b28 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/02-disk-format.log @@ -0,0 +1,26 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=2 sabotage=0 +open took 0.008 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.03 ops/s= 6883319 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.11s ops/cpu-s=1773490 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=1 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=5343451 notfound=0 +== point:uniform@1 T=1 ops=5343451 secs= 8.00 ops/s= 667909 lat us p50=1.39 p99=2.46 p99.9=7.74 max=444.6 mean=1.45 + cpu=8.00s ops/cpu-s=668265 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=9 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 6708226 ops (+670752 ops/s) + found=27481449 notfound=0 +== point:uniform@8 T=8 ops=27481449 secs= 8.00 ops/s= 3435059 lat us p50=2.14 p99=4.29 p99.9=9.09 max=746.2 mean=2.27 + cpu=63.97s ops/cpu-s=429611 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=1 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1022628 ops (+1844530756790672128 ops/s) +== range:uniform:1@8 T=8 ops=2943204 secs= 8.00 ops/s= 367889 lat us p50=21.25 p99=39.42 p99.9=52.74 max=4017.7 mean=21.68 + cpu=63.97s ops/cpu-s=46006 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=20.22 p99=37.38 mean=20.70 per-tuple=nan ns deinit mean=0.92 us tuples/s=367889 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11275 ops (+1844538724393464576 ops/s) +close took 0.086 s +== range:uniform:10000@8 T=8 ops=23300 secs= 8.00 ops/s= 2912 lat us p50=2850.82 p99=4259.84 p99.9=4521.98 max=5677.2 mean=2746.43 + cpu=63.99s ops/cpu-s=364 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=22.78 p99=46.59 mean=24.29 per-tuple=278.8 ns deinit mean=1.16 us tuples/s=28422566 + insert/run=8.00s drain(close)=0.09s total=8.09s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/03-disk-format.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/03-disk-format.log new file mode 100644 index 000000000..d6f165cda --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/03-disk-format.log @@ -0,0 +1,26 @@ +kvbench version=disk-format splinter=splinterdb_build_version 2ba43edd-dirty db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=3 sabotage=0 +open took 0.014 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.04 ops/s= 4837517 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.17s ops/cpu-s=1170029 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=5583921 notfound=0 +== point:uniform@1 T=1 ops=5583921 secs= 8.00 ops/s= 697962 lat us p50=1.36 p99=1.87 p99.9=5.82 max=158.4 mean=1.39 + cpu=8.00s ops/cpu-s=697980 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 7542839 ops (+754189 ops/s) + found=29966090 notfound=0 +== point:uniform@8 T=8 ops=29966090 secs= 8.00 ops/s= 3745575 lat us p50=1.97 p99=4.00 p99.9=10.11 max=2445.1 mean=2.08 + cpu=63.94s ops/cpu-s=468662 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=3 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1093426 ops (+1844505337797640960 ops/s) +== range:uniform:1@8 T=8 ops=3011053 secs= 8.00 ops/s= 376370 lat us p50=20.74 p99=38.40 p99.9=52.74 max=674.2 mean=21.19 + cpu=63.97s ops/cpu-s=47071 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=19.71 p99=36.35 mean=20.23 per-tuple=nan ns deinit mean=0.90 us tuples/s=376370 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11637 ops (+1844500387612877312 ops/s) +close took 0.076 s +== range:uniform:10000@8 T=8 ops=23372 secs= 8.00 ops/s= 2921 lat us p50=2850.82 p99=4259.84 p99.9=4521.98 max=4894.0 mean=2738.21 + cpu=63.97s ops/cpu-s=365 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=22.78 p99=46.59 mean=24.25 per-tuple=278.0 ns deinit mean=1.17 us tuples/s=28498966 + insert/run=8.00s drain(close)=0.08s total=8.08s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/04-filtered-ranges.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/04-filtered-ranges.log new file mode 100644 index 000000000..e734c882b --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/04-filtered-ranges.log @@ -0,0 +1,26 @@ +kvbench version=filtered-ranges splinter=splinterdb_build_version cf7dc4f7 db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=4 sabotage=0 +open took 0.011 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.04 ops/s= 5211542 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.17s ops/cpu-s=1209497 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=2 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=5532543 notfound=0 +== point:uniform@1 T=1 ops=5532543 secs= 8.00 ops/s= 691544 lat us p50=1.36 p99=2.21 p99.9=6.08 max=149.4 mean=1.40 + cpu=8.00s ops/cpu-s=691851 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 7601362 ops (+760023 ops/s) + found=30782885 notfound=0 +== point:uniform@8 T=8 ops=30782885 secs= 8.00 ops/s= 3847725 lat us p50=1.90 p99=3.87 p99.9=9.34 max=2096.7 mean=2.02 + cpu=63.98s ops/cpu-s=481146 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1092437 ops (+1844529535438249472 ops/s) +== range:uniform:1@8 T=8 ops=2985717 secs= 8.00 ops/s= 373202 lat us p50=21.25 p99=38.40 p99.9=52.74 max=163.2 mean=21.37 + cpu=63.99s ops/cpu-s=46660 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=20.22 p99=37.38 mean=20.42 per-tuple=nan ns deinit mean=0.89 us tuples/s=373202 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11611 ops (+1844522080281629696 ops/s) +close took 0.071 s +== range:uniform:10000@8 T=8 ops=23589 secs= 8.00 ops/s= 2949 lat us p50=2785.28 p99=4259.84 p99.9=4521.98 max=4700.1 mean=2712.64 + cpu=63.98s ops/cpu-s=369 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=1 swapout=0 majflt=0 + startup us p50=22.27 p99=44.54 mean=24.15 per-tuple=275.5 ns deinit mean=1.24 us tuples/s=28760766 + insert/run=8.00s drain(close)=0.07s total=8.07s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/05-main.log b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/05-main.log new file mode 100644 index 000000000..b43dcd8b2 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/05-main.log @@ -0,0 +1,26 @@ +kvbench version=main splinter=splinterdb_build_version 0fa3bd6c db=/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db mode=open n=200000 cache_mib=128 disk_gib=8 log=0 threads=8 mt_bg=0 normal_bg=0 memtable_mib=4 secs=8.0 warmup=1.0 rep=5 sabotage=0 +open took 0.012 s +phase warm: T=8 dist=uniform (fixed count) +== warm T=8 ops=200000 secs= 0.04 ops/s= 5432022 lat us p50=nan p99=nan p99.9=nan max=0.0 mean=nan + cpu=0.15s ops/cpu-s=1351863 rd=138.4 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@1: T=1 dist=uniform (fixed duration) + found=5510338 notfound=0 +== point:uniform@1 T=1 ops=5510338 secs= 8.00 ops/s= 688766 lat us p50=1.39 p99=1.84 p99.9=5.31 max=222.7 mean=1.41 + cpu=8.00s ops/cpu-s=688518 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 +phase point:uniform@8: T=8 dist=uniform (fixed duration) + [point:uniform@8] progress 7458027 ops (+745694 ops/s) + found=30104822 notfound=0 +== point:uniform@8 T=8 ops=30104822 secs= 8.00 ops/s= 3762979 lat us p50=1.94 p99=3.94 p99.9=10.11 max=2100.5 mean=2.07 + cpu=63.97s ops/cpu-s=470628 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=1 swapout=0 majflt=0 +phase range:uniform:1@8: T=8 dist=uniform (fixed duration) + [range:uniform:1@8] progress 1069920 ops (+1844540206003088128 ops/s) +== range:uniform:1@8 T=8 ops=3003959 secs= 8.00 ops/s= 375482 lat us p50=20.74 p99=38.40 p99.9=51.71 max=159.8 mean=21.24 + cpu=63.99s ops/cpu-s=46946 rd=0.0 B/op wr=0.0 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=19.71 p99=36.35 mean=20.29 per-tuple=nan ns deinit mean=0.89 us tuples/s=375482 +phase range:uniform:10000@8: T=8 dist=uniform (fixed duration) + [range:uniform:10000@8] progress 11540 ops (+1844532814945015808 ops/s) +close took 0.087 s +== range:uniform:10000@8 T=8 ops=23451 secs= 8.00 ops/s= 2931 lat us p50=2850.82 p99=4259.84 p99.9=4521.98 max=4879.6 mean=2728.90 + cpu=63.98s ops/cpu-s=367 rd=0.0 B/op wr=0.2 B/op violations=0 errors=0 swapin=0 swapout=0 majflt=0 + startup us p50=22.27 p99=47.62 mean=24.21 per-tuple=277.1 ns deinit mean=1.20 us tuples/s=28593225 + insert/run=8.00s drain(close)=0.09s total=8.09s drain_cpu=0.01s drain_rd=0 drain_wr=311296 wamp=nan db_apparent=162664448 db_alloc=28655616 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/manifest.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/manifest.json new file mode 100644 index 000000000..c736feda2 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/manifest.json @@ -0,0 +1,487 @@ +{ + "mode": "resident", + "storage": "ext4 /dev/mapper/snafu-root_crypt[/home/rob/projects/splinterdb]", + "io": "O_DIRECT", + "cpu_affinity": "0-11", + "runner_sha256": "ad1a38ada90f6f275d482205f75eb359fda6bea5d21a58547a4724472db64da1", + "runs": [ + { + "label": "00-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/results.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "0", + "--rep", + "0", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791922402.1739323, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791922438.3284268, + "log_sha256": "94b181b0dea1e7faf167567fda59b8f7b5f49d322e0a15f881b2aea75af05274", + "process_cpu_s": 225.016528, + "process_read_bytes": 28295168, + "process_write_bytes": 339968, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "01-filtered-ranges", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/results.csv", + "--vtag", + "filtered-ranges", + "--stag", + "resident", + "--otag", + "1", + "--rep", + "1", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791922438.5836442, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "c97dd13b1c6041adf9a8499e06ac158c25961d7a8f5fc05a97050f103e124ce0", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791922474.7322187, + "log_sha256": "4dd228f28c7d054085fb4ad8271b3525dce900507706c43b3ca3d479c49fb572", + "process_cpu_s": 224.86496400000007, + "process_read_bytes": 28295168, + "process_write_bytes": 331776, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "02-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/results.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "2", + "--rep", + "2", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791922474.9739664, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791922511.1114485, + "log_sha256": "70fa3a858470944e9ecc26ab90fa085b308ad6f316156e2eee0be2776d5856ae", + "process_cpu_s": 225.06114299999996, + "process_read_bytes": 28295168, + "process_write_bytes": 344064, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "03-disk-format", + "profile": "disk-format", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/disk-format/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/results.csv", + "--vtag", + "disk-format", + "--stag", + "resident", + "--otag", + "3", + "--rep", + "3", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791922511.4086962, + "source_head": "3e302c04c60babb8afecb58d27f808848624f83c", + "binary_sha256": "1286d7a7f7004b36526296142aa31fbc746263b937761dad08c4dcd0e7334940", + "library_sha256": "4b2eb5decdc347afe224775b61566483c95f8ee58def3f30a2f9755a3721a1e4", + "verified": true, + "returncode": 0, + "finished": 1791922547.5722349, + "log_sha256": "1ead91833e3cfbdfd752896ca690912a741e0a86085c25f70273074f019d17e0", + "process_cpu_s": 225.10063699999998, + "process_read_bytes": 28295168, + "process_write_bytes": 344064, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-disk-format.db", + "fixture_sha256": "c853e677543afdfa7efcd67fa13e5c9d57b0a1465aff245641584f65c1dbecc4", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "04-filtered-ranges", + "profile": "filtered-ranges", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/filtered-ranges/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/results.csv", + "--vtag", + "filtered-ranges", + "--stag", + "resident", + "--otag", + "4", + "--rep", + "4", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791922547.8538508, + "source_head": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "binary_sha256": "c97dd13b1c6041adf9a8499e06ac158c25961d7a8f5fc05a97050f103e124ce0", + "library_sha256": "814e3899769002e29fae045a54cd3c58949c17bfa84b4a11198503fcf9ff6b51", + "verified": true, + "returncode": 0, + "finished": 1791922584.0136704, + "log_sha256": "0f4b57dee27b33f0abfd8d8439c07b8b890821d920a7ac355536b50c2d9f12b0", + "process_cpu_s": 225.14779599999991, + "process_read_bytes": 28295168, + "process_write_bytes": 348160, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + }, + { + "label": "05-main", + "profile": "main", + "command": [ + "taskset", + "-c", + "0-11", + "/home/rob/projects/splinterdb/.cache/projects-20261011/coordination/filter-predicate-20261013/benchmarks/bin/main/kvbench", + "--db", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/scratch.db", + "--open", + "--n", + "200000", + "--cache-mib", + "128", + "--disk-gib", + "8", + "--log", + "0", + "--threads", + "8", + "--mt-bg", + "0", + "--normal-bg", + "0", + "--memtable-mib", + "4", + "--secs", + "8", + "--warmup", + "1", + "--csv", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident/results.csv", + "--vtag", + "main", + "--stag", + "resident", + "--otag", + "5", + "--rep", + "5", + "--seed", + "42", + "--hang-secs", + "60", + "--hang-dir", + "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/resident", + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ], + "environment": {}, + "started": 1791922584.2982929, + "source_head": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "binary_sha256": "7bb9699e6097a4a9a3b608846d1c2531110be97f8496fe6dd35263a5c8b07597", + "library_sha256": "6d38ee29228da7b5b3fc9803357dbd868f25f80bec3e670c7a92c84312bd3a7f", + "verified": true, + "returncode": 0, + "finished": 1791922620.4970422, + "log_sha256": "14870934fcc0ac5f3b2dc9a9133bb128b705c6c7e45abbdb0338db7704233e77", + "process_cpu_s": 225.1307890000001, + "process_read_bytes": 28295168, + "process_write_bytes": 339968, + "verification_kind": "full warm scan and timed key/value/range checks; successful close", + "errors": 0, + "violations": 0, + "fixture": { + "fixture": "/home/rob/projects/splinterdb/.cache/projects-20261011/benchmarks/filter-predicate-20261013/fixtures/resident-main.db", + "fixture_sha256": "ca92c8cbc69dfd2cff48c94d92fc27cd801bd9a1062319e2bb41b94daf419718", + "allocated_bytes": 28905472, + "apparent_bytes": 162664448 + }, + "fixture_copy_fsynced": true, + "allocated_bytes": 28655616 + } + ], + "complete": true, + "order": [ + "main", + "filtered-ranges", + "disk-format", + "disk-format", + "filtered-ranges", + "main" + ], + "phases": [ + "warm", + "point:uniform@1", + "point:uniform@8", + "range:uniform:1@8", + "range:uniform:10000@8" + ] +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/results.csv b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/results.csv new file mode 100644 index 000000000..128e0a61f --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/resident/results.csv @@ -0,0 +1,31 @@ +version,size,order,log,phase,threads,rep,n,ops,secs,ops_s,p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s,rd_bytes,wr_bytes,rd_B_op,wr_B_op,startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns,deinit_mean_us,tuples,tuples_s,insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp,db_apparent,db_alloc,open_s,violations,errors,swapin,swapout,majflt +main,resident,0,0,warm,8,0,200000,200000,0.031,6357250.4,nan,nan,nan,0.000,nan,0.128,1561804.5,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,6357250.4,0.031,0.000,0.031,0.000,0,0,nan,0,0,0.009,0,0,0,0,1 +main,resident,0,0,point:uniform@1,1,0,200000,5541952,8.000,692716.3,1.360,1.872,4.800,67.998,1.401,8.000,692708.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.009,0,0,0,0,0 +main,resident,0,0,point:uniform@8,8,0,200000,29981074,8.000,3747424.5,1.968,4.288,10.624,4039.085,2.077,63.910,469112.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.009,0,0,0,0,0 +main,resident,0,0,range:uniform:1@8,8,0,200000,2904386,8.000,363033.5,21.248,40.448,60.928,901.033,21.969,63.966,45405.1,0,0,0.00,0.00,20.224,38.400,21.005,nan,0.903,2904386,363033.5,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.009,0,0,0,0,0 +main,resident,0,0,range:uniform:10000@8,8,0,200000,23159,8.000,2894.8,2785.280,4521.984,4915.200,5893.136,2763.223,63.984,362.0,0,4096,0.00,0.18,22.784,53.760,25.145,280.57,1.220,225847553,28229844.8,8.000,0.063,8.063,0.014,0,311296,nan,162664448,28655616,0.009,0,0,0,0,0 +filtered-ranges,resident,1,0,warm,8,1,200000,200000,0.035,5780389.8,nan,nan,nan,0.000,nan,0.135,1485177.9,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,5780389.8,0.035,0.000,0.035,0.000,0,0,nan,0,0,0.009,0,0,0,0,0 +filtered-ranges,resident,1,0,point:uniform@1,1,1,200000,5239678,8.000,654932.6,1.424,2.592,8.320,535.395,1.482,7.998,655095.0,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.009,0,0,90,0,0 +filtered-ranges,resident,1,0,point:uniform@8,8,1,200000,29719807,8.000,3714776.4,1.936,4.288,15.488,3081.390,2.095,63.929,464890.1,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.009,0,0,1,0,0 +filtered-ranges,resident,1,0,range:uniform:1@8,8,1,200000,2912265,8.000,364020.2,21.248,39.424,53.760,1466.234,21.908,63.981,45517.7,0,4096,0.00,0.00,20.224,37.376,20.931,nan,0.917,2912265,364020.2,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.009,0,0,2,0,0 +filtered-ranges,resident,1,0,range:uniform:10000@8,8,1,200000,22676,8.000,2834.4,2850.816,4521.984,5177.344,6820.967,2822.208,63.954,354.6,0,0,0.00,0.00,23.808,52.736,26.299,286.37,1.378,221217799,27651276.0,8.000,0.077,8.077,0.014,0,311296,nan,162664448,28655616,0.009,0,0,7,0,0 +disk-format,resident,2,0,warm,8,2,200000,200000,0.029,6883319.1,nan,nan,nan,0.000,nan,0.113,1773489.9,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,6883319.1,0.029,0.000,0.029,0.000,0,0,nan,0,0,0.008,0,0,0,0,1 +disk-format,resident,2,0,point:uniform@1,1,2,200000,5343451,8.000,667909.4,1.392,2.464,7.744,444.643,1.454,7.996,668264.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.008,0,0,9,0,0 +disk-format,resident,2,0,point:uniform@8,8,2,200000,27481449,8.000,3435058.8,2.144,4.288,9.088,746.225,2.268,63.968,429610.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.008,0,0,1,0,0 +disk-format,resident,2,0,range:uniform:1@8,8,2,200000,2943204,8.000,367889.5,21.248,39.424,52.736,4017.739,21.679,63.974,46006.4,0,0,0.00,0.00,20.224,37.376,20.700,nan,0.920,2943204,367889.5,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.008,0,0,0,0,0 +disk-format,resident,2,0,range:uniform:10000@8,8,2,200000,23300,8.000,2912.4,2850.816,4259.840,4521.984,5677.194,2746.428,63.987,364.1,0,4096,0.00,0.18,22.784,46.592,24.294,278.84,1.164,227391977,28422565.8,8.000,0.086,8.086,0.014,0,311296,nan,162664448,28655616,0.008,0,0,0,0,0 +disk-format,resident,3,0,warm,8,3,200000,200000,0.041,4837517.5,nan,nan,nan,0.000,nan,0.171,1170028.5,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,4837517.5,0.041,0.000,0.041,0.000,0,0,nan,0,0,0.014,0,0,0,0,0 +disk-format,resident,3,0,point:uniform@1,1,3,200000,5583921,8.000,697961.6,1.360,1.872,5.824,158.384,1.390,8.000,697980.1,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.014,0,0,0,0,0 +disk-format,resident,3,0,point:uniform@8,8,3,200000,29966090,8.000,3745574.9,1.968,4.000,10.112,2445.074,2.079,63.940,468662.2,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.014,0,0,3,0,0 +disk-format,resident,3,0,range:uniform:1@8,8,3,200000,3011053,8.000,376369.6,20.736,38.400,52.736,674.241,21.189,63.969,47070.8,0,4096,0.00,0.00,19.712,36.352,20.227,nan,0.904,3011053,376369.6,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.014,0,0,0,0,0 +disk-format,resident,3,0,range:uniform:10000@8,8,3,200000,23372,8.000,2921.4,2850.816,4259.840,4521.984,4894.025,2738.214,63.973,365.3,0,4096,0.00,0.18,22.784,46.592,24.251,278.01,1.167,227999183,28498966.3,8.000,0.076,8.076,0.014,0,311296,nan,162664448,28655616,0.014,0,0,0,0,0 +filtered-ranges,resident,4,0,warm,8,4,200000,200000,0.038,5211541.6,nan,nan,nan,0.000,nan,0.165,1209497.0,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,5211541.6,0.038,0.000,0.038,0.000,0,0,nan,0,0,0.011,0,0,0,0,2 +filtered-ranges,resident,4,0,point:uniform@1,1,4,200000,5532543,8.000,691544.1,1.360,2.208,6.080,149.377,1.403,7.997,691850.6,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.011,0,0,0,0,0 +filtered-ranges,resident,4,0,point:uniform@8,8,4,200000,30782885,8.000,3847725.4,1.904,3.872,9.344,2096.716,2.023,63.978,481146.1,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.011,0,0,0,0,0 +filtered-ranges,resident,4,0,range:uniform:1@8,8,4,200000,2985717,8.000,373202.3,21.248,38.400,52.736,163.170,21.371,63.989,46659.7,0,4096,0.00,0.00,20.224,37.376,20.424,nan,0.888,2985717,373202.3,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.011,0,0,0,0,0 +filtered-ranges,resident,4,0,range:uniform:10000@8,8,4,200000,23589,8.000,2948.5,2785.280,4259.840,4521.984,4700.062,2712.635,63.981,368.7,0,4096,0.00,0.17,22.272,44.544,24.152,275.51,1.235,230094388,28760765.5,8.000,0.071,8.071,0.007,0,311296,nan,162664448,28655616,0.011,0,0,1,0,0 +main,resident,5,0,warm,8,5,200000,200000,0.037,5432022.3,nan,nan,nan,0.000,nan,0.148,1351862.9,27672576,0,138.36,0.00,nan,nan,nan,nan,nan,200000,5432022.3,0.037,0.000,0.037,0.000,0,0,nan,0,0,0.012,0,0,0,0,0 +main,resident,5,0,point:uniform@1,1,5,200000,5510338,8.000,688766.2,1.392,1.840,5.312,222.702,1.409,8.003,688517.8,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.012,0,0,0,0,0 +main,resident,5,0,point:uniform@8,8,5,200000,30104822,8.000,3762978.7,1.936,3.936,10.112,2100.502,2.069,63.967,470627.7,0,0,0.00,0.00,nan,nan,nan,nan,nan,0,0.0,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.012,0,0,1,0,0 +main,resident,5,0,range:uniform:1@8,8,5,200000,3003959,8.000,375481.6,20.736,38.400,51.712,159.849,21.241,63.987,46946.2,0,4096,0.00,0.00,19.712,36.352,20.295,nan,0.887,3003959,375481.6,8.000,0.000,8.000,0.000,0,0,nan,0,0,0.012,0,0,0,0,0 +main,resident,5,0,range:uniform:10000@8,8,5,200000,23451,8.000,2931.2,2850.816,4259.840,4521.984,4879.598,2728.895,63.983,366.5,0,4096,0.00,0.17,22.272,47.616,24.213,277.11,1.199,228756533,28593224.7,8.000,0.087,8.087,0.014,0,311296,nan,162664448,28655616,0.012,0,0,0,0,0 diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/run.py b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/run.py new file mode 100644 index 000000000..a5eff00ca --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/run.py @@ -0,0 +1,151 @@ +"""Run each mode via run_job benchmark, only after root explicitly clears timing.""" +import argparse, csv, hashlib, json, os, re, resource, signal, subprocess, time +from pathlib import Path +ROOT=Path('/home/rob/projects/splinterdb'); HERE=Path(__file__).resolve().parent +OUT=ROOT/'.cache/projects-20261011/benchmarks/filter-predicate-20261013' +ORDER=['main','filtered-ranges','disk-format','disk-format','filtered-ranges','main'] + +def sha(p): + h=hashlib.sha256() + with Path(p).open('rb') as f: + for chunk in iter(lambda:f.read(1024*1024),b''): h.update(chunk) + return h.hexdigest() +def save(p,x): p.write_text(json.dumps(x,indent=2)+'\n') +def csvread(p): + with p.open() as f:return list(csv.DictReader(f)) +def build(name): + r=json.loads((HERE/(name+'-build.json')).read_text()) + ready=json.loads((HERE/'timing-ready.json').read_text()) + assert ready['heads'][name]==r['head'] + assert subprocess.check_output(['git','rev-parse','HEAD'],cwd=r['tree'],text=True).strip()==r['head'] + assert sha(r['library'])==r['library_sha256'] + for path,expected in r['source_files_sha256'].items(): assert sha(Path(r['tree'])/path)==expected,(name,path) + for b in r['binaries'].values(): + assert sha(b['binary'])==b['binary_sha256'] and sha(b['source'])==b['source_sha256'] + return r + +def mode_start(name): + folder=OUT/name; folder.mkdir(parents=True,exist_ok=False) + fs=subprocess.check_output(['findmnt','-n','-o','FSTYPE,SOURCE','-T',str(folder)],text=True).strip() + assert fs.split()[0] not in ('tmpfs','ramfs') and fs.split()[1].startswith('/dev/') + m=dict(mode=name,storage=fs,io='O_DIRECT',cpu_affinity='0-11',runner_sha256=sha(__file__),runs=[],complete=False) + save(folder/'manifest.json',m); return folder,m + +def run(folder,m,label,profile,binary,args,environment=None,timeout=600): + b=build(profile); exe=b['binaries'][binary] + command=['taskset','-c','0-11',exe['binary'],*map(str,args)] + rec=dict(label=label,profile=profile,command=command,environment=environment or {},started=time.time(), + source_head=b['head'],binary_sha256=exe['binary_sha256'],library_sha256=b['library_sha256'],verified=False) + m['runs'].append(rec);save(folder/'manifest.json',m); print(label,flush=True) + log=folder/(label+'.log'); before=resource.getrusage(resource.RUSAGE_CHILDREN) + try: + with log.open('w') as f: + p=subprocess.Popen(command,stdout=f,stderr=subprocess.STDOUT,env=dict(os.environ,**(environment or {})),start_new_session=True) + try: p.wait(timeout=timeout) + except BaseException: + try: os.killpg(p.pid,signal.SIGKILL) + except ProcessLookupError: pass + p.wait(); raise + after=resource.getrusage(resource.RUSAGE_CHILDREN) + rec.update(returncode=p.returncode,finished=time.time(),log_sha256=sha(log), + process_cpu_s=after.ru_utime+after.ru_stime-before.ru_utime-before.ru_stime, + process_read_bytes=512*(after.ru_inblock-before.ru_inblock),process_write_bytes=512*(after.ru_oublock-before.ru_oublock)) + assert p.returncode==0,rec + assert 'FINDING: splinterdb_close' not in log.read_text(),rec + except BaseException as error: + rec.update(exception=str(error),finished=time.time());save(folder/'manifest.json',m);raise + save(folder/'manifest.json',m);return rec + +def args(db,csvfile,profile,regime,index,create=False): + return ['--db',db,'--create' if create else '--open','--n',200000 if regime=='resident' else 2000000, + '--cache-mib',128 if regime=='resident' else 64,'--disk-gib',8,'--log',0, + '--threads',1 if create else 8,'--mt-bg',0,'--normal-bg',0,'--memtable-mib',4, + '--secs',8 if regime=='resident' else 6,'--warmup',1,'--csv',csvfile,'--vtag',profile, + '--stag',regime,'--otag',index,'--rep',index,'--seed',42,'--hang-secs',60,'--hang-dir',str(csvfile.parent)] + +def verify_csv(path,index,phases,profile): + rows=[r for r in csvread(path) if r['order']==str(index)] + assert [r['phase'] for r in rows]==phases and all(r['version']==profile for r in rows) + assert all(int(r['ops'])>0 and int(r['errors'])==0 and int(r['violations'])==0 for r in rows) + assert all(int(r['ops'])==int(r['n']) for r in rows if r['phase']=='warm' or r['phase'].startswith('load:')) + return rows + +def fixtures(): + folder,m=mode_start('fixtures'); fixtures={} + for regime in ['resident','disk']: + for profile in ['main','disk-format']: + label=regime+'-'+profile; db=folder/(label+'.db'); csvfile=folder/(label+'.csv') + rec=run(folder,m,label,profile,'kvbench',args(db,csvfile,profile,regime,0,True)+['load:random']) + rr=verify_csv(csvfile,0,['load:random'],profile) + with db.open('rb') as image:os.fsync(image.fileno()) + rec.update(verified=True,verification_kind='load count/API errors/close; warm scans later validate every key/value', + errors=0,violations=0,fixture=str(db),fixture_sha256=sha(db),fixture_fsynced=True, + allocated_bytes=db.stat().st_blocks*512,apparent_bytes=db.stat().st_size) + fixtures[label]={k:rec[k] for k in ['fixture','fixture_sha256','allocated_bytes','apparent_bytes']} + save(folder/'manifest.json',m);save(folder/'fixtures.json',fixtures) + m['complete']=True;save(folder/'manifest.json',m) + +def queries(regime): + folder,m=mode_start(regime) + phases=['warm','point:uniform@1','point:uniform@8','range:uniform:1@8','range:uniform:10000@8'] if regime=='resident' else ['warm','point:uniform@8','range:uniform:1@8','range:uniform:10000@8'] + m.update(order=ORDER,phases=phases) + fixtures=json.loads((OUT/'fixtures/fixtures.json').read_text()) + fixture_manifest=json.loads((OUT/'fixtures/manifest.json').read_text()) + assert fixture_manifest['complete'] and all(r['verified'] for r in fixture_manifest['runs']) + for i,profile in enumerate(ORDER): + fixture=fixtures[regime+'-'+('disk-format' if profile=='disk-format' else 'main')] + src=Path(fixture['fixture']); metadata=src.stat(); db=folder/'scratch.db'; assert not db.exists() + assert sha(src)==fixture['fixture_sha256'],fixture['fixture'] + subprocess.run(['cp','--reflink=auto','--sparse=always',str(src),str(db)],check=True) + with db.open('rb') as image:os.fsync(image.fileno()) + csvfile=folder/'results.csv'; label=f'{i:02d}-{profile}' + rec=run(folder,m,label,profile,'kvbench',args(db,csvfile,profile,regime,i)+phases) + rr=verify_csv(csvfile,i,phases,profile) + if regime=='resident':assert all(int(r['rd_bytes'])==0 for r in rr if r['phase']!='warm'),rr + rec.update(verified=True,verification_kind='full warm scan and timed key/value/range checks; successful close', + errors=sum(int(r['errors']) for r in rr),violations=sum(int(r['violations']) for r in rr),fixture=fixture, + fixture_copy_fsynced=True,allocated_bytes=db.stat().st_blocks*512) + assert src.stat().st_mtime_ns==metadata.st_mtime_ns and src.stat().st_size==metadata.st_size + save(folder/'manifest.json',m); db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +def expected_checksum(rows,queries,limit): + seed=42;checksum=0 + for q in range(queries): + seed=(seed*6364136223846793005+1442695040888963407)&((1<<64)-1) + start=(seed>>32)%(rows-limit) + checksum+=limit*(start+q%32)+limit*(limit-1)//2 + return checksum + +def feature(): + folder,m=mode_start('feature'); count=10000; n=16384 + cases=[('main','application'),('filtered-ranges','predicate'),('filtered-ranges','pruned'), + ('filtered-ranges','pruned'),('filtered-ranges','predicate'),('main','application')] + m.update(queries_per_limit=count,rows_per_tenant=n,warmup_queries_per_limit=500,order=cases) + for collisions in [0,1]: + for i,(profile,mode) in enumerate(cases): + label=f'collisions{collisions}-{i:02d}-{profile}-{mode}';db=folder/'scratch.db';assert not db.exists() + rec=run(folder,m,label,profile,'hash_range_benchmark',[db,n,count], + {'FILTER_MODE':mode,'FILTER_HASH_COLLISIONS':str(collisions)},timeout=900) + lines=(folder/(label+'.log')).read_text().splitlines() + assert sum(line.startswith('loaded=524288 ') for line in lines)==1 + header=next(i for i,line in enumerate(lines) if line.startswith('mode,collisions,')) + rr=list(csv.DictReader(lines[header:]));assert len(rr)==4 + assert [int(r['limit']) for r in rr]==[1,2,10,100] + for r in rr: + limit=int(r['limit']) + assert r['mode']==mode and int(r['collisions'])==collisions and int(r['queries'])==count and int(r['keys_per_tenant'])==n + assert int(r['checksum'])==expected_checksum(n,count,limit) + assert int(r['warmup_queries'])==500 and float(r['ns_per_query'])>0 and float(r['cpu_s'])>0 + if mode!='application':assert int(r['visited'])==limit*count + else:assert int(r['predicate_calls'])==int(r['visited']) + rec.update(verified=True,verification_kind='per-query expected key/value/count checks and independently computed checksums', + rows=rr,errors=0,violations=0,allocated_bytes=db.stat().st_blocks*512) + save(folder/'manifest.json',m);db.unlink() + m['complete']=True;save(folder/'manifest.json',m) + +if __name__=='__main__': + def interrupted(signum,frame):raise InterruptedError(f'signal {signum}') + for s in [signal.SIGTERM,signal.SIGHUP]:signal.signal(s,interrupted) + p=argparse.ArgumentParser();p.add_argument('mode',choices=['fixtures','resident','disk','feature']);a=p.parse_args() + {'fixtures':fixtures,'resident':lambda:queries('resident'),'disk':lambda:queries('disk'),'feature':feature}[a.mode]() diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/hash_range_benchmark.c b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/hash_range_benchmark.c new file mode 100644 index 000000000..7be6887e5 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/hash_range_benchmark.c @@ -0,0 +1,262 @@ +// Copyright 2026 VMware, Inc. +// SPDX-License-Identifier: Apache-2.0 + +/* Public-API comparison for predicate-filtered ranges. Compile the same source + * against the baseline (scan + application predicate) and with + * -DPREDICATE_RANGE_FILTER_API against the candidate. Run only under the + * campaign's exclusive benchmark wrapper, with db_path on the verified disk + * filesystem. */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "splinterdb/default_data_config.h" +#include "splinterdb/splinterdb.h" + +#define TENANTS 32 + +static bool hash_collisions; + +typedef struct tenant_filter { + uint64_t tenant; + uint64_t calls; +} tenant_filter; + +static uint64_t +read_be(const unsigned char *p) +{ + uint64_t v = 0; + for (int i = 0; i < 8; i++) { + v = (v << 8) | p[i]; + } + return v; +} + +static void +write_be(unsigned char *p, uint64_t v) +{ + for (int i = 7; i >= 0; i--) { + p[i] = v; + v >>= 8; + } +} + +static uint32_t +hash_tenant(const data_config *cfg, user_key k, uint32_t seed) +{ + const unsigned char *bytes = slice_data(k.key); + assert(slice_length(k.key) == 16); + uint32_t tenant = (uint32_t)read_be(bytes + 8); + return (hash_collisions ? tenant / 2 : tenant) * 2654435761U ^ seed; +} + +static bool +tenant_matches(void *context, slice key) +{ + tenant_filter *filter = context; + filter->calls++; + assert(slice_length(key) == 16); + return read_be((const unsigned char *)slice_data(key) + 8) == filter->tenant; +} + +static uint64_t +nanos(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000 + ts.tv_nsec; +} + +static void +check(int rc) +{ + if (rc != 0) { + fprintf(stderr, "SplinterDB error: %s (%d)\n", strerror(rc), rc); + exit(1); + } +} + +static void +run_queries(splinterdb *db, + uint64_t rows, + uint64_t queries, + int limit, + const char *mode, + tenant_filter *filter, + uint64_t *visited, + uint64_t *checksum) +{ + uint64_t seed = 42; + unsigned char key[16]; + for (uint64_t q = 0; q < queries; q++) { + seed = seed * 6364136223846793005ULL + 1442695040888963407ULL; + uint64_t start = (seed >> 32) % (rows - limit); + uint64_t tenant = q % TENANTS; + filter->tenant = tenant; + unsigned char hash_key[16] = {0}; + write_be(hash_key + 8, tenant); + write_be(key, start); + write_be(key + 8, 0); + splinterdb_iterator *it = NULL; +#ifdef PREDICATE_RANGE_FILTER_API + check(splinterdb_iterator_init_with_filter( + db, + &it, + greater_than_or_equal, + slice_create(sizeof(key), key), + tenant_matches, + filter, + strcmp(mode, "pruned") == 0 ? slice_create(sizeof(hash_key), hash_key) + : NULL_SLICE)); +#else + check(splinterdb_iterator_init( + db, &it, greater_than_or_equal, slice_create(sizeof(key), key))); +#endif + int count = 0; + while (splinterdb_iterator_valid(it)) { + slice k, v; + splinterdb_iterator_get_current(it, &k, &v); + (*visited)++; + const unsigned char *bytes = slice_data(k); +#ifdef PREDICATE_RANGE_FILTER_API + bool matches = read_be(bytes + 8) == tenant; +#else + bool matches = tenant_matches(filter, k); +#endif + if (matches) { + assert(read_be(bytes) == start + count); + assert(slice_length(v) == 64); + assert(memcmp(slice_data(v), bytes, sizeof(key)) == 0); + *checksum += read_be(bytes) + tenant; + if (++count == limit) { + break; + } + } +#ifdef PREDICATE_RANGE_FILTER_API + else + { + assert(0 && "nonmatching key from filtered iterator"); + } +#endif + splinterdb_iterator_next(it); + } + check(splinterdb_iterator_status(it)); + assert(count == limit); + splinterdb_iterator_deinit(it); + } +} + +int +main(int argc, char **argv) +{ + if (argc < 2 || argc > 4) { + fprintf(stderr, + "usage: %s db_path [keys_per_tenant=16384] [queries=2000]\n", + argv[0]); + return 2; + } + uint64_t rows = argc > 2 ? strtoull(argv[2], NULL, 10) : 16384; + uint64_t queries = argc > 3 ? strtoull(argv[3], NULL, 10) : 2000; + assert(rows > 100 && queries > 0); + const char *collision_env = getenv("FILTER_HASH_COLLISIONS"); + hash_collisions = collision_env && strcmp(collision_env, "1") == 0; +#ifdef PREDICATE_RANGE_FILTER_API + const char *mode = getenv("FILTER_MODE"); + if (mode == NULL) { + mode = "pruned"; + } + if (strcmp(mode, "pruned") != 0 && strcmp(mode, "predicate") != 0) { + fprintf(stderr, "FILTER_MODE must be pruned or predicate\n"); + return 2; + } +#else + const char *mode = "application"; +#endif + data_config data_cfg; + default_data_config_init(&data_cfg); + data_cfg.key_hash = hash_tenant; + splinterdb_config cfg = {.filename = argv[1], + .io_flags = O_RDWR | O_CREAT | O_DIRECT, + .cache_size = 64ULL << 20, + .disk_size = 2ULL << 30, + .memtable_capacity = 4ULL << 20, + .data_cfg = &data_cfg}; + splinterdb *db = NULL; + check(splinterdb_create(&cfg, &db)); + unsigned char key[16]; + unsigned char value[64] = {0}; + uint64_t begin = nanos(); + /* Grouped ingestion leaves some immutable branches selective by tenant; + * queries run before close/reopen compacts away that arrangement. */ + for (uint64_t tenant = 0; tenant < TENANTS; tenant++) { + write_be(key + 8, tenant); + for (uint64_t i = 0; i < rows; i++) { + write_be(key, i); + memcpy(value, key, sizeof(key)); + check(splinterdb_insert(db, + slice_create(sizeof(key), key), + slice_create(sizeof(value), value), + NULL)); + } + } + fprintf(stderr, + "loaded=%" PRIu64 " elapsed_ns=%" PRIu64 "\n", + rows * TENANTS, + nanos() - begin); + printf( + "mode,collisions,keys_per_tenant,limit,queries,ns_per_query,visited," + "predicate_calls,checksum,warmup_queries,cpu_s,read_bytes,write_bytes\n"); + const int limits[] = {1, 2, 10, 100}; + for (unsigned int l = 0; l < sizeof(limits) / sizeof(limits[0]); l++) { + const uint64_t warmup_queries = queries < 500 ? queries : 500; + uint64_t visited = 0, checksum = 0; + tenant_filter filter = {0}; + /* Replay the same deterministic query sequence before timing. Warmup + * does the same validation, and none of its counts enter the result. */ + run_queries(db, + rows, + warmup_queries, + limits[l], + mode, + &filter, + &visited, + &checksum); + visited = checksum = filter.calls = 0; + struct rusage before, after; + check(getrusage(RUSAGE_SELF, &before)); + begin = nanos(); + run_queries( + db, rows, queries, limits[l], mode, &filter, &visited, &checksum); + uint64_t elapsed = nanos() - begin; + check(getrusage(RUSAGE_SELF, &after)); + double cpu = after.ru_utime.tv_sec - before.ru_utime.tv_sec + + after.ru_stime.tv_sec - before.ru_stime.tv_sec + + (after.ru_utime.tv_usec - before.ru_utime.tv_usec + + after.ru_stime.tv_usec - before.ru_stime.tv_usec) + / 1e6; + printf("%s,%d,%" PRIu64 ",%d,%" PRIu64 ",%.1f,%" PRIu64 ",%" PRIu64 + ",%" PRIu64 ",%" PRIu64 ",%.6f,%" PRIu64 ",%" PRIu64 "\n", + mode, + hash_collisions, + rows, + limits[l], + queries, + (double)elapsed / queries, + visited, + filter.calls, + checksum, + warmup_queries, + cpu, + (uint64_t)(after.ru_inblock - before.ru_inblock) * 512, + (uint64_t)(after.ru_oublock - before.ru_oublock) * 512); + fflush(stdout); + } + check(splinterdb_close(&db, false)); + return 0; +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/kvbench.c b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/kvbench.c new file mode 100644 index 000000000..9b91f4ef9 --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/sources/kvbench.c @@ -0,0 +1,1707 @@ +// kvbench.c -- SplinterDB public-API benchmark used to compare two versions. +// +// Uses only splinterdb/splinterdb.h + splinterdb/default_data_config.h + libc, +// so the same source builds against versions whose internals differ. The one +// public-API difference handled here is splinterdb_close(): +// -DKVBENCH_CLOSE_HAS_FORCE => int splinterdb_close(splinterdb **, bool32) +// otherwise => void splinterdb_close(splinterdb **) +// +// Key scheme: logical id in [0,N). Present key = 8-byte big-endian(2*id) +// followed by 16 bytes of fixed padding (24 bytes total), so memcmp order == +// id order. Absent ("negative") keys use 2*id+1, interleaved with present +// keys. Value = 100 bytes: id (LE u64), generation (LE u64), filler. +// +// Each invocation: create/open the DB, run the listed phases in order, close. +// Phase syntax (optional "@T" overrides the thread count): +// load:random | load:seq fixed-count insert of all N ids, then close +// (close timed separately as "drain") +// warm parallel full scan; verifies every id once +// point:uniform|zipf|seq|neg fixed-duration lookups +// range:uniform|zipf:L fixed-duration forward range queries +// ow:uniform|zipf fixed-duration overwrites (insert existing) +// One CSV row per phase is appended to --csv. + +#ifndef _GNU_SOURCE +# define _GNU_SOURCE +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "splinterdb/default_data_config.h" +#include "splinterdb/splinterdb.h" + +#define KEY_SIZE 24 +#define VALUE_SIZE 100 +#define MAX_PHASES 64 +#define MAX_WORKERS 32 + +static const char KEY_PAD[16] = "kvbench-padding!"; + +/* ------------------------------------------------------------------------ */ +/* Time, RNG, permutations, zipf */ +/* ------------------------------------------------------------------------ */ + +static inline uint64_t +now_ns(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; +} + +/* Bijective 64-bit mixer (splitmix64 finalizer). */ +static inline uint64_t +mix(uint64_t x) +{ + x = (x ^ (x >> 30)) * UINT64_C(0xbf58476d1ce4e5b9); + x = (x ^ (x >> 27)) * UINT64_C(0x94d049bb133111eb); + return x ^ (x >> 31); +} + +static inline uint64_t +rng_next(uint64_t *s) +{ + *s += UINT64_C(0x9e3779b97f4a7c15); + return mix(*s); +} + +static inline double +rng_unit(uint64_t *s) +{ + return (double)(rng_next(s) >> 11) * 0x1.0p-53; +} + +/* + * Pseudo-random bijection on [0,n): a 4-round balanced Feistel network over + * 2*half bits (2^(2*half) >= n), with cycle-walking to stay inside [0,n). + */ +typedef struct { + uint64_t n; + unsigned half; + uint64_t mask; + uint64_t keys[4]; +} perm_t; + +static void +perm_init(perm_t *p, uint64_t n, uint64_t seed) +{ + unsigned bits = 1; + while (bits < 64 && (UINT64_C(1) << bits) < n) { + bits++; + } + p->n = n; + p->half = (bits + 1) / 2; + if (p->half < 1) { + p->half = 1; + } + p->mask = (UINT64_C(1) << p->half) - 1; + for (int r = 0; r < 4; r++) { + p->keys[r] = mix(seed * UINT64_C(0x632be59bd9b4e019) + r + 1); + } +} + +static inline uint64_t +feistel(const perm_t *p, uint64_t x) +{ + uint64_t l = x >> p->half, r = x & p->mask; + for (int i = 0; i < 4; i++) { + uint64_t t = l ^ (mix(r ^ p->keys[i]) & p->mask); + l = r; + r = t; + } + return (l << p->half) | r; +} + +static inline uint64_t +perm_apply(const perm_t *p, uint64_t i) +{ + uint64_t x = i; + do { + x = feistel(p, x); + } while (x >= p->n); + return x; +} + +/* YCSB-style zipfian over ranks [0,n). */ +typedef struct { + uint64_t n; + double theta, zetan, zeta2, alpha, eta, half_pow_theta; +} zipf_t; + +static double +zeta_sum(uint64_t n, double theta) +{ + uint64_t m = n < 2000000 ? n : 2000000; + double s = 0; + for (uint64_t i = 1; i <= m; i++) { + s += pow((double)i, -theta); + } + if (n > m) { + /* Euler-Maclaurin tail: sum_{k=m+1}^{n} k^-t */ + s += (pow((double)n, 1 - theta) - pow((double)m, 1 - theta)) / (1 - theta) + + 0.5 * (pow((double)n, -theta) - pow((double)m, -theta)); + } + return s; +} + +static void +zipf_init(zipf_t *z, uint64_t n, double theta) +{ + z->n = n; + z->theta = theta; + z->zetan = zeta_sum(n, theta); + z->zeta2 = 1.0 + pow(0.5, theta); + z->alpha = 1.0 / (1.0 - theta); + z->eta = + (1.0 - pow(2.0 / (double)n, 1.0 - theta)) / (1.0 - z->zeta2 / z->zetan); + z->half_pow_theta = pow(0.5, theta); +} + +static inline uint64_t +zipf_next(const zipf_t *z, uint64_t *rng) +{ + double u = rng_unit(rng); + double uz = u * z->zetan; + if (uz < 1.0) { + return 0; + } + if (uz < 1.0 + z->half_pow_theta) { + return 1; + } + uint64_t r = + (uint64_t)((double)z->n * pow(z->eta * u - z->eta + 1.0, z->alpha)); + return r >= z->n ? z->n - 1 : r; +} + +/* ------------------------------------------------------------------------ */ +/* Histogram: log2 buckets with 32 linear sub-buckets (~3% resolution) */ +/* ------------------------------------------------------------------------ */ + +#define HSUB_BITS 5 +#define HSUB (1u << HSUB_BITS) +#define HBUCKETS (64 * HSUB) + +typedef struct { + uint64_t b[HBUCKETS]; + uint64_t count, sum, max; +} hist_t; + +static inline unsigned +hidx(uint64_t v) +{ + if (v < HSUB) { + return (unsigned)v; + } + unsigned e = 63 - __builtin_clzll(v); + unsigned sub = (unsigned)((v >> (e - HSUB_BITS)) & (HSUB - 1)); + return (e - HSUB_BITS + 1) * HSUB + sub; +} + +static inline uint64_t +hval(unsigned idx) +{ + if (idx < HSUB) { + return idx; + } + unsigned e = idx / HSUB + HSUB_BITS - 1; + unsigned sub = idx % HSUB; + uint64_t lo = (UINT64_C(1) << e) | ((uint64_t)sub << (e - HSUB_BITS)); + uint64_t width = UINT64_C(1) << (e - HSUB_BITS); + return lo + width / 2; +} + +static inline void +hist_add(hist_t *h, uint64_t v) +{ + h->b[hidx(v)]++; + h->count++; + h->sum += v; + if (v > h->max) { + h->max = v; + } +} + +static void +hist_merge(hist_t *dst, const hist_t *src) +{ + for (unsigned i = 0; i < HBUCKETS; i++) { + dst->b[i] += src->b[i]; + } + dst->count += src->count; + dst->sum += src->sum; + if (src->max > dst->max) { + dst->max = src->max; + } +} + +static double +hist_pct_us(const hist_t *h, double p) +{ + if (!h->count) { + return NAN; + } + uint64_t rank = (uint64_t)ceil(p * (double)h->count); + if (rank < 1) { + rank = 1; + } + uint64_t acc = 0; + for (unsigned i = 0; i < HBUCKETS; i++) { + acc += h->b[i]; + if (acc >= rank) { + uint64_t v = hval(i); + return (double)(v > h->max ? h->max : v) / 1e3; + } + } + return (double)h->max / 1e3; +} + +static double +hist_mean_us(const hist_t *h) +{ + return h->count ? (double)h->sum / (double)h->count / 1e3 : NAN; +} + +/* ------------------------------------------------------------------------ */ +/* Options / global state */ +/* ------------------------------------------------------------------------ */ + +enum phase_kind { PH_LOAD, PH_WARM, PH_POINT, PH_RANGE, PH_OW }; +enum dist { D_UNIFORM, D_ZIPF, D_SEQ, D_NEG, D_RANDOM }; + +typedef struct { + char name[64]; + enum phase_kind kind; + enum dist dist; + uint64_t L; + int threads; +} phase_t; + +typedef struct { + const char *db, *csv, *hang_dir; + const char *vtag, *stag, *otag; + int rep; + bool create; + uint64_t n, cache_mib, disk_gib, memtable_mib; + int use_log, threads, mt_bg, normal_bg; + double secs, warmup; + uint64_t sabotage; + uint64_t hang_secs; + uint64_t seed; + int nphases; + phase_t phases[MAX_PHASES]; +} options; + +static options opt; +static splinterdb *kvs; +static data_config data_cfg; +static perm_t load_perm, zipf_perm; +static zipf_t zipf; +static uint64_t generation; /* value generation for overwrites */ + +enum { ST_WAIT = 0, ST_WARMUP = 1, ST_MEASURE = 2, ST_STOP = 3 }; +static _Atomic int run_state; +static _Atomic uint64_t seq_counter; + +/* Watchdog */ +enum { WD_IDLE, WD_OPS, WD_OPENCLOSE }; +static _Atomic int wd_mode; +static _Atomic uint64_t wd_mode_since_ns; +static const char *wd_what = ""; + +typedef struct worker { + pthread_t th; + int idx; + const phase_t *ph; + hist_t lat; + hist_t startup; + uint64_t ops, tuples, pt_ns, pt_div, deinit_ns, deinit_n; + uint64_t violations, errors, notfound, found; + uint64_t rng; + uint64_t measure_rng; + bool measure_rng_captured; + _Atomic uint64_t progress; + char first_violation[256]; +} worker_t; + +static worker_t *workers; +static int nworkers; + +/* ------------------------------------------------------------------------ */ +/* Keys / values */ +/* ------------------------------------------------------------------------ */ + +static inline void +make_key(uint64_t k, char *key) +{ + for (int i = 0; i < 8; i++) { + key[i] = (char)(k >> (56 - 8 * i)); + } + memcpy(key + 8, KEY_PAD, 16); +} + +static inline bool +decode_key(slice s, uint64_t *k) +{ + if (slice_length(s) != KEY_SIZE) { + return false; + } + const unsigned char *p = slice_data(s); + uint64_t x = 0; + for (int i = 0; i < 8; i++) { + x = (x << 8) | p[i]; + } + *k = x; + return memcmp(p + 8, KEY_PAD, 16) == 0; +} + +static inline void +make_value(uint64_t id, uint64_t gen, char *val) +{ + memcpy(val, &id, 8); + memcpy(val + 8, &gen, 8); + memset(val + 16, 'v', VALUE_SIZE - 16); +} + +static inline bool +value_id_ok(slice v, uint64_t id) +{ + if (slice_length(v) != VALUE_SIZE) { + return false; + } + uint64_t got; + memcpy(&got, slice_data(v), 8); + return got == id; +} + +static inline bool +value_generation_ok(slice v, uint64_t id, uint64_t expected_generation) +{ + if (!value_id_ok(v, id)) { + return false; + } + uint64_t got_generation; + memcpy(&got_generation, (const char *)slice_data(v) + 8, 8); + return got_generation == expected_generation; +} + +static void +violation(worker_t *w, const char *fmt, ...) + __attribute__((format(printf, 2, 3))); + +static void +violation(worker_t *w, const char *fmt, ...) +{ + w->violations++; + if (w->violations <= 3) { + char buf[256]; + va_list ap; + va_start(ap, fmt); + vsnprintf(buf, sizeof(buf), fmt, ap); + va_end(ap); + if (w->violations == 1) { + snprintf(w->first_violation, sizeof(w->first_violation), "%s", buf); + } + fprintf(stderr, "VIOLATION [thread %d]: %s\n", w->idx, buf); + } +} + +/* ------------------------------------------------------------------------ */ +/* Process metrics */ +/* ------------------------------------------------------------------------ */ + +typedef struct { + uint64_t t_ns; + double cpu_s; + uint64_t rd, wr; + uint64_t swpin, swpout; + uint64_t majflt; +} snap_t; + +static void +proc_io(uint64_t *rd, uint64_t *wr) +{ + *rd = *wr = 0; + FILE *f = fopen("/proc/self/io", "r"); + if (!f) { + return; + } + char name[64]; + uint64_t v; + while (fscanf(f, "%63s %" SCNu64, name, &v) == 2) { + if (!strcmp(name, "read_bytes:")) { + *rd = v; + } else if (!strcmp(name, "write_bytes:")) { + *wr = v; + } + } + fclose(f); +} + +static void +vm_swap(uint64_t *in, uint64_t *out) +{ + *in = *out = 0; + FILE *f = fopen("/proc/vmstat", "r"); + if (!f) { + return; + } + char name[64]; + uint64_t v; + while (fscanf(f, "%63s %" SCNu64, name, &v) == 2) { + if (!strcmp(name, "pswpin")) { + *in = v; + } else if (!strcmp(name, "pswpout")) { + *out = v; + } + } + fclose(f); +} + +static void +take_snap(snap_t *s) +{ + struct rusage ru; + getrusage(RUSAGE_SELF, &ru); + s->cpu_s = ru.ru_utime.tv_sec + ru.ru_utime.tv_usec / 1e6 + + ru.ru_stime.tv_sec + ru.ru_stime.tv_usec / 1e6; + s->majflt = ru.ru_majflt; + proc_io(&s->rd, &s->wr); + vm_swap(&s->swpin, &s->swpout); + s->t_ns = now_ns(); +} + +/* ------------------------------------------------------------------------ */ +/* DB lifecycle */ +/* ------------------------------------------------------------------------ */ + +static void +fill_config(splinterdb_config *cfg) +{ + *cfg = (splinterdb_config){ + .filename = opt.db, + .cache_size = opt.cache_mib << 20, + .disk_size = opt.disk_gib << 30, + .data_cfg = &data_cfg, + .io_flags = O_RDWR | O_CREAT | O_DIRECT, + .io_perms = 0644, + .use_log = opt.use_log ? 1 : 0, + .memtable_capacity = opt.memtable_mib << 20, + .fanout = 8, + .num_memtable_bg_threads = (uint64)opt.mt_bg, + .num_normal_bg_threads = (uint64)opt.normal_bg, + .queue_scale_percent = 100, + }; +#ifdef KVBENCH_SPACE_MODE + cfg->use_stats = 1; +#endif +} + +static double open_secs; + +static void +db_open(bool create) +{ + splinterdb_config cfg; + fill_config(&cfg); + wd_what = create ? "create" : "open"; + atomic_store(&wd_mode_since_ns, now_ns()); + atomic_store(&wd_mode, WD_OPENCLOSE); + uint64_t t0 = now_ns(); + int rc = + create ? splinterdb_create(&cfg, &kvs) : splinterdb_open(&cfg, &kvs); + open_secs = (now_ns() - t0) / 1e9; + atomic_store(&wd_mode, WD_IDLE); + if (rc) { + fprintf(stderr, "FATAL: splinterdb_%s failed rc=%d\n", wd_what, rc); + exit(3); + } + fprintf(stderr, "%s took %.3f s\n", wd_what, open_secs); +} + +static double +db_close(void) +{ +#ifdef KVBENCH_SPACE_MODE + /* Space/stats build only: dump insertion + allocator stats for the session. + */ + if (getenv("KVBENCH_STATS_BEFORE_CLOSE")) { + platform_set_log_streams(stdout, stderr); + splinterdb_stats_print_insertion(kvs); + fflush(stdout); + } +#endif + wd_what = "close"; + atomic_store(&wd_mode_since_ns, now_ns()); + atomic_store(&wd_mode, WD_OPENCLOSE); + uint64_t t0 = now_ns(); +#ifdef KVBENCH_CLOSE_HAS_FORCE + int rc = splinterdb_close(&kvs, 0); + if (rc) { + fprintf(stderr, + "FINDING: splinterdb_close(force=0) refused rc=%d; forcing\n", + rc); + rc = splinterdb_close(&kvs, 1); + if (rc) { + fprintf(stderr, "FINDING: splinterdb_close(force=1) rc=%d\n", rc); + } + } +#else + splinterdb_close(&kvs); +#endif + double s = (now_ns() - t0) / 1e9; + atomic_store(&wd_mode, WD_IDLE); + kvs = NULL; + fprintf(stderr, "close took %.3f s\n", s); + return s; +} + +/* ------------------------------------------------------------------------ */ +/* Watchdog */ +/* ------------------------------------------------------------------------ */ + +static void +hang_dump(const char *why) +{ + char path[512], cmd[1024]; + snprintf(path, + sizeof(path), + "%s/hang-%s-%s-%d-%ld.txt", + opt.hang_dir ? opt.hang_dir : ".", + opt.vtag, + wd_what, + (int)getpid(), + (long)time(NULL)); + fprintf(stderr, + "HANG DETECTED (%s) during %s; dumping stacks to %s\n", + why, + wd_what, + path); + snprintf(cmd, + sizeof(cmd), + "gdb -batch -ex 'info threads' -ex 'thread apply all bt' -p %d > " + "'%s' 2>&1", + (int)getpid(), + path); + int rc = system(cmd); + fprintf(stderr, "gdb rc=%d; exiting with status 99\n", rc); + fflush(stderr); + _exit(99); +} + +static void * +watchdog_main(void *arg) +{ + (void)arg; + uint64_t last_prog = 0, last_change = now_ns(), last_report = now_ns(); + uint64_t last_report_prog = 0; + double last_cpu = -1; + uint64_t last_io = 0; + int last_mode = WD_IDLE; + for (;;) { + sleep(1); + int mode = atomic_load(&wd_mode); + uint64_t now = now_ns(); + if (mode != last_mode) { + last_mode = mode; + last_change = now; + last_prog = 0; + last_report_prog = 0; + last_report = now; + last_cpu = -1; + } + if (mode == WD_OPS) { + uint64_t prog = 0; + for (int i = 0; i < nworkers; i++) { + prog += + atomic_load_explicit(&workers[i].progress, memory_order_relaxed); + } + if (prog != last_prog) { + last_prog = prog; + last_change = now; + } else if (now - last_change > opt.hang_secs * 1000000000ull) { + hang_dump("no op progress for hang_secs"); + } + if (now - last_report >= 10ull * 1000000000ull) { + fprintf(stderr, + " [%s] progress %" PRIu64 " ops (+%.0f ops/s)\n", + wd_what, + prog, + (prog - last_report_prog) / ((now - last_report) / 1e9)); + last_report = now; + last_report_prog = prog; + } + } else if (mode == WD_OPENCLOSE) { + snap_t s; + take_snap(&s); + uint64_t io = s.rd + s.wr; + if (last_cpu < 0 || s.cpu_s - last_cpu > 0.05 || io != last_io) { + last_cpu = s.cpu_s; + last_io = io; + last_change = now; + } else if (now - last_change > opt.hang_secs * 1000000000ull) { + hang_dump("no CPU or IO progress for hang_secs"); + } + uint64_t since = atomic_load(&wd_mode_since_ns); + if (now - since > 2400ull * 1000000000ull) { + hang_dump("open/close exceeded 40 minutes"); + } + } + } + return NULL; +} + +/* ------------------------------------------------------------------------ */ +/* Worker bodies */ +/* ------------------------------------------------------------------------ */ + +static inline uint64_t +pick_id(worker_t *w, enum dist d, uint64_t *seqpos, uint64_t lo, uint64_t hi) +{ + switch (d) { + case D_ZIPF: + return perm_apply(&zipf_perm, zipf_next(&zipf, &w->rng)); + case D_SEQ: + { + uint64_t id = *seqpos; + *seqpos = (id + 1 >= hi) ? lo : id + 1; + return id; + } + case D_UNIFORM: + case D_NEG: + default: + return rng_next(&w->rng) % opt.n; + } +} + +static inline void +wait_start(void) +{ + while (atomic_load_explicit(&run_state, memory_order_acquire) == ST_WAIT) { + struct timespec d = {.tv_nsec = 100000}; + nanosleep(&d, NULL); + } +} + +static void * +load_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE], val[VALUE_SIZE]; + uint64_t T = (uint64_t)nworkers; + uint64_t lo = opt.n * (uint64_t)w->idx / T, + hi = opt.n * (uint64_t)(w->idx + 1) / T; + wait_start(); + uint64_t pos = lo, chunk_end = 0, cur = 0; + for (;;) { + uint64_t id; + if (w->ph->dist == D_RANDOM) { + if (pos >= hi) { + break; + } + id = perm_apply(&load_perm, pos++); + } else { + if (cur >= chunk_end) { + cur = atomic_fetch_add(&seq_counter, 64); + if (cur >= opt.n) { + break; + } + chunk_end = cur + 64 > opt.n ? opt.n : cur + 64; + } + id = cur++; + } + make_key(2 * id, key); + make_value(id, 0, val); + uint64_t t0 = now_ns(); + int rc = splinterdb_insert( + kvs, slice_create(KEY_SIZE, key), slice_create(VALUE_SIZE, val), NULL); + uint64_t t1 = now_ns(); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: insert id=%" PRIu64 " rc=%d\n", id, rc); + } + } + hist_add(&w->lat, t1 - t0); + w->ops++; + atomic_store_explicit(&w->progress, w->ops, memory_order_relaxed); + } + return NULL; +} + +static void * +warm_worker(void *arg) +{ + worker_t *w = arg; + uint64_t T = (uint64_t)nworkers; + uint64_t lo = opt.n * (uint64_t)w->idx / T, + hi = opt.n * (uint64_t)(w->idx + 1) / T; + if ((opt.sabotage & 8) && w->idx == nworkers - 1) { + hi++; /* sabotage: expect one more id than exists */ + } + char key[KEY_SIZE]; + make_key(2 * lo, key); + wait_start(); + splinterdb_iterator *it; + int rc = splinterdb_iterator_init( + kvs, &it, greater_than_or_equal, slice_create(KEY_SIZE, key)); + if (rc) { + w->errors++; + fprintf(stderr, "ERROR: warm iterator_init rc=%d\n", rc); + return NULL; + } + uint64_t expect = lo; + while (expect < hi && splinterdb_iterator_valid(it)) { + slice k, v; + uint64_t kk; + splinterdb_iterator_get_current(it, &k, &v); + if (!decode_key(k, &kk) || kk != 2 * expect || !value_id_ok(v, expect)) { + violation(w, + "warm: expected id %" PRIu64 " got key %" PRIu64 + " (len %" PRIu64 ")", + expect, + kk, + slice_length(k)); + if (kk > 2 * expect && (kk & 1) == 0) { + expect = kk / 2; /* resync after a gap */ + } + } + expect++; + w->ops++; + if ((w->ops & 1023) == 0) { + atomic_store_explicit(&w->progress, w->ops, memory_order_relaxed); + } + if (expect < hi) { + splinterdb_iterator_next(it); + } + } + rc = splinterdb_iterator_status(it); + if (rc) { + w->errors++; + fprintf(stderr, "ERROR: warm iterator status rc=%d\n", rc); + } + if (expect != hi) { + violation(w, + "warm: range [%" PRIu64 ",%" PRIu64 ") ended at %" PRIu64, + lo, + hi, + expect); + } + splinterdb_iterator_deinit(it); + atomic_store_explicit(&w->progress, w->ops, memory_order_relaxed); + return NULL; +} + +static void * +point_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE]; + char buf[256]; + splinterdb_lookup_result res; + splinterdb_lookup_result_init( + kvs, &res, SPLINTERDB_LOOKUP_VALUE, sizeof(buf), buf); + uint64_t T = (uint64_t)nworkers; + uint64_t lo = opt.n * (uint64_t)w->idx / T, + hi = opt.n * (uint64_t)(w->idx + 1) / T; + uint64_t seqpos = lo; + bool neg = w->ph->dist == D_NEG; + wait_start(); + uint64_t n = 0; + for (;;) { + int st = atomic_load_explicit(&run_state, memory_order_relaxed); + if (st == ST_STOP) { + break; + } + uint64_t id = pick_id(w, w->ph->dist, &seqpos, lo, hi); + bool q_neg = neg; + bool e_neg = neg; + if (opt.sabotage & 1) { + q_neg = true; /* sabotage: query absent key, expect present */ + } + if ((opt.sabotage & 16) && n > 1000) { + sleep(100000); /* test hook: simulate a hang for the watchdog */ + } + if (opt.sabotage & 2) { + q_neg = false; /* sabotage: query present key, expect absent */ + } + make_key(q_neg ? 2 * id + 1 : 2 * id, key); + uint64_t t0 = now_ns(); + int rc = splinterdb_lookup(kvs, slice_create(KEY_SIZE, key), &res); + uint64_t t1 = now_ns(); + n++; + atomic_store_explicit(&w->progress, n, memory_order_relaxed); + if (st != ST_MEASURE) { + continue; + } + w->ops++; + hist_add(&w->lat, t1 - t0); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: lookup id=%" PRIu64 " rc=%d\n", id, rc); + } + continue; + } + bool found = splinterdb_lookup_found(&res); + if (found) { + w->found++; + } else { + w->notfound++; + } + if (e_neg) { + if (found) { + violation(w, "negative key 2*%" PRIu64 "+1 found", id); + } + } else { + slice v; + if (!found) { + violation(w, "present id %" PRIu64 " not found", id); + } else if (splinterdb_lookup_result_value(&res, &v) != 0 + || !value_id_ok(v, id)) + { + violation(w, "present id %" PRIu64 " wrong value", id); + } + } + } + splinterdb_lookup_result_deinit(&res); + return NULL; +} + +static void * +range_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE]; + uint64_t L = w->ph->L; + uint64_t seqpos = 0; + uint64_t n = 0; + wait_start(); + for (;;) { + int st = atomic_load_explicit(&run_state, memory_order_relaxed); + if (st == ST_STOP) { + break; + } + uint64_t start = pick_id(w, w->ph->dist, &seqpos, 0, opt.n); + uint64_t expect = opt.n - start < L ? opt.n - start : L; + if (opt.sabotage & 4) { + expect++; /* sabotage: expect one extra tuple */ + } + make_key(2 * start, key); + uint64_t t0 = now_ns(), t1 = 0, tl = 0; + splinterdb_iterator *it; + int rc = splinterdb_iterator_init( + kvs, &it, greater_than_or_equal, slice_create(KEY_SIZE, key)); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: iterator_init rc=%d\n", rc); + } + continue; + } + uint64_t got = 0; + bool bad = false; + while (got < L && splinterdb_iterator_valid(it)) { + slice k, v; + uint64_t kk = 0; + splinterdb_iterator_get_current(it, &k, &v); + if (!bad + && (!decode_key(k, &kk) || kk != 2 * (start + got) + || !value_id_ok(v, start + got))) + { + bad = true; + if (st == ST_MEASURE) { + violation(w, + "range start %" PRIu64 " tuple %" PRIu64 + ": expected key %" PRIu64 " got %" PRIu64, + start, + got, + 2 * (start + got), + kk); + } + } + got++; + if (got == 1) { + t1 = now_ns(); + } + if (got == L) { + tl = now_ns(); + break; + } + splinterdb_iterator_next(it); + } + int status = splinterdb_iterator_status(it); + uint64_t t2 = now_ns(); + splinterdb_iterator_deinit(it); + uint64_t t3 = now_ns(); + n++; + atomic_store_explicit(&w->progress, n, memory_order_relaxed); + if (st != ST_MEASURE) { + continue; + } + if (status) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: iterator status rc=%d\n", status); + } + } + if (got != expect) { + violation(w, + "range start %" PRIu64 " L=%" PRIu64 ": got %" PRIu64 + " tuples, expected %" PRIu64, + start, + L, + got, + expect); + } + w->ops++; + w->tuples += got; + hist_add(&w->lat, t3 - t0); + if (got >= 1) { + hist_add(&w->startup, t1 - t0); + } + if (L > 1 && got == L) { + w->pt_ns += tl - t1; + w->pt_div += L - 1; + } + w->deinit_ns += t3 - t2; + w->deinit_n++; + } + return NULL; +} + +static void * +ow_worker(void *arg) +{ + worker_t *w = arg; + char key[KEY_SIZE], val[VALUE_SIZE]; + uint64_t seqpos = 0, n = 0; + wait_start(); + for (;;) { + int st = atomic_load_explicit(&run_state, memory_order_relaxed); + if (st == ST_STOP) { + break; + } + if (st == ST_MEASURE && !w->measure_rng_captured) { + w->measure_rng = w->rng; + w->measure_rng_captured = true; + } + uint64_t id = pick_id(w, w->ph->dist, &seqpos, 0, opt.n); + make_key(2 * id, key); + make_value(id, generation, val); + uint64_t t0 = now_ns(); + int rc = splinterdb_insert( + kvs, slice_create(KEY_SIZE, key), slice_create(VALUE_SIZE, val), NULL); + uint64_t t1 = now_ns(); + n++; + atomic_store_explicit(&w->progress, n, memory_order_relaxed); + if (rc) { + w->errors++; + if (w->errors <= 3) { + fprintf(stderr, "ERROR: overwrite id=%" PRIu64 " rc=%d\n", id, rc); + } + } + if (st != ST_MEASURE) { + continue; + } + w->ops++; + hist_add(&w->lat, t1 - t0); + } + return NULL; +} + +/* + * Validate an untimed deterministic sample from the beginning of each + * overwrite worker's measured stream. Replaying the saved RNG state keeps all + * validation work out of the measured hot path. + */ +static void +verify_overwrite_sample(const phase_t *ph, int threads) +{ + enum { SAMPLES_PER_WORKER = 4096 }; + char key[KEY_SIZE], buffer[VALUE_SIZE]; + splinterdb_lookup_result result; + splinterdb_lookup_result_init( + kvs, &result, SPLINTERDB_LOOKUP_VALUE, sizeof(buffer), buffer); + + for (int i = 0; i < threads; i++) { + worker_t *original = &workers[i]; + if (original->ops == 0) { + continue; + } + if (!original->measure_rng_captured) { + original->errors++; + fprintf( + stderr, + "ERROR: overwrite verifier has no measurement RNG for thread %d\n", + i); + continue; + } + worker_t replay = {.rng = original->measure_rng}; + uint64_t seqpos = 0; + uint64_t samples = original->ops < SAMPLES_PER_WORKER + ? original->ops + : SAMPLES_PER_WORKER; + for (uint64_t sample = 0; sample < samples; sample++) { + uint64_t id = pick_id(&replay, ph->dist, &seqpos, 0, opt.n); + make_key(2 * id, key); + int rc = splinterdb_lookup(kvs, slice_create(KEY_SIZE, key), &result); + if (rc != 0) { + original->errors++; + if (original->errors <= 3) { + fprintf(stderr, + "ERROR: overwrite verification id=%" PRIu64 " rc=%d\n", + id, + rc); + } + continue; + } + slice value; + if (!splinterdb_lookup_found(&result) + || splinterdb_lookup_result_value(&result, &value) != 0 + || !value_generation_ok(value, id, generation)) + { + violation(original, + "overwrite verification id=%" PRIu64 + " did not return generation %" PRIu64, + id, + generation); + } + } + } + splinterdb_lookup_result_deinit(&result); +} + +/* ------------------------------------------------------------------------ */ +/* Phase driver */ +/* ------------------------------------------------------------------------ */ + +static const char * +dist_name(enum dist d) +{ + switch (d) { + case D_UNIFORM: + return "uniform"; + case D_ZIPF: + return "zipf"; + case D_SEQ: + return "seq"; + case D_NEG: + return "neg"; + case D_RANDOM: + return "random"; + } + return "?"; +} + +static void +sleep_secs(double s) +{ + uint64_t end = now_ns() + (uint64_t)(s * 1e9); + for (;;) { + uint64_t now = now_ns(); + if (now >= end) { + break; + } + uint64_t rem = end - now; + struct timespec d = {.tv_sec = rem / 1000000000ull, + .tv_nsec = rem % 1000000000ull}; + nanosleep(&d, NULL); + } +} + +static void +file_sizes(uint64_t *apparent, uint64_t *alloc) +{ + struct stat st; + if (stat(opt.db, &st) == 0) { + *apparent = (uint64_t)st.st_size; + *alloc = (uint64_t)st.st_blocks * 512; + } else { + *apparent = *alloc = 0; + } +} + +static int csv_header_written; + +static void +csv_row(const phase_t *ph, + int threads, + hist_t *lat, + hist_t *startup, + uint64_t ops, + uint64_t tuples, + uint64_t pt_ns, + uint64_t pt_div, + uint64_t deinit_ns, + uint64_t deinit_n, + uint64_t viol, + uint64_t errs, + const snap_t *a, + const snap_t *b, + double insert_s, + double drain_s, + const snap_t *da, + const snap_t *db_, + uint64_t app, + uint64_t alc) +{ + double secs = (b->t_ns - a->t_ns) / 1e9; + double cpu = b->cpu_s - a->cpu_s; + uint64_t rd = b->rd - a->rd, wr = b->wr - a->wr; + double d_cpu = da ? db_->cpu_s - da->cpu_s : 0; + uint64_t d_rd = da ? db_->rd - da->rd : 0, d_wr = da ? db_->wr - da->wr : 0; + uint64_t swin = b->swpin - a->swpin + (da ? db_->swpin - da->swpin : 0); + uint64_t swout = b->swpout - a->swpout + (da ? db_->swpout - da->swpout : 0); + uint64_t majflt = b->majflt - a->majflt; + double logical = (double)ops * (KEY_SIZE + VALUE_SIZE); + double wamp = (ph->kind == PH_LOAD || ph->kind == PH_OW) && ops + ? (double)(wr + d_wr) / logical + : NAN; + double pt = pt_div ? (double)pt_ns / (double)pt_div : NAN; + double dmean = deinit_n ? (double)deinit_ns / (double)deinit_n / 1e3 : NAN; + + printf("== %-22s T=%-2d ops=%-10" PRIu64 " secs=%7.2f ops/s=%11.0f " + "lat us p50=%.2f p99=%.2f p99.9=%.2f max=%.1f mean=%.2f\n", + ph->name, + threads, + ops, + secs, + ops / secs, + hist_pct_us(lat, 0.5), + hist_pct_us(lat, 0.99), + hist_pct_us(lat, 0.999), + (double)lat->max / 1e3, + hist_mean_us(lat)); + printf(" cpu=%.2fs ops/cpu-s=%.0f rd=%.1f B/op wr=%.1f B/op " + "violations=%" PRIu64 " errors=%" PRIu64 " swapin=%" PRIu64 + " swapout=%" PRIu64 " majflt=%" PRIu64 "\n", + cpu, + ops / cpu, + ops ? (double)rd / ops : 0, + ops ? (double)wr / ops : 0, + viol, + errs, + swin, + swout, + majflt); + if (ph->kind == PH_RANGE) { + printf(" startup us p50=%.2f p99=%.2f mean=%.2f per-tuple=%.1f ns " + "deinit mean=%.2f us tuples/s=%.0f\n", + hist_pct_us(startup, 0.5), + hist_pct_us(startup, 0.99), + hist_mean_us(startup), + pt, + dmean, + tuples / secs); + } + if (da) { + printf( + " insert/run=%.2fs drain(close)=%.2fs total=%.2fs drain_cpu=%.2fs " + "drain_rd=%" PRIu64 " drain_wr=%" PRIu64 " wamp=%.3f " + "db_apparent=%" PRIu64 " db_alloc=%" PRIu64 "\n", + insert_s, + drain_s, + insert_s + drain_s, + d_cpu, + d_rd, + d_wr, + wamp, + app, + alc); + } + fflush(stdout); + + if (!opt.csv) { + return; + } + FILE *f = fopen(opt.csv, "a"); + if (!f) { + perror(opt.csv); + return; + } + if (!csv_header_written) { + fseek(f, 0, SEEK_END); + if (ftell(f) == 0) { + fprintf(f, + "version,size,order,log,phase,threads,rep,n,ops,secs,ops_s," + "p50_us,p99_us,p999_us,max_us,mean_us,cpu_s,ops_per_cpu_s," + "rd_bytes,wr_bytes,rd_B_op,wr_B_op," + "startup_p50_us,startup_p99_us,startup_mean_us,per_tuple_ns," + "deinit_mean_us,tuples,tuples_s," + "insert_s,drain_s,total_s,drain_cpu_s,drain_rd,drain_wr,wamp," + "db_apparent,db_alloc,open_s,violations,errors,swapin,swapout," + "majflt\n"); + } + csv_header_written = 1; + } + fprintf(f, + "%s,%s,%s,%d,%s,%d,%d,%" PRIu64 ",%" PRIu64 ",%.3f,%.1f," + "%.3f,%.3f,%.3f,%.3f,%.3f,%.3f,%.1f," + "%" PRIu64 ",%" PRIu64 ",%.2f,%.2f," + "%.3f,%.3f,%.3f,%.2f," + "%.3f,%" PRIu64 ",%.1f," + "%.3f,%.3f,%.3f,%.3f,%" PRIu64 ",%" PRIu64 ",%.4f," + "%" PRIu64 ",%" PRIu64 ",%.3f,%" PRIu64 ",%" PRIu64 ",%" PRIu64 + ",%" PRIu64 ",%" PRIu64 "\n", + opt.vtag, + opt.stag, + opt.otag, + opt.use_log, + ph->name, + threads, + opt.rep, + opt.n, + ops, + secs, + ops / secs, + hist_pct_us(lat, 0.5), + hist_pct_us(lat, 0.99), + hist_pct_us(lat, 0.999), + (double)lat->max / 1e3, + hist_mean_us(lat), + cpu, + ops / cpu, + rd, + wr, + ops ? (double)rd / ops : 0, + ops ? (double)wr / ops : 0, + hist_pct_us(startup, 0.5), + hist_pct_us(startup, 0.99), + hist_mean_us(startup), + pt, + dmean, + tuples, + tuples / secs, + insert_s, + drain_s, + insert_s + drain_s, + d_cpu, + d_rd, + d_wr, + wamp, + app, + alc, + open_secs, + viol, + errs, + swin, + swout, + majflt); + fclose(f); +} + +static void +run_phase(const phase_t *ph, bool last) +{ + int T = ph->threads > 0 ? ph->threads : opt.threads; + nworkers = T; + workers = calloc((size_t)T, sizeof(worker_t)); + if (!workers) { + fprintf(stderr, "FATAL: out of memory\n"); + exit(3); + } + void *(*fn)(void *) = NULL; + switch (ph->kind) { + case PH_LOAD: + fn = load_worker; + break; + case PH_WARM: + fn = warm_worker; + break; + case PH_POINT: + fn = point_worker; + break; + case PH_RANGE: + fn = range_worker; + break; + case PH_OW: + fn = ow_worker; + break; + } + bool fixed_count = ph->kind == PH_LOAD || ph->kind == PH_WARM; + atomic_store(&run_state, ST_WAIT); + atomic_store(&seq_counter, 0); + for (int i = 0; i < T; i++) { + workers[i].idx = i; + workers[i].ph = ph; + workers[i].rng = mix(opt.seed * 1000003 + (uint64_t)i * 7919 + 17 + + (uint64_t)ph->kind * 131 + (uint64_t)ph->dist); + if (pthread_create(&workers[i].th, NULL, fn, &workers[i])) { + fprintf(stderr, "FATAL: pthread_create\n"); + exit(3); + } + } + fprintf(stderr, + "phase %s: T=%d dist=%s %s\n", + ph->name, + T, + dist_name(ph->dist), + fixed_count ? "(fixed count)" : "(fixed duration)"); + wd_what = ph->name; + atomic_store(&wd_mode, WD_OPS); + snap_t a, b; + if (fixed_count) { + take_snap(&a); + atomic_store_explicit(&run_state, ST_MEASURE, memory_order_release); + for (int i = 0; i < T; i++) { + pthread_join(workers[i].th, NULL); + } + take_snap(&b); + } else { + atomic_store_explicit(&run_state, ST_WARMUP, memory_order_release); + sleep_secs(opt.warmup); + take_snap(&a); + atomic_store_explicit(&run_state, ST_MEASURE, memory_order_release); + sleep_secs(opt.secs); + atomic_store_explicit(&run_state, ST_STOP, memory_order_release); + take_snap(&b); + for (int i = 0; i < T; i++) { + pthread_join(workers[i].th, NULL); + } + } + atomic_store(&wd_mode, WD_IDLE); + + if (ph->kind == PH_OW) { + verify_overwrite_sample(ph, T); + } + + hist_t *lat = calloc(1, sizeof(hist_t)), + *startup = calloc(1, sizeof(hist_t)); + uint64_t ops = 0, tuples = 0, pt_ns = 0, pt_div = 0, dns = 0, dn = 0, + viol = 0, errs = 0, found = 0, notfound = 0; + for (int i = 0; i < T; i++) { + worker_t *w = &workers[i]; + hist_merge(lat, &w->lat); + hist_merge(startup, &w->startup); + ops += w->ops; + tuples += w->tuples; + pt_ns += w->pt_ns; + pt_div += w->pt_div; + dns += w->deinit_ns; + dn += w->deinit_n; + viol += w->violations; + errs += w->errors; + found += w->found; + notfound += w->notfound; + } + if (ph->kind == PH_WARM) { + tuples = ops; + if (ops != opt.n && !(opt.sabotage & 8)) { + fprintf(stderr, + "VIOLATION: warm scanned %" PRIu64 " tuples, expected %" PRIu64 + "\n", + ops, + opt.n); + viol++; + } + } + if (ph->kind == PH_POINT) { + fprintf(stderr, + " found=%" PRIu64 " notfound=%" PRIu64 "\n", + found, + notfound); + } + + double insert_s = (b.t_ns - a.t_ns) / 1e9, drain_s = 0; + snap_t da, dbs; + uint64_t app = 0, alc = 0; + bool closed = false; + if (ph->kind == PH_LOAD || last) { + take_snap(&da); + drain_s = db_close(); + take_snap(&dbs); + file_sizes(&app, &alc); + closed = true; + } + csv_row(ph, + T, + lat, + startup, + ops, + tuples, + pt_ns, + pt_div, + dns, + dn, + viol, + errs, + &a, + &b, + insert_s, + drain_s, + closed ? &da : NULL, + closed ? &dbs : NULL, + app, + alc); + if (viol) { + printf(" FINDING: %" PRIu64 + " correctness violations in %s (first: %s)\n", + viol, + ph->name, + workers[0].first_violation[0] ? workers[0].first_violation + : "see stderr"); + } + free(lat); + free(startup); + free(workers); + workers = NULL; + nworkers = 0; + if (closed && !last) { + db_open(false); + } +} + +/* ------------------------------------------------------------------------ */ +/* CLI */ +/* ------------------------------------------------------------------------ */ + +static void +usage(void) +{ + fprintf( + stderr, + "usage: kvbench --db PATH (--create|--open) --n N --cache-mib M " + "--disk-gib G --log 0|1\n" + " [--threads 6] [--mt-bg 1] [--normal-bg 4] [--memtable-mib 24]\n" + " [--secs 20] [--warmup 3] [--csv FILE] [--hang-dir DIR]\n" + " [--vtag S] [--stag S] [--otag S] [--rep R] [--seed S]\n" + " [--sabotage MASK] (1: point present->query absent; 2: point " + "neg->query present;\n" + " 4: range expects +1; 8: warm expects +1; 16: " + "point threads hang)\n" + " [--hang-secs 120] (no-progress timeout before gdb dump + exit 99\n" + " phase... load:random|load:seq warm point:uniform|zipf|seq|neg\n" + " range:uniform|zipf:L ow:uniform|zipf (suffix @T = " + "threads)\n"); + exit(2); +} + +static bool +parse_phase(const char *s, phase_t *ph) +{ + memset(ph, 0, sizeof(*ph)); + snprintf(ph->name, sizeof(ph->name), "%s", s); + char buf[64]; + snprintf(buf, sizeof(buf), "%s", s); + char *at = strchr(buf, '@'); + if (at) { + *at = 0; + ph->threads = atoi(at + 1); + } + char *a = strtok(buf, ":"), *b = strtok(NULL, ":"), *c = strtok(NULL, ":"); + if (!a) { + return false; + } + enum dist d = D_UNIFORM; + if (b) { + if (!strcmp(b, "uniform")) + d = D_UNIFORM; + else if (!strcmp(b, "zipf")) + d = D_ZIPF; + else if (!strcmp(b, "seq")) + d = D_SEQ; + else if (!strcmp(b, "neg")) + d = D_NEG; + else if (!strcmp(b, "random")) + d = D_RANDOM; + else + return false; + } + ph->dist = d; + if (!strcmp(a, "load")) { + ph->kind = PH_LOAD; + return b && (d == D_RANDOM || d == D_SEQ); + } else if (!strcmp(a, "warm")) { + ph->kind = PH_WARM; + return true; + } else if (!strcmp(a, "point")) { + ph->kind = PH_POINT; + return b != NULL; + } else if (!strcmp(a, "range")) { + ph->kind = PH_RANGE; + if (!b || !c || (d != D_UNIFORM && d != D_ZIPF)) { + return false; + } + ph->L = strtoull(c, NULL, 10); + return ph->L >= 1; + } else if (!strcmp(a, "ow")) { + ph->kind = PH_OW; + return b && (d == D_UNIFORM || d == D_ZIPF); + } + return false; +} + +int +main(int argc, char **argv) +{ + opt = (options){.threads = 6, + .mt_bg = 1, + .normal_bg = 4, + .memtable_mib = 24, + .secs = 20, + .warmup = 3, + .vtag = "?", + .stag = "?", + .otag = "?", + .rep = 1, + .seed = 42, + .hang_secs = 120, + .use_log = -1}; + int mode = -1; +#ifdef KVBENCH_SPACE_MODE + /* + * Space-check build only: open with use_stats, print insertion stats + * (which include allocator "Currently Allocated" extents, recomputed from + * the persisted allocator map at open), close. No phases, no writes. + */ + bool space_mode = false; +#endif + for (int i = 1; i < argc; i++) { + const char *s = argv[i]; +#define ARG(name) (!strcmp(s, name) && i + 1 < argc) + if (!strcmp(s, "--create")) + mode = 1; + else if (!strcmp(s, "--open")) + mode = 0; + else if (ARG("--db")) + opt.db = argv[++i]; + else if (ARG("--csv")) + opt.csv = argv[++i]; + else if (ARG("--hang-dir")) + opt.hang_dir = argv[++i]; + else if (ARG("--vtag")) + opt.vtag = argv[++i]; + else if (ARG("--stag")) + opt.stag = argv[++i]; + else if (ARG("--otag")) + opt.otag = argv[++i]; + else if (ARG("--rep")) + opt.rep = atoi(argv[++i]); + else if (ARG("--n")) + opt.n = strtoull(argv[++i], NULL, 0); + else if (ARG("--cache-mib")) + opt.cache_mib = strtoull(argv[++i], NULL, 0); + else if (ARG("--disk-gib")) + opt.disk_gib = strtoull(argv[++i], NULL, 0); + else if (ARG("--memtable-mib")) + opt.memtable_mib = strtoull(argv[++i], NULL, 0); + else if (ARG("--log")) + opt.use_log = atoi(argv[++i]); + else if (ARG("--threads")) + opt.threads = atoi(argv[++i]); + else if (ARG("--mt-bg")) + opt.mt_bg = atoi(argv[++i]); + else if (ARG("--normal-bg")) + opt.normal_bg = atoi(argv[++i]); + else if (ARG("--secs")) + opt.secs = atof(argv[++i]); + else if (ARG("--warmup")) + opt.warmup = atof(argv[++i]); + else if (ARG("--sabotage")) + opt.sabotage = strtoull(argv[++i], NULL, 0); + else if (ARG("--hang-secs")) + opt.hang_secs = strtoull(argv[++i], NULL, 0); + else if (ARG("--seed")) + opt.seed = strtoull(argv[++i], NULL, 0); +#ifdef KVBENCH_SPACE_MODE + else if (!strcmp(s, "space")) + space_mode = true; +#endif + else if (s[0] == '-') + usage(); + else { + if (opt.nphases >= MAX_PHASES + || !parse_phase(s, &opt.phases[opt.nphases])) + { + fprintf(stderr, "bad phase '%s'\n", s); + usage(); + } + opt.nphases++; + } +#undef ARG + } +#ifdef KVBENCH_SPACE_MODE + if (space_mode && opt.db && opt.cache_mib && opt.disk_gib + && opt.use_log >= 0) + { + default_data_config_init(&data_cfg); + platform_set_log_streams(stdout, stderr); + db_open(false); + splinterdb_stats_print_insertion(kvs); + fflush(stdout); + db_close(); + uint64_t app, alc; + file_sizes(&app, &alc); + printf("space: db_apparent=%" PRIu64 " db_alloc=%" PRIu64 "\n", app, alc); + return 0; + } +#endif + if (!opt.db || mode < 0 || !opt.n || !opt.cache_mib || !opt.disk_gib + || opt.use_log < 0 || !opt.nphases || opt.threads < 1 + || opt.threads > MAX_WORKERS) + { + usage(); + } + for (int i = 0; i < opt.nphases; i++) { + if (opt.phases[i].threads > MAX_WORKERS) { + usage(); + } + } + prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY, 0, 0, 0); + setvbuf(stdout, NULL, _IOLBF, 0); + + default_data_config_init(&data_cfg); + perm_init(&load_perm, opt.n, opt.seed * 2 + 1); + perm_init(&zipf_perm, opt.n, opt.seed * 2 + 2); + zipf_init(&zipf, opt.n, 0.99); + generation = (uint64_t)time(NULL); + + printf("kvbench version=%s splinter=%s db=%s mode=%s n=%" PRIu64 + " cache_mib=%" PRIu64 " disk_gib=%" PRIu64 + " log=%d threads=%d mt_bg=%d normal_bg=%d memtable_mib=%" PRIu64 + " secs=%.1f warmup=%.1f rep=%d sabotage=%" PRIu64 "\n", + opt.vtag, + splinterdb_get_version(), + opt.db, + mode ? "create" : "open", + opt.n, + opt.cache_mib, + opt.disk_gib, + opt.use_log, + opt.threads, + opt.mt_bg, + opt.normal_bg, + opt.memtable_mib, + opt.secs, + opt.warmup, + opt.rep, + opt.sabotage); + + pthread_t wd; + pthread_create(&wd, NULL, watchdog_main, NULL); + + db_open(mode == 1); + for (int i = 0; i < opt.nphases; i++) { + run_phase(&opt.phases[i], i == opt.nphases - 1); + } + if (kvs) { + db_close(); + } + return 0; +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/status.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/status.json new file mode 100644 index 000000000..7dfea292a --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/status.json @@ -0,0 +1,13 @@ +{ + "complete": true, + "runs": { + "fixtures": 4, + "resident": 6, + "disk": 6, + "feature": 12, + "repeat-resident-disk-point8": 4 + }, + "errors": 0, + "violations": 0, + "verification": "All processes succeeded; all manifests verified; exact warm/load counts and output checks validated." +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/summary.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/summary.json new file mode 100644 index 000000000..93ccdaf7f --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/summary.json @@ -0,0 +1,3493 @@ +{ + "resident": [ + { + "profile": "main", + "phase": "point:uniform@1", + "throughput": { + "mean": 690741.25, + "min": 688766.2, + "max": 692716.3, + "samples": 2, + "values": [ + 692716.3, + 688766.2 + ] + }, + "baseline": { + "mean": 690741.25, + "min": 688766.2, + "max": 692716.3, + "samples": 2, + "values": [ + 692716.3, + 688766.2 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 1.4479478476724446, + "min": 1.4435346967999723, + "max": 1.4523609985449168, + "samples": 2, + "values": [ + 1.4435346967999723, + 1.4523609985449168 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 1.856, + "min": 1.84, + "max": 1.872, + "samples": 2, + "values": [ + 1.872, + 1.84 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "filtered-ranges", + "phase": "point:uniform@1", + "throughput": { + "mean": 673238.35, + "min": 654932.6, + "max": 691544.1, + "samples": 2, + "values": [ + 654932.6, + 691544.1 + ] + }, + "baseline": { + "mean": 690741.25, + "min": 688766.2, + "max": 692716.3, + "samples": 2, + "values": [ + 692716.3, + 688766.2 + ] + }, + "throughput_delta_pct": -2.5339300353062755, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 1.485938548287411, + "min": 1.4454474190259343, + "max": 1.5264296775488875, + "samples": 2, + "values": [ + 1.5264296775488875, + 1.4454474190259343 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 2.4000000000000004, + "min": 2.208, + "max": 2.592, + "samples": 2, + "values": [ + 2.592, + 2.208 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "disk-format", + "phase": "point:uniform@1", + "throughput": { + "mean": 682935.5, + "min": 667909.4, + "max": 697961.6, + "samples": 2, + "values": [ + 667909.4, + 697961.6 + ] + }, + "baseline": { + "mean": 690741.25, + "min": 688766.2, + "max": 692716.3, + "samples": 2, + "values": [ + 692716.3, + 688766.2 + ] + }, + "throughput_delta_pct": -1.1300541266356934, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 1.4645481199256678, + "min": 1.4326850254507542, + "max": 1.4964112144005812, + "samples": 2, + "values": [ + 1.4964112144005812, + 1.4326850254507542 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 2.168, + "min": 1.872, + "max": 2.464, + "samples": 2, + "values": [ + 2.464, + 1.872 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "main", + "phase": "point:uniform@8", + "throughput": { + "mean": 3755201.6, + "min": 3747424.5, + "max": 3762978.7, + "samples": 2, + "values": [ + 3747424.5, + 3762978.7 + ] + }, + "baseline": { + "mean": 3755201.6, + "min": 3747424.5, + "max": 3762978.7, + "samples": 2, + "values": [ + 3747424.5, + 3762978.7 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 2.128243623334213, + "min": 2.1248091086537566, + "max": 2.131678138014669, + "samples": 2, + "values": [ + 2.131678138014669, + 2.1248091086537566 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 4.112, + "min": 3.936, + "max": 4.288, + "samples": 2, + "values": [ + 4.288, + 3.936 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "filtered-ranges", + "phase": "point:uniform@8", + "throughput": { + "mean": 3781250.9, + "min": 3714776.4, + "max": 3847725.4, + "samples": 2, + "values": [ + 3714776.4, + 3847725.4 + ] + }, + "baseline": { + "mean": 3755201.6, + "min": 3747424.5, + "max": 3762978.7, + "samples": 2, + "values": [ + 3747424.5, + 3762978.7 + ] + }, + "throughput_delta_pct": 0.6936857930610163, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 2.11470986902777, + "min": 2.0783627005720873, + "max": 2.1510570374834534, + "samples": 2, + "values": [ + 2.1510570374834534, + 2.0783627005720873 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 4.08, + "min": 3.872, + "max": 4.288, + "samples": 2, + "values": [ + 4.288, + 3.872 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "disk-format", + "phase": "point:uniform@8", + "throughput": { + "mean": 3590316.8499999996, + "min": 3435058.8, + "max": 3745574.9, + "samples": 2, + "values": [ + 3435058.8, + 3745574.9 + ] + }, + "baseline": { + "mean": 3755201.6, + "min": 3747424.5, + "max": 3762978.7, + "samples": 2, + "values": [ + 3747424.5, + 3762978.7 + ] + }, + "throughput_delta_pct": -4.39083616709155, + "ranges_overlap": false, + "cpu_us_per_op": { + "mean": 2.230712238837726, + "min": 2.133745176631319, + "max": 2.3276793010441335, + "samples": 2, + "values": [ + 2.3276793010441335, + 2.133745176631319 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 4.144, + "min": 4.0, + "max": 4.288, + "samples": 2, + "values": [ + 4.288, + 4.0 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "main", + "phase": "range:uniform:1@8", + "throughput": { + "mean": 369257.55, + "min": 363033.5, + "max": 375481.6, + "samples": 2, + "values": [ + 363033.5, + 375481.6 + ] + }, + "baseline": { + "mean": 369257.55, + "min": 363033.5, + "max": 375481.6, + "samples": 2, + "values": [ + 363033.5, + 375481.6 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 21.662411003124024, + "min": 21.300889925594856, + "max": 22.023932080653193, + "samples": 2, + "values": [ + 22.023932080653193, + 21.300889925594856 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 39.424, + "min": 38.4, + "max": 40.448, + "samples": 2, + "values": [ + 40.448, + 38.4 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "filtered-ranges", + "phase": "range:uniform:1@8", + "throughput": { + "mean": 368611.25, + "min": 364020.2, + "max": 373202.3, + "samples": 2, + "values": [ + 364020.2, + 373202.3 + ] + }, + "baseline": { + "mean": 369257.55, + "min": 363033.5, + "max": 375481.6, + "samples": 2, + "values": [ + 363033.5, + 375481.6 + ] + }, + "throughput_delta_pct": -0.17502688841433134, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 21.70060048637415, + "min": 21.43170300467191, + "max": 21.969497968076393, + "samples": 2, + "values": [ + 21.969497968076393, + 21.43170300467191 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 38.912, + "min": 38.4, + "max": 39.424, + "samples": 2, + "values": [ + 39.424, + 38.4 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "disk-format", + "phase": "range:uniform:1@8", + "throughput": { + "mean": 372129.55, + "min": 367889.5, + "max": 376369.6, + "samples": 2, + "values": [ + 367889.5, + 376369.6 + ] + }, + "baseline": { + "mean": 369257.55, + "min": 363033.5, + "max": 375481.6, + "samples": 2, + "values": [ + 363033.5, + 375481.6 + ] + }, + "throughput_delta_pct": 0.7777769202010898, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 21.49045164632014, + "min": 21.244727342893, + "max": 21.73617594974728, + "samples": 2, + "values": [ + 21.73617594974728, + 21.244727342893 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 38.912, + "min": 38.4, + "max": 39.424, + "samples": 2, + "values": [ + 39.424, + 38.4 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "main", + "phase": "range:uniform:10000@8", + "throughput": { + "mean": 2913.0, + "min": 2894.8, + "max": 2931.2, + "samples": 2, + "values": [ + 2894.8, + 2931.2 + ] + }, + "baseline": { + "mean": 2913.0, + "min": 2894.8, + "max": 2931.2, + "samples": 2, + "values": [ + 2894.8, + 2931.2 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 2745.5916926602786, + "min": 2728.3697923329496, + "max": 2762.8135929876075, + "samples": 2, + "values": [ + 2762.8135929876075, + 2728.3697923329496 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.175, + "min": 0.17, + "max": 0.18, + "samples": 2, + "values": [ + 0.18, + 0.17 + ] + }, + "p99_us": { + "mean": 4390.912, + "min": 4259.84, + "max": 4521.984, + "samples": 2, + "values": [ + 4521.984, + 4259.84 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "filtered-ranges", + "phase": "range:uniform:10000@8", + "throughput": { + "mean": 2891.45, + "min": 2834.4, + "max": 2948.5, + "samples": 2, + "values": [ + 2834.4, + 2948.5 + ] + }, + "baseline": { + "mean": 2913.0, + "min": 2894.8, + "max": 2931.2, + "samples": 2, + "values": [ + 2894.8, + 2931.2 + ] + }, + "throughput_delta_pct": -0.7397871610024143, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 2766.3311123523054, + "min": 2712.323540633346, + "max": 2820.3386840712647, + "samples": 2, + "values": [ + 2820.3386840712647, + 2712.323540633346 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.085, + "min": 0.0, + "max": 0.17, + "samples": 2, + "values": [ + 0.0, + 0.17 + ] + }, + "p99_us": { + "mean": 4390.912, + "min": 4259.84, + "max": 4521.984, + "samples": 2, + "values": [ + 4521.984, + 4259.84 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "disk-format", + "phase": "range:uniform:10000@8", + "throughput": { + "mean": 2916.9, + "min": 2912.4, + "max": 2921.4, + "samples": 2, + "values": [ + 2912.4, + 2921.4 + ] + }, + "baseline": { + "mean": 2913.0, + "min": 2894.8, + "max": 2931.2, + "samples": 2, + "values": [ + 2894.8, + 2931.2 + ] + }, + "throughput_delta_pct": 0.13388259526261326, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 2741.693651991048, + "min": 2737.16412801643, + "max": 2746.223175965665, + "samples": 2, + "values": [ + 2746.223175965665, + 2737.16412801643 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.18, + "min": 0.18, + "max": 0.18, + "samples": 2, + "values": [ + 0.18, + 0.18 + ] + }, + "p99_us": { + "mean": 4259.84, + "min": 4259.84, + "max": 4259.84, + "samples": 2, + "values": [ + 4259.84, + 4259.84 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + } + ], + "disk": [ + { + "profile": "main", + "phase": "point:uniform@8", + "throughput": { + "mean": 142173.84999999998, + "min": 140322.8, + "max": 144024.9, + "samples": 2, + "values": [ + 144024.9, + 140322.8 + ] + }, + "baseline": { + "mean": 142173.84999999998, + "min": 140322.8, + "max": 144024.9, + "samples": 2, + "values": [ + 144024.9, + 140322.8 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 14.844508636579043, + "min": 14.510563551408358, + "max": 15.178453721749726, + "samples": 2, + "values": [ + 14.510563551408358, + 15.178453721749726 + ] + }, + "read_B_per_op": { + "mean": 3229.0, + "min": 3228.74, + "max": 3229.26, + "samples": 2, + "values": [ + 3228.74, + 3229.26 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 100.352, + "min": 99.328, + "max": 101.376, + "samples": 2, + "values": [ + 101.376, + 99.328 + ] + }, + "physical_read_bytes": 5509255168, + "errors": 0, + "violations": 0 + }, + { + "profile": "filtered-ranges", + "phase": "point:uniform@8", + "throughput": { + "mean": 141417.34999999998, + "min": 140069.9, + "max": 142764.8, + "samples": 2, + "values": [ + 142764.8, + 140069.9 + ] + }, + "baseline": { + "mean": 142173.84999999998, + "min": 140322.8, + "max": 144024.9, + "samples": 2, + "values": [ + 144024.9, + 140322.8 + ] + }, + "throughput_delta_pct": -0.5320950371675237, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 14.628175993716908, + "min": 14.465973788007686, + "max": 14.79037819942613, + "samples": 2, + "values": [ + 14.79037819942613, + 14.465973788007686 + ] + }, + "read_B_per_op": { + "mean": 3228.4300000000003, + "min": 3228.34, + "max": 3228.52, + "samples": 2, + "values": [ + 3228.34, + 3228.52 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 102.4, + "min": 101.376, + "max": 103.424, + "samples": 2, + "values": [ + 101.376, + 103.424 + ] + }, + "physical_read_bytes": 5478944768, + "errors": 0, + "violations": 0 + }, + { + "profile": "disk-format", + "phase": "point:uniform@8", + "throughput": { + "mean": 149095.2, + "min": 141560.3, + "max": 156630.1, + "samples": 2, + "values": [ + 156630.1, + 141560.3 + ] + }, + "baseline": { + "mean": 142173.84999999998, + "min": 140322.8, + "max": 144024.9, + "samples": 2, + "values": [ + 144024.9, + 140322.8 + ] + }, + "throughput_delta_pct": 4.8682299874414525, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 13.329262927304237, + "min": 11.557544591032954, + "max": 15.10098126357552, + "samples": 2, + "values": [ + 11.557544591032954, + 15.10098126357552 + ] + }, + "read_B_per_op": { + "mean": 3229.07, + "min": 3229.01, + "max": 3229.13, + "samples": 2, + "values": [ + 3229.01, + 3229.13 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 100.352, + "min": 99.328, + "max": 101.376, + "samples": 2, + "values": [ + 99.328, + 101.376 + ] + }, + "physical_read_bytes": 5777555456, + "errors": 0, + "violations": 0 + }, + { + "profile": "main", + "phase": "range:uniform:1@8", + "throughput": { + "mean": 61259.05, + "min": 60885.8, + "max": 61632.3, + "samples": 2, + "values": [ + 61632.3, + 60885.8 + ] + }, + "baseline": { + "mean": 61259.05, + "min": 60885.8, + "max": 61632.3, + "samples": 2, + "values": [ + 61632.3, + 60885.8 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 138.9042283197839, + "min": 138.3216336949585, + "max": 139.4868229446093, + "samples": 2, + "values": [ + 138.3216336949585, + 139.4868229446093 + ] + }, + "read_B_per_op": { + "mean": 19820.18, + "min": 19816.96, + "max": 19823.4, + "samples": 2, + "values": [ + 19823.4, + 19816.96 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 263.168, + "min": 251.904, + "max": 274.432, + "samples": 2, + "values": [ + 251.904, + 274.432 + ] + }, + "physical_read_bytes": 14570700800, + "errors": 0, + "violations": 0 + }, + { + "profile": "filtered-ranges", + "phase": "range:uniform:1@8", + "throughput": { + "mean": 61078.05, + "min": 60071.0, + "max": 62085.1, + "samples": 2, + "values": [ + 62085.1, + 60071.0 + ] + }, + "baseline": { + "mean": 61259.05, + "min": 60885.8, + "max": 61632.3, + "samples": 2, + "values": [ + 61632.3, + 60885.8 + ] + }, + "throughput_delta_pct": -0.29546654739177614, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 139.26327008593103, + "min": 136.8869466716415, + "max": 141.63959350022057, + "samples": 2, + "values": [ + 136.8869466716415, + 141.63959350022057 + ] + }, + "read_B_per_op": { + "mean": 19828.085, + "min": 19827.07, + "max": 19829.1, + "samples": 2, + "values": [ + 19827.07, + 19829.1 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 261.12, + "min": 256.0, + "max": 266.24, + "samples": 2, + "values": [ + 256.0, + 266.24 + ] + }, + "physical_read_bytes": 14533656576, + "errors": 0, + "violations": 0 + }, + { + "profile": "disk-format", + "phase": "range:uniform:1@8", + "throughput": { + "mean": 62125.35, + "min": 61649.2, + "max": 62601.5, + "samples": 2, + "values": [ + 61649.2, + 62601.5 + ] + }, + "baseline": { + "mean": 61259.05, + "min": 60885.8, + "max": 61632.3, + "samples": 2, + "values": [ + 61632.3, + 60885.8 + ] + }, + "throughput_delta_pct": 1.4141583978203975, + "ranges_overlap": false, + "cpu_us_per_op": { + "mean": 136.91109827010092, + "min": 136.16820762929578, + "max": 137.65398891090607, + "samples": 2, + "values": [ + 137.65398891090607, + 136.16820762929578 + ] + }, + "read_B_per_op": { + "mean": 19822.555, + "min": 19802.72, + "max": 19842.39, + "samples": 2, + "values": [ + 19802.72, + 19842.39 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 251.904, + "min": 243.712, + "max": 260.096, + "samples": 2, + "values": [ + 260.096, + 243.712 + ] + }, + "physical_read_bytes": 14778699776, + "errors": 0, + "violations": 0 + }, + { + "profile": "main", + "phase": "range:uniform:10000@8", + "throughput": { + "mean": 742.45, + "min": 741.3, + "max": 743.6, + "samples": 2, + "values": [ + 741.3, + 743.6 + ] + }, + "baseline": { + "mean": 742.45, + "min": 741.3, + "max": 743.6, + "samples": 2, + "values": [ + 741.3, + 743.6 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 5409.137946254381, + "min": 5379.202151501569, + "max": 5439.073741007194, + "samples": 2, + "values": [ + 5439.073741007194, + 5379.202151501569 + ] + }, + "read_B_per_op": { + "mean": 2356914.9450000003, + "min": 2349744.48, + "max": 2364085.41, + "samples": 2, + "values": [ + 2364085.41, + 2349744.48 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 23592.96, + "min": 23330.816, + "max": 23855.104, + "samples": 2, + "values": [ + 23330.816, + 23855.104 + ] + }, + "physical_read_bytes": 21000011776, + "errors": 0, + "violations": 0 + }, + { + "profile": "filtered-ranges", + "phase": "range:uniform:10000@8", + "throughput": { + "mean": 731.35, + "min": 719.1, + "max": 743.6, + "samples": 2, + "values": [ + 743.6, + 719.1 + ] + }, + "baseline": { + "mean": 742.45, + "min": 741.3, + "max": 743.6, + "samples": 2, + "values": [ + 741.3, + 743.6 + ] + }, + "throughput_delta_pct": -1.49505017172874, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 5407.2744322729395, + "min": 5314.432989690722, + "max": 5500.115874855156, + "samples": 2, + "values": [ + 5314.432989690722, + 5500.115874855156 + ] + }, + "read_B_per_op": { + "mean": 2371315.71, + "min": 2363148.74, + "max": 2379482.68, + "samples": 2, + "values": [ + 2363148.74, + 2379482.68 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 25165.824, + "min": 23855.104, + "max": 26476.544, + "samples": 2, + "values": [ + 23855.104, + 26476.544 + ] + }, + "physical_read_bytes": 20811837440, + "errors": 0, + "violations": 0 + }, + { + "profile": "disk-format", + "phase": "range:uniform:10000@8", + "throughput": { + "mean": 736.6, + "min": 728.6, + "max": 744.6, + "samples": 2, + "values": [ + 744.6, + 728.6 + ] + }, + "baseline": { + "mean": 742.45, + "min": 741.3, + "max": 743.6, + "samples": 2, + "values": [ + 741.3, + 743.6 + ] + }, + "throughput_delta_pct": -0.7879318472624464, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 5358.529312029592, + "min": 5271.038495971352, + "max": 5446.020128087832, + "samples": 2, + "values": [ + 5271.038495971352, + 5446.020128087832 + ] + }, + "read_B_per_op": { + "mean": 2366069.675, + "min": 2363294.57, + "max": 2368844.78, + "samples": 2, + "values": [ + 2368844.78, + 2363294.57 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 23855.104, + "min": 23855.104, + "max": 23855.104, + "samples": 2, + "values": [ + 23855.104, + 23855.104 + ] + }, + "physical_read_bytes": 20916322304, + "errors": 0, + "violations": 0 + } + ], + "repeat-resident-disk-point8": [ + { + "profile": "disk-format", + "phase": "point:uniform@8", + "throughput": { + "mean": 3755396.15, + "min": 3651128.5, + "max": 3859663.8, + "samples": 2, + "values": [ + 3859663.8, + 3651128.5 + ] + }, + "baseline": { + "mean": 4070674.25, + "min": 4026197.4, + "max": 4115151.1, + "samples": 2, + "values": [ + 4026197.4, + 4115151.1 + ] + }, + "throughput_delta_pct": -7.74510758260748, + "ranges_overlap": false, + "cpu_us_per_op": { + "mean": 2.1313329717191527, + "min": 2.072114317797465, + "max": 2.19055162564084, + "samples": 2, + "values": [ + 2.072114317797465, + 2.19055162564084 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 4.24, + "min": 4.064, + "max": 4.416, + "samples": 2, + "values": [ + 4.064, + 4.416 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + }, + { + "profile": "main", + "phase": "point:uniform@8", + "throughput": { + "mean": 4070674.25, + "min": 4026197.4, + "max": 4115151.1, + "samples": 2, + "values": [ + 4026197.4, + 4115151.1 + ] + }, + "baseline": { + "mean": 4070674.25, + "min": 4026197.4, + "max": 4115151.1, + "samples": 2, + "values": [ + 4026197.4, + 4115151.1 + ] + }, + "throughput_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_op": { + "mean": 1.965077537239115, + "min": 1.943553471129346, + "max": 1.9866016033488838, + "samples": 2, + "values": [ + 1.9866016033488838, + 1.943553471129346 + ] + }, + "read_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_op": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "p99_us": { + "mean": 3.488, + "min": 3.296, + "max": 3.68, + "samples": 2, + "values": [ + 3.68, + 3.296 + ] + }, + "physical_read_bytes": 0, + "errors": 0, + "violations": 0 + } + ], + "feature": [ + { + "collisions": 0, + "limit": 1, + "mode": "application", + "latency_ns": { + "mean": 17532.05, + "min": 17470.1, + "max": 17594.0, + "samples": 2, + "values": [ + 17594.0, + 17470.1 + ] + }, + "baseline": { + "mean": 17532.05, + "min": 17470.1, + "max": 17594.0, + "samples": 2, + "values": [ + 17594.0, + 17470.1 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 16.254, + "min": 16.2358, + "max": 16.2722, + "samples": 2, + "values": [ + 16.2358, + 16.2722 + ] + }, + "read_B_per_query": { + "mean": 882.8928000000001, + "min": 819.2, + "max": 946.5856, + "samples": 2, + "values": [ + 819.2, + 946.5856 + ] + }, + "write_B_per_query": { + "mean": 19.660800000000002, + "min": 6.9632, + "max": 32.3584, + "samples": 2, + "values": [ + 6.9632, + 32.3584 + ] + }, + "visited_per_query": { + "mean": 16.4872, + "min": 16.4872, + "max": 16.4872, + "samples": 2, + "values": [ + 16.4872, + 16.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 16.4872, + "min": 16.4872, + "max": 16.4872, + "samples": 2, + "values": [ + 16.4872, + 16.4872 + ] + }, + "checksum": "82092296", + "verified": true + }, + { + "collisions": 0, + "limit": 1, + "mode": "predicate", + "latency_ns": { + "mean": 16885.15, + "min": 16617.1, + "max": 17153.2, + "samples": 2, + "values": [ + 17153.2, + 16617.1 + ] + }, + "baseline": { + "mean": 17532.05, + "min": 17470.1, + "max": 17594.0, + "samples": 2, + "values": [ + 17594.0, + 17470.1 + ] + }, + "latency_delta_pct": -3.6898137981582146, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 15.60015, + "min": 15.4058, + "max": 15.7945, + "samples": 2, + "values": [ + 15.7945, + 15.4058 + ] + }, + "read_B_per_query": { + "mean": 903.7824, + "min": 883.5072, + "max": 924.0576, + "samples": 2, + "values": [ + 924.0576, + 883.5072 + ] + }, + "write_B_per_query": { + "mean": 50.790400000000005, + "min": 24.576, + "max": 77.0048, + "samples": 2, + "values": [ + 77.0048, + 24.576 + ] + }, + "visited_per_query": { + "mean": 1.0, + "min": 1.0, + "max": 1.0, + "samples": 2, + "values": [ + 1.0, + 1.0 + ] + }, + "predicate_calls_per_query": { + "mean": 16.4872, + "min": 16.4872, + "max": 16.4872, + "samples": 2, + "values": [ + 16.4872, + 16.4872 + ] + }, + "checksum": "82092296", + "verified": true + }, + { + "collisions": 0, + "limit": 1, + "mode": "pruned", + "latency_ns": { + "mean": 14674.15, + "min": 14467.8, + "max": 14880.5, + "samples": 2, + "values": [ + 14467.8, + 14880.5 + ] + }, + "baseline": { + "mean": 17532.05, + "min": 17470.1, + "max": 17594.0, + "samples": 2, + "values": [ + 17594.0, + 17470.1 + ] + }, + "latency_delta_pct": -16.301003020183035, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 13.618500000000001, + "min": 13.5416, + "max": 13.6954, + "samples": 2, + "values": [ + 13.5416, + 13.6954 + ] + }, + "read_B_per_query": { + "mean": 865.0752, + "min": 862.208, + "max": 867.9424, + "samples": 2, + "values": [ + 867.9424, + 862.208 + ] + }, + "write_B_per_query": { + "mean": 27.852800000000002, + "min": 12.6976, + "max": 43.008, + "samples": 2, + "values": [ + 12.6976, + 43.008 + ] + }, + "visited_per_query": { + "mean": 1.0, + "min": 1.0, + "max": 1.0, + "samples": 2, + "values": [ + 1.0, + 1.0 + ] + }, + "predicate_calls_per_query": { + "mean": 12.9959, + "min": 12.9959, + "max": 12.9959, + "samples": 2, + "values": [ + 12.9959, + 12.9959 + ] + }, + "checksum": "82092296", + "verified": true + }, + { + "collisions": 0, + "limit": 2, + "mode": "application", + "latency_ns": { + "mean": 13014.45, + "min": 12844.0, + "max": 13184.9, + "samples": 2, + "values": [ + 13184.9, + 12844.0 + ] + }, + "baseline": { + "mean": 13014.45, + "min": 12844.0, + "max": 13184.9, + "samples": 2, + "values": [ + 13184.9, + 12844.0 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 12.8644, + "min": 12.687699999999998, + "max": 13.0411, + "samples": 2, + "values": [ + 13.0411, + 12.687699999999998 + ] + }, + "read_B_per_query": { + "mean": 31.744, + "min": 31.5392, + "max": 31.9488, + "samples": 2, + "values": [ + 31.5392, + 31.9488 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 48.4872, + "min": 48.4872, + "max": 48.4872, + "samples": 2, + "values": [ + 48.4872, + 48.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 48.4872, + "min": 48.4872, + "max": 48.4872, + "samples": 2, + "values": [ + 48.4872, + 48.4872 + ] + }, + "checksum": "165340036", + "verified": true + }, + { + "collisions": 0, + "limit": 2, + "mode": "predicate", + "latency_ns": { + "mean": 11918.05, + "min": 11888.4, + "max": 11947.7, + "samples": 2, + "values": [ + 11888.4, + 11947.7 + ] + }, + "baseline": { + "mean": 13014.45, + "min": 12844.0, + "max": 13184.9, + "samples": 2, + "values": [ + 13184.9, + 12844.0 + ] + }, + "latency_delta_pct": -8.42448201806455, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 11.8613, + "min": 11.8098, + "max": 11.9128, + "samples": 2, + "values": [ + 11.8098, + 11.9128 + ] + }, + "read_B_per_query": { + "mean": 33.5872, + "min": 29.9008, + "max": 37.2736, + "samples": 2, + "values": [ + 37.2736, + 29.9008 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 2.0, + "min": 2.0, + "max": 2.0, + "samples": 2, + "values": [ + 2.0, + 2.0 + ] + }, + "predicate_calls_per_query": { + "mean": 48.4872, + "min": 48.4872, + "max": 48.4872, + "samples": 2, + "values": [ + 48.4872, + 48.4872 + ] + }, + "checksum": "165340036", + "verified": true + }, + { + "collisions": 0, + "limit": 2, + "mode": "pruned", + "latency_ns": { + "mean": 10029.25, + "min": 9743.7, + "max": 10314.8, + "samples": 2, + "values": [ + 10314.8, + 9743.7 + ] + }, + "baseline": { + "mean": 13014.45, + "min": 12844.0, + "max": 13184.9, + "samples": 2, + "values": [ + 13184.9, + 12844.0 + ] + }, + "latency_delta_pct": -22.937580919670065, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 9.95835, + "min": 9.7276, + "max": 10.1891, + "samples": 2, + "values": [ + 10.1891, + 9.7276 + ] + }, + "read_B_per_query": { + "mean": 55.705600000000004, + "min": 49.9712, + "max": 61.44, + "samples": 2, + "values": [ + 49.9712, + 61.44 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 2.0, + "min": 2.0, + "max": 2.0, + "samples": 2, + "values": [ + 2.0, + 2.0 + ] + }, + "predicate_calls_per_query": { + "mean": 38.9746, + "min": 38.9746, + "max": 38.9746, + "samples": 2, + "values": [ + 38.9746, + 38.9746 + ] + }, + "checksum": "165340036", + "verified": true + }, + { + "collisions": 0, + "limit": 10, + "mode": "application", + "latency_ns": { + "mean": 31227.199999999997, + "min": 30789.8, + "max": 31664.6, + "samples": 2, + "values": [ + 31664.6, + 30789.8 + ] + }, + "baseline": { + "mean": 31227.199999999997, + "min": 30789.8, + "max": 31664.6, + "samples": 2, + "values": [ + 31664.6, + 30789.8 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 31.11, + "min": 30.6814, + "max": 31.5386, + "samples": 2, + "values": [ + 31.5386, + 30.6814 + ] + }, + "read_B_per_query": { + "mean": 0.4096, + "min": 0.4096, + "max": 0.4096, + "samples": 2, + "values": [ + 0.4096, + 0.4096 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 304.4872, + "min": 304.4872, + "max": 304.4872, + "samples": 2, + "values": [ + 304.4872, + 304.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 304.4872, + "min": 304.4872, + "max": 304.4872, + "samples": 2, + "values": [ + 304.4872, + 304.4872 + ] + }, + "checksum": "819143660", + "verified": true + }, + { + "collisions": 0, + "limit": 10, + "mode": "predicate", + "latency_ns": { + "mean": 27108.699999999997, + "min": 26638.3, + "max": 27579.1, + "samples": 2, + "values": [ + 26638.3, + 27579.1 + ] + }, + "baseline": { + "mean": 31227.199999999997, + "min": 30789.8, + "max": 31664.6, + "samples": 2, + "values": [ + 31664.6, + 30789.8 + ] + }, + "latency_delta_pct": -13.188822564943381, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 27.023049999999998, + "min": 26.5547, + "max": 27.4914, + "samples": 2, + "values": [ + 26.5547, + 27.4914 + ] + }, + "read_B_per_query": { + "mean": 0.4096, + "min": 0.4096, + "max": 0.4096, + "samples": 2, + "values": [ + 0.4096, + 0.4096 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 10.0, + "min": 10.0, + "max": 10.0, + "samples": 2, + "values": [ + 10.0, + 10.0 + ] + }, + "predicate_calls_per_query": { + "mean": 304.4872, + "min": 304.4872, + "max": 304.4872, + "samples": 2, + "values": [ + 304.4872, + 304.4872 + ] + }, + "checksum": "819143660", + "verified": true + }, + { + "collisions": 0, + "limit": 10, + "mode": "pruned", + "latency_ns": { + "mean": 21599.6, + "min": 21175.9, + "max": 22023.3, + "samples": 2, + "values": [ + 22023.3, + 21175.9 + ] + }, + "baseline": { + "mean": 31227.199999999997, + "min": 30789.8, + "max": 31664.6, + "samples": 2, + "values": [ + 31664.6, + 30789.8 + ] + }, + "latency_delta_pct": -30.830814162012597, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 21.55005, + "min": 21.1409, + "max": 21.9592, + "samples": 2, + "values": [ + 21.9592, + 21.1409 + ] + }, + "read_B_per_query": { + "mean": 0.8192, + "min": 0.8192, + "max": 0.8192, + "samples": 2, + "values": [ + 0.8192, + 0.8192 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 10.0, + "min": 10.0, + "max": 10.0, + "samples": 2, + "values": [ + 10.0, + 10.0 + ] + }, + "predicate_calls_per_query": { + "mean": 246.7878, + "min": 246.7878, + "max": 246.7878, + "samples": 2, + "values": [ + 246.7878, + 246.7878 + ] + }, + "checksum": "819143660", + "verified": true + }, + { + "collisions": 0, + "limit": 100, + "mode": "application", + "latency_ns": { + "mean": 221920.55, + "min": 218968.3, + "max": 224872.8, + "samples": 2, + "values": [ + 224872.8, + 218968.3 + ] + }, + "baseline": { + "mean": 221920.55, + "min": 218968.3, + "max": 224872.8, + "samples": 2, + "values": [ + 224872.8, + 218968.3 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 221.8961, + "min": 218.9401, + "max": 224.8521, + "samples": 2, + "values": [ + 224.8521, + 218.9401 + ] + }, + "read_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 3184.4872, + "min": 3184.4872, + "max": 3184.4872, + "samples": 2, + "values": [ + 3184.4872, + 3184.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 3184.4872, + "min": 3184.4872, + "max": 3184.4872, + "samples": 2, + "values": [ + 3184.4872, + 3184.4872 + ] + }, + "checksum": "8268776600", + "verified": true + }, + { + "collisions": 0, + "limit": 100, + "mode": "predicate", + "latency_ns": { + "mean": 190372.25, + "min": 188615.8, + "max": 192128.7, + "samples": 2, + "values": [ + 188615.8, + 192128.7 + ] + }, + "baseline": { + "mean": 221920.55, + "min": 218968.3, + "max": 224872.8, + "samples": 2, + "values": [ + 224872.8, + 218968.3 + ] + }, + "latency_delta_pct": -14.216033621041401, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 190.34575, + "min": 188.5794, + "max": 192.1121, + "samples": 2, + "values": [ + 188.5794, + 192.1121 + ] + }, + "read_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 100.0, + "min": 100.0, + "max": 100.0, + "samples": 2, + "values": [ + 100.0, + 100.0 + ] + }, + "predicate_calls_per_query": { + "mean": 3184.4872, + "min": 3184.4872, + "max": 3184.4872, + "samples": 2, + "values": [ + 3184.4872, + 3184.4872 + ] + }, + "checksum": "8268776600", + "verified": true + }, + { + "collisions": 0, + "limit": 100, + "mode": "pruned", + "latency_ns": { + "mean": 150711.55, + "min": 149478.0, + "max": 151945.1, + "samples": 2, + "values": [ + 151945.1, + 149478.0 + ] + }, + "baseline": { + "mean": 221920.55, + "min": 218968.3, + "max": 224872.8, + "samples": 2, + "values": [ + 224872.8, + 218968.3 + ] + }, + "latency_delta_pct": -32.08760973240198, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 150.70065, + "min": 149.4755, + "max": 151.9258, + "samples": 2, + "values": [ + 151.9258, + 149.4755 + ] + }, + "read_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 100.0, + "min": 100.0, + "max": 100.0, + "samples": 2, + "values": [ + 100.0, + 100.0 + ] + }, + "predicate_calls_per_query": { + "mean": 2584.6811, + "min": 2584.6811, + "max": 2584.6811, + "samples": 2, + "values": [ + 2584.6811, + 2584.6811 + ] + }, + "checksum": "8268776600", + "verified": true + }, + { + "collisions": 1, + "limit": 1, + "mode": "application", + "latency_ns": { + "mean": 17774.95, + "min": 16848.7, + "max": 18701.2, + "samples": 2, + "values": [ + 16848.7, + 18701.2 + ] + }, + "baseline": { + "mean": 17774.95, + "min": 16848.7, + "max": 18701.2, + "samples": 2, + "values": [ + 16848.7, + 18701.2 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 16.37245, + "min": 15.5823, + "max": 17.1626, + "samples": 2, + "values": [ + 15.5823, + 17.1626 + ] + }, + "read_B_per_query": { + "mean": 899.072, + "min": 870.4, + "max": 927.744, + "samples": 2, + "values": [ + 870.4, + 927.744 + ] + }, + "write_B_per_query": { + "mean": 60.416, + "min": 36.864, + "max": 83.968, + "samples": 2, + "values": [ + 36.864, + 83.968 + ] + }, + "visited_per_query": { + "mean": 16.4872, + "min": 16.4872, + "max": 16.4872, + "samples": 2, + "values": [ + 16.4872, + 16.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 16.4872, + "min": 16.4872, + "max": 16.4872, + "samples": 2, + "values": [ + 16.4872, + 16.4872 + ] + }, + "checksum": "82092296", + "verified": true + }, + { + "collisions": 1, + "limit": 1, + "mode": "predicate", + "latency_ns": { + "mean": 16435.55, + "min": 15954.9, + "max": 16916.2, + "samples": 2, + "values": [ + 16916.2, + 15954.9 + ] + }, + "baseline": { + "mean": 17774.95, + "min": 16848.7, + "max": 18701.2, + "samples": 2, + "values": [ + 16848.7, + 18701.2 + ] + }, + "latency_delta_pct": -7.5353235874081275, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 15.3072, + "min": 14.8736, + "max": 15.7408, + "samples": 2, + "values": [ + 15.7408, + 14.8736 + ] + }, + "read_B_per_query": { + "mean": 869.7855999999999, + "min": 854.8352, + "max": 884.736, + "samples": 2, + "values": [ + 854.8352, + 884.736 + ] + }, + "write_B_per_query": { + "mean": 26.624000000000002, + "min": 12.288, + "max": 40.96, + "samples": 2, + "values": [ + 12.288, + 40.96 + ] + }, + "visited_per_query": { + "mean": 1.0, + "min": 1.0, + "max": 1.0, + "samples": 2, + "values": [ + 1.0, + 1.0 + ] + }, + "predicate_calls_per_query": { + "mean": 16.4872, + "min": 16.4872, + "max": 16.4872, + "samples": 2, + "values": [ + 16.4872, + 16.4872 + ] + }, + "checksum": "82092296", + "verified": true + }, + { + "collisions": 1, + "limit": 1, + "mode": "pruned", + "latency_ns": { + "mean": 15252.599999999999, + "min": 15069.9, + "max": 15435.3, + "samples": 2, + "values": [ + 15069.9, + 15435.3 + ] + }, + "baseline": { + "mean": 17774.95, + "min": 16848.7, + "max": 18701.2, + "samples": 2, + "values": [ + 16848.7, + 18701.2 + ] + }, + "latency_delta_pct": -14.19047592257645, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 14.06405, + "min": 13.7219, + "max": 14.4062, + "samples": 2, + "values": [ + 13.7219, + 14.4062 + ] + }, + "read_B_per_query": { + "mean": 883.5072, + "min": 854.016, + "max": 912.9984, + "samples": 2, + "values": [ + 912.9984, + 854.016 + ] + }, + "write_B_per_query": { + "mean": 70.656, + "min": 48.3328, + "max": 92.9792, + "samples": 2, + "values": [ + 92.9792, + 48.3328 + ] + }, + "visited_per_query": { + "mean": 1.0, + "min": 1.0, + "max": 1.0, + "samples": 2, + "values": [ + 1.0, + 1.0 + ] + }, + "predicate_calls_per_query": { + "mean": 13.0895, + "min": 13.0895, + "max": 13.0895, + "samples": 2, + "values": [ + 13.0895, + 13.0895 + ] + }, + "checksum": "82092296", + "verified": true + }, + { + "collisions": 1, + "limit": 2, + "mode": "application", + "latency_ns": { + "mean": 14169.5, + "min": 12844.5, + "max": 15494.5, + "samples": 2, + "values": [ + 12844.5, + 15494.5 + ] + }, + "baseline": { + "mean": 14169.5, + "min": 12844.5, + "max": 15494.5, + "samples": 2, + "values": [ + 12844.5, + 15494.5 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 13.99595, + "min": 12.7171, + "max": 15.2748, + "samples": 2, + "values": [ + 12.7171, + 15.2748 + ] + }, + "read_B_per_query": { + "mean": 31.9488, + "min": 31.9488, + "max": 31.9488, + "samples": 2, + "values": [ + 31.9488, + 31.9488 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 48.4872, + "min": 48.4872, + "max": 48.4872, + "samples": 2, + "values": [ + 48.4872, + 48.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 48.4872, + "min": 48.4872, + "max": 48.4872, + "samples": 2, + "values": [ + 48.4872, + 48.4872 + ] + }, + "checksum": "165340036", + "verified": true + }, + { + "collisions": 1, + "limit": 2, + "mode": "predicate", + "latency_ns": { + "mean": 11870.75, + "min": 11843.7, + "max": 11897.8, + "samples": 2, + "values": [ + 11843.7, + 11897.8 + ] + }, + "baseline": { + "mean": 14169.5, + "min": 12844.5, + "max": 15494.5, + "samples": 2, + "values": [ + 12844.5, + 15494.5 + ] + }, + "latency_delta_pct": -16.223225943046682, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 11.735, + "min": 11.6943, + "max": 11.7757, + "samples": 2, + "values": [ + 11.6943, + 11.7757 + ] + }, + "read_B_per_query": { + "mean": 33.1776, + "min": 29.0816, + "max": 37.2736, + "samples": 2, + "values": [ + 29.0816, + 37.2736 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 2.0, + "min": 2.0, + "max": 2.0, + "samples": 2, + "values": [ + 2.0, + 2.0 + ] + }, + "predicate_calls_per_query": { + "mean": 48.4872, + "min": 48.4872, + "max": 48.4872, + "samples": 2, + "values": [ + 48.4872, + 48.4872 + ] + }, + "checksum": "165340036", + "verified": true + }, + { + "collisions": 1, + "limit": 2, + "mode": "pruned", + "latency_ns": { + "mean": 10158.5, + "min": 10051.6, + "max": 10265.4, + "samples": 2, + "values": [ + 10051.6, + 10265.4 + ] + }, + "baseline": { + "mean": 14169.5, + "min": 12844.5, + "max": 15494.5, + "samples": 2, + "values": [ + 12844.5, + 15494.5 + ] + }, + "latency_delta_pct": -28.30727972052648, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 10.030249999999999, + "min": 9.9355, + "max": 10.125, + "samples": 2, + "values": [ + 9.9355, + 10.125 + ] + }, + "read_B_per_query": { + "mean": 63.283199999999994, + "min": 62.2592, + "max": 64.3072, + "samples": 2, + "values": [ + 64.3072, + 62.2592 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 2.0, + "min": 2.0, + "max": 2.0, + "samples": 2, + "values": [ + 2.0, + 2.0 + ] + }, + "predicate_calls_per_query": { + "mean": 39.2556, + "min": 39.2556, + "max": 39.2556, + "samples": 2, + "values": [ + 39.2556, + 39.2556 + ] + }, + "checksum": "165340036", + "verified": true + }, + { + "collisions": 1, + "limit": 10, + "mode": "application", + "latency_ns": { + "mean": 31609.75, + "min": 30457.9, + "max": 32761.6, + "samples": 2, + "values": [ + 30457.9, + 32761.6 + ] + }, + "baseline": { + "mean": 31609.75, + "min": 30457.9, + "max": 32761.6, + "samples": 2, + "values": [ + 30457.9, + 32761.6 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 30.73085, + "min": 30.3617, + "max": 31.1, + "samples": 2, + "values": [ + 30.3617, + 31.1 + ] + }, + "read_B_per_query": { + "mean": 0.4096, + "min": 0.4096, + "max": 0.4096, + "samples": 2, + "values": [ + 0.4096, + 0.4096 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 304.4872, + "min": 304.4872, + "max": 304.4872, + "samples": 2, + "values": [ + 304.4872, + 304.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 304.4872, + "min": 304.4872, + "max": 304.4872, + "samples": 2, + "values": [ + 304.4872, + 304.4872 + ] + }, + "checksum": "819143660", + "verified": true + }, + { + "collisions": 1, + "limit": 10, + "mode": "predicate", + "latency_ns": { + "mean": 26943.9, + "min": 26801.9, + "max": 27085.9, + "samples": 2, + "values": [ + 26801.9, + 27085.9 + ] + }, + "baseline": { + "mean": 31609.75, + "min": 30457.9, + "max": 32761.6, + "samples": 2, + "values": [ + 30457.9, + 32761.6 + ] + }, + "latency_delta_pct": -14.760793742437063, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 26.851300000000002, + "min": 26.7066, + "max": 26.996, + "samples": 2, + "values": [ + 26.7066, + 26.996 + ] + }, + "read_B_per_query": { + "mean": 0.4096, + "min": 0.4096, + "max": 0.4096, + "samples": 2, + "values": [ + 0.4096, + 0.4096 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 10.0, + "min": 10.0, + "max": 10.0, + "samples": 2, + "values": [ + 10.0, + 10.0 + ] + }, + "predicate_calls_per_query": { + "mean": 304.4872, + "min": 304.4872, + "max": 304.4872, + "samples": 2, + "values": [ + 304.4872, + 304.4872 + ] + }, + "checksum": "819143660", + "verified": true + }, + { + "collisions": 1, + "limit": 10, + "mode": "pruned", + "latency_ns": { + "mean": 22060.75, + "min": 21833.9, + "max": 22287.6, + "samples": 2, + "values": [ + 22287.6, + 21833.9 + ] + }, + "baseline": { + "mean": 31609.75, + "min": 30457.9, + "max": 32761.6, + "samples": 2, + "values": [ + 30457.9, + 32761.6 + ] + }, + "latency_delta_pct": -30.20903360513766, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 22.0116, + "min": 21.7797, + "max": 22.2435, + "samples": 2, + "values": [ + 22.2435, + 21.7797 + ] + }, + "read_B_per_query": { + "mean": 0.8192, + "min": 0.8192, + "max": 0.8192, + "samples": 2, + "values": [ + 0.8192, + 0.8192 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 10.0, + "min": 10.0, + "max": 10.0, + "samples": 2, + "values": [ + 10.0, + 10.0 + ] + }, + "predicate_calls_per_query": { + "mean": 248.5662, + "min": 248.5662, + "max": 248.5662, + "samples": 2, + "values": [ + 248.5662, + 248.5662 + ] + }, + "checksum": "819143660", + "verified": true + }, + { + "collisions": 1, + "limit": 100, + "mode": "application", + "latency_ns": { + "mean": 217807.84999999998, + "min": 217525.3, + "max": 218090.4, + "samples": 2, + "values": [ + 217525.3, + 218090.4 + ] + }, + "baseline": { + "mean": 217807.84999999998, + "min": 217525.3, + "max": 218090.4, + "samples": 2, + "values": [ + 217525.3, + 218090.4 + ] + }, + "latency_delta_pct": 0.0, + "ranges_overlap": true, + "cpu_us_per_query": { + "mean": 217.77530000000002, + "min": 217.4846, + "max": 218.066, + "samples": 2, + "values": [ + 217.4846, + 218.066 + ] + }, + "read_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 3184.4872, + "min": 3184.4872, + "max": 3184.4872, + "samples": 2, + "values": [ + 3184.4872, + 3184.4872 + ] + }, + "predicate_calls_per_query": { + "mean": 3184.4872, + "min": 3184.4872, + "max": 3184.4872, + "samples": 2, + "values": [ + 3184.4872, + 3184.4872 + ] + }, + "checksum": "8268776600", + "verified": true + }, + { + "collisions": 1, + "limit": 100, + "mode": "predicate", + "latency_ns": { + "mean": 189426.40000000002, + "min": 188514.2, + "max": 190338.6, + "samples": 2, + "values": [ + 188514.2, + 190338.6 + ] + }, + "baseline": { + "mean": 217807.84999999998, + "min": 217525.3, + "max": 218090.4, + "samples": 2, + "values": [ + 217525.3, + 218090.4 + ] + }, + "latency_delta_pct": -13.030499130311402, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 189.41965, + "min": 188.5154, + "max": 190.3239, + "samples": 2, + "values": [ + 188.5154, + 190.3239 + ] + }, + "read_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 100.0, + "min": 100.0, + "max": 100.0, + "samples": 2, + "values": [ + 100.0, + 100.0 + ] + }, + "predicate_calls_per_query": { + "mean": 3184.4872, + "min": 3184.4872, + "max": 3184.4872, + "samples": 2, + "values": [ + 3184.4872, + 3184.4872 + ] + }, + "checksum": "8268776600", + "verified": true + }, + { + "collisions": 1, + "limit": 100, + "mode": "pruned", + "latency_ns": { + "mean": 150981.9, + "min": 147735.9, + "max": 154227.9, + "samples": 2, + "values": [ + 147735.9, + 154227.9 + ] + }, + "baseline": { + "mean": 217807.84999999998, + "min": 217525.3, + "max": 218090.4, + "samples": 2, + "values": [ + 217525.3, + 218090.4 + ] + }, + "latency_delta_pct": -30.681148544462467, + "ranges_overlap": false, + "cpu_us_per_query": { + "mean": 150.96895, + "min": 147.7217, + "max": 154.2162, + "samples": 2, + "values": [ + 147.7217, + 154.2162 + ] + }, + "read_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "write_B_per_query": { + "mean": 0.0, + "min": 0.0, + "max": 0.0, + "samples": 2, + "values": [ + 0.0, + 0.0 + ] + }, + "visited_per_query": { + "mean": 100.0, + "min": 100.0, + "max": 100.0, + "samples": 2, + "values": [ + 100.0, + 100.0 + ] + }, + "predicate_calls_per_query": { + "mean": 2603.3223, + "min": 2603.3223, + "max": 2603.3223, + "samples": 2, + "values": [ + 2603.3223, + 2603.3223 + ] + }, + "checksum": "8268776600", + "verified": true + } + ] +} diff --git a/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/timing-ready.json b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/timing-ready.json new file mode 100644 index 000000000..fe1d9230a --- /dev/null +++ b/benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/timing-ready.json @@ -0,0 +1,10 @@ +{ + "authorized_by": "root explicit TIMING READY message", + "time": 1791922360.9153633, + "heads": { + "main": "0fa3bd6c7723690c760294d33ba610e3724bd8d2", + "filtered-ranges": "cf7dc4f710aad91dfad9050dca0d326ff0fccac5", + "disk-format": "3e302c04c60babb8afecb58d27f808848624f83c" + }, + "resumed_after_companion_tests": 1791922976.7800455 +} diff --git a/docs/disk-format-pr.md b/docs/disk-format-pr.md new file mode 100644 index 000000000..b37585bda --- /dev/null +++ b/docs/disk-format-pr.md @@ -0,0 +1,11 @@ +Persist the versions and reader settings needed to interpret existing database structures. Format 8 rejects older files clearly; reopen uses saved routing-filter widths/hash seeds and log checksum seeds, while current configuration controls newly constructed instances. Branch versions travel with their references, mini-allocator pages validate their headers, and each owning module decides which versions it supports. The newest valid superblock is selected before compatibility checks, so unsupported versions recognizable under the current checksum layout cannot silently fall back to an older checkpoint. + +This branch includes main `0fa3bd6c`, retaining transferable cache handles, multi-level blobs, widened statistics and the notification-based durable-barrier API. Applications must still supply compatible comparison, hash and merge callbacks. There is no migration reader or portable cross-architecture encoding. + +Validation on this revision: + +- Debug and release each passed 150 unit-test executions, including notification/optimize coverage with both private and shared heaps, the full routing-filter functional suite and the complete functional log suite with a 500,000-record round trip. Each mode skipped the expected notification allocation-failure case once per heap mode because fault injection was disabled. +- Both inspection-tool builds read the previous format-8 fixture; the release tool clearly rejected main's format-7 fixture. Tests cover invalid versions, changed configuration across reopen/compaction, synchronous/asynchronous reads, cache errors, recovery and blobs. +- All targets, tools and benchmark programs build; the exact clean-export CI formatting check and full PR diff check pass. The full compiler/ASan CI matrix was not rerun. + +Fresh benchmarks use serialized O_DIRECT files on real disk, with fsynced fixture copies and zero query errors/violations. Resident eight-reader point throughput lost 4.39% in the broad screen and 7.75% in a focused reversed-order repeat (non-overlapping ranges, zero disk reads; mean CPU/op 1.965 us to 2.131 us). Other read results varied in both directions. This remaining cost is unresolved: the branch is prepared for review, but performance neutrality and merge readiness are not established. No fresh write-throughput matrix was run. [Raw samples, commands, hashes and control drift](../benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.txt) and [correctness evidence](../benchmarks/results-20261013/disk-format-pr-0fa3bd6c/README.md) accompany the change. diff --git a/docs/on-disk-format.md b/docs/on-disk-format.md index a719b5a42..b226e583b 100644 --- a/docs/on-disk-format.md +++ b/docs/on-disk-format.md @@ -47,7 +47,56 @@ schemas cannot be persisted as portable identities. The persisted hash seed is passed to the supplied hash callback. This change does not introduce a portable byte-order encoding or promise compatibility between arbitrary architectures. -# Validation and measurements +## Current PR readiness and performance + +The branch includes main `0fa3bd6c`. Measured source is `3e302c04`; later changes +record evidence and documentation. Debug/release each passed 150 unit-test +executions, with two expected allocation-fault-injection skips per mode, plus +full routing-filter and log suites (a 500,000-record round trip). Inspection +reads the previous format-8 fixture and clearly rejects main's format-7 fixture. +All targets/tools/benchmarks build; clean-export CI formatting and PR whitespace +checks pass. The full compiler/sanitizer CI matrix was not rerun. + +**The branch is prepared for review, but its resident point-query regression +remains unresolved.** The broad resident screen measured -4.39% eight-reader +point throughput. A separate 15-second, disk-format / main / main / disk-format +repeat reproduced a -7.75% loss with non-overlapping sample ranges: + +| Focused resident point queries, 8 readers | Main | Disk-format | +| --- | ---: | ---: | +| Throughput range | 4.026–4.115 million/s | 3.651–3.860 million/s | +| Mean CPU per operation | 1.965 us | 2.131 us | +| Measured physical read bytes | 0 | 0 | + +This is a CPU-side performance concern; the measurements do not isolate its +source within the format changes. The formats require independently constructed +fixtures, so tree shape remains a comparability limitation. Performance neutrality +has not been established, and this evidence does not justify calling it merge-ready +on performance grounds. + +| Broad-screen throughput change | Resident | Disk-backed working set | +| --- | ---: | ---: | +| Point query, 1 reader | -1.13% | Not measured | +| Point query, 8 readers | -4.39% | +4.87% | +| Range length 1, 8 readers | +0.78% | +1.41% | +| Range length 10,000, 8 readers | +0.13% | -0.79% | + +All checks passed with zero query errors or result violations. The disk results +include device variation and do not establish a speedup. Each profile has two +samples per regime; observed ranges are not confidence intervals. No fresh +write-throughput matrix was run. Timings compare the whole feature with current +main, not just the main merge. All database files used O_DIRECT on physical ext4 +storage, with exclusive benchmark/machine locks and explicit fsync after copying +fixtures. Source/library/executable hashes, commands, samples, control drift and +CPU/I/O are in [the current benchmark report](../benchmarks/results-20261013/predicate-disk-main-0fa3bd6c/report.txt). + +A separate companion branch, `robj/filtered-ranges-disk-format` at `36183807`, +passed 125 unit cases per debug/release mode plus the full filter suite. It proves +correct predicate scans under saved seeds 11/22 with current construction default99, +cache eviction over six seeds, and copied-key ownership when a later seed is used. +The two feature PRs remain independent; this is correctness evidence only. + +# Historical validation and measurements Debug validation: 16 superblock tests, 33 internal Splinter tests, 47 public quick tests, full functional routing-filter and log suites. Added regressions cover @@ -196,7 +245,7 @@ does not make earlier format-8 pages unreadable. Debug and release each passed 116 unit cases: B-tree 11, superblock 16, core 38, public quick API 49, and async trunk 2. Both also passed the complete functional -routing-filter suite and the log suite with 10,000 inserts. These cover changed +routing-filter suite and the complete log suite (500,000-record round trip). These cover changed configuration across reopen, invalid structure versions, cache read failures, recovery, multi-level blobs, and cross-thread iterator handoff. The first core run used the old 4 GiB test device limit and could not allocate the sparse @@ -226,7 +275,7 @@ durability and checkpoint paths are retained, including the two-argument public barrier API and blocking `NULL` notification semantics. The disk format remains 8. Debug and release each passed 150 unit cases and the complete functional -routing-filter and 10,000-insert log suites. Each mode intentionally skipped the +routing-filter and log suites (500,000-record round trip). Each mode intentionally skipped the allocation-failure notification case twice because fault injection is disabled. Both inspection tools read the previous format-8 fixture; the release tool clearly rejects main's format-7 fixture. All builds, tools, benchmarks and the