Preserve typed map keys in MessagePack and binary archives

docs/content/standard_containers.md

@@ -21,7 +21,7 @@ This will be represented as follows:
 {"firstName":"Homer","lastName":"Simpson"}
 ```
 
-The same is true for numeric types:
+For JSON and other formats that use string map keys, the same is true for numeric types:
 
 ```cpp
 std::map<int, std::string> homer;
@@ -56,3 +56,25 @@ will be represented as arrays. Containers for which the `value_type`
 is a key-value-pair will be represented as arrays of pairs.
 
 `std::expected` is an exception to this: it is serialized as its value type, or as an object with a single `error` field. Refer to the [std::expected](../expected) section for details.
+
+## Typed map keys in binary formats
+
+MessagePack, Cereal, YAS, and Boost.Serialization serialize the keys of the
+maps described above as values of their original type. This applies to both
+`std::map` and `std::unordered_map`, including numeric `ReflectionType` wrappers
+such as `rfl::Validator`. Validation is applied when reading keys.
+
+For example, `std::map<int, std::string>{{1, "Homer"}}` contains an integer key
+in MessagePack, rather than the string `"1"`. The archive backends write the
+entry count followed by each typed key and its value. No intermediate container
+or conversion to/from decimal strings is needed. Floating-point keys retain
+their precision instead of being rounded by `std::to_string`.
+
+This changes the wire format for numeric keys in these backends. There is no
+automatic detection or compatibility mode for previously serialized string
+keys; readers and writers must use the same representation.
+
+Other backends retain their existing representation. In particular, the CBOR
+adapter currently reads through a JSON DOM with string object keys, so native
+typed CBOR map keys are not supported by this change. Maps with compound keys
+and multimaps continue to use arrays of pairs.

docs/content/supported_formats/supporting_your_own_format.md

@@ -443,3 +443,53 @@ struct Reader {
 };
 ```
 
+
+### Optional typed map keys
+
+Backends that preserve map key types provide explicit map-entry APIs for
+writing and reading. They are detected independently by
+`SupportsTypedMapWriter` and `SupportsTypedMapReader`.
+
+A writer defines an `OutputMapEntryType` context and the following methods:
+
+```cpp
+OutputMapEntryType map_entry(OutputMapType* map) const;
+OutputArrayType add_array_to_map_entry(size_t size, OutputMapEntryType*) const;
+OutputObjectType add_object_to_map_entry(size_t size, OutputMapEntryType*) const;
+OutputVarType add_null_to_map_entry(OutputMapEntryType*) const;
+
+template <class T>
+OutputVarType add_value_to_map_entry(const T& value, OutputMapEntryType*) const;
+
+// Additionally required for schemaful writers:
+OutputMapType add_map_to_map_entry(size_t size, OutputMapEntryType*) const;
+OutputUnionType add_union_to_map_entry(OutputMapEntryType*) const;
+```
+
+Schemaless writers use `OutputObjectType*` instead of `OutputMapType*` for
+`map_entry`. The map is already open: `MapParser` uses the normal map header
+for schemaful backends, or the object header for schemaless backends, and closes
+it after all entries. For each entry, it creates a context and passes it through
+`Parent::MapEntry` to the ordinary key and value parsers, in that order.
+The context does not introduce an array or a pair-length prefix. Streaming
+backends can use an empty context; other backends can retain their entry state
+in it. The `add_*_to_map_entry` methods emit only the requested key or value.
+
+The corresponding reader provides:
+
+```cpp
+template <class MapReader>
+std::optional<Error> read_typed_map(const MapReader& reader,
+                                  const InputMapType& map) const noexcept;
+```
+
+Schemaless readers use `InputObjectType` instead of `InputMapType`. For each
+entry, call `reader.read(key, value)` with two `InputVarType` values and return
+immediately if it returns an error. The shared `TypedMapReader` parses the key
+before the value, so sequential archives may pass two handles to the same
+archive. Backend traversal handles the map entry count and boundaries.
+
+Schemaful readers can provide either `read_map` for string keys or
+`read_typed_map` for typed keys. They do not need to implement both. Backends
+without the typed-map interfaces retain string-key serialization. Objects
+representing struct fields continue to use their existing field-name behavior.

include/rfl/boost_serialization/Reader.hpp

@@ -155,24 +155,20 @@ struct Reader {
     return std::nullopt;
   }
 
-  /// Reads all key-value pairs from a map using the provided map reader.
-  /// Keys are read as strings from the archive.
-  /// @tparam MapReader The type of reader that processes individual map entries
-  /// @param _map_reader The reader object that processes each key-value pair
-  /// @param _map The map structure containing size information
-  /// @return std::nullopt on success, or an Error if reading fails
+  /// Reads keys and values directly from the archive in sequence.
   template <class MapReader>
-  std::optional<Error> read_map(const MapReader& _map_reader,
-                                const InputMapType& _map) const noexcept {
+  std::optional<Error> read_typed_map(const MapReader& _reader,
+                                      const InputMapType& _map) const noexcept {
     try {
-      for (size_t i = 0; i < _map.size; ++i) {
-        std::string key;
-        *_map.ar >> key;
-        auto var = InputVarType{_map.ar};
-        _map_reader.read(std::string_view(key), var);
+      const auto size = _map.size;
+      for (size_t i = 0; i < size; ++i) {
+        const auto var = InputVarType{_map.ar};
+        if (const auto err = _reader.read(var, var)) {
+          return err;
+        }
       }
       return std::nullopt;
-    } catch (std::exception& e) {
+    } catch (const std::exception& e) {
       return Error(e.what());
     }
   }

include/rfl/boost_serialization/Writer.hpp

@@ -31,6 +31,41 @@ class Writer {
   using OutputUnionType = BoostOutputUnion;
   using OutputVarType = BoostOutputVar;
 
+  /// A context for writing a typed map entry's key, then its value.
+  struct OutputMapEntryType {};
+
+  OutputMapEntryType map_entry(OutputMapType*) const { return {}; }
+
+  OutputArrayType add_array_to_map_entry(const size_t _size,
+                                         OutputMapEntryType*) const {
+    return array_as_root(_size);
+  }
+
+  OutputObjectType add_object_to_map_entry(const size_t _size,
+                                           OutputMapEntryType*) const {
+    return object_as_root(_size);
+  }
+
+  template <class T>
+  OutputVarType add_value_to_map_entry(const T& _var,
+                                       OutputMapEntryType*) const {
+    new_value(_var);
+    return OutputVarType{};
+  }
+
+  OutputVarType add_null_to_map_entry(OutputMapEntryType*) const {
+    return null_as_root();
+  }
+
+  OutputMapType add_map_to_map_entry(const size_t _size,
+                                     OutputMapEntryType*) const {
+    return map_as_root(_size);
+  }
+
+  OutputUnionType add_union_to_map_entry(OutputMapEntryType*) const {
+    return union_as_root();
+  }
+
   /// Constructs a Writer with a Boost output archive.
   /// @param _ar Pointer to the Boost output archive to write to
   Writer(OArchive* _ar) : ar_(_ar) {}

include/rfl/cereal/Reader.hpp

@@ -146,26 +146,22 @@ struct Reader {
     }
   }
 
-  /// Reads all key-value pairs from a Cereal map using the provided map reader.
-  /// Keys are stored as strings in the archive.
-  /// @tparam MapReader The type of reader that processes individual map entries
-  /// @param _map_reader The reader object that processes each key-value pair
-  /// @param _map The Cereal map structure
-  /// @return std::nullopt on success, or an Error if reading fails
+  /// Reads keys and values directly from the archive in sequence.
   template <class MapReader>
-  std::optional<Error> read_map(const MapReader& _map_reader,
-                                const InputMapType& _map) const noexcept {
+  std::optional<Error> read_typed_map(const MapReader& _reader,
+                                      const InputMapType& _map) const noexcept {
     try {
       size_t size;
       (*_map.archive_)(::cereal::make_size_tag(size));
       for (size_t i = 0; i < size; ++i) {
-        std::string key;
-        (*_map.archive_)(key);
-        _map_reader.read(std::string_view(key), InputVarType{_map.archive_});
+        const auto var = InputVarType{_map.archive_};
+        if (const auto err = _reader.read(var, var)) {
+          return err;
+        }
       }
       return std::nullopt;
-    } catch (std::exception& e) {
-      return Error(std::string("Cereal map read error: ") + e.what());
+    } catch (const std::exception& e) {
+      return Error(e.what());
     }
   }
 

include/rfl/cereal/Writer.hpp

@@ -42,6 +42,41 @@ class Writer {
   using OutputUnionType = CerealOutputUnion;
   using OutputVarType = CerealOutputVar;
 
+  /// A context for writing a typed map entry's key, then its value.
+  struct OutputMapEntryType {};
+
+  OutputMapEntryType map_entry(OutputMapType*) const { return {}; }
+
+  OutputArrayType add_array_to_map_entry(const size_t _size,
+                                         OutputMapEntryType*) const {
+    return array_as_root(_size);
+  }
+
+  OutputObjectType add_object_to_map_entry(const size_t _size,
+                                           OutputMapEntryType*) const {
+    return object_as_root(_size);
+  }
+
+  template <class T>
+  OutputVarType add_value_to_map_entry(const T& _var,
+                                       OutputMapEntryType*) const {
+    add_value(_var);
+    return OutputVarType{};
+  }
+
+  OutputVarType add_null_to_map_entry(OutputMapEntryType*) const {
+    return null_as_root();
+  }
+
+  OutputMapType add_map_to_map_entry(const size_t _size,
+                                     OutputMapEntryType*) const {
+    return map_as_root(_size);
+  }
+
+  OutputUnionType add_union_to_map_entry(OutputMapEntryType*) const {
+    return union_as_root();
+  }
+
   /// Constructs a Writer with a Cereal output archive.
   /// @param _archive Pointer to the Cereal output archive to write to
   Writer(CerealArchive* _archive);

include/rfl/msgpack/Reader.hpp

@@ -184,6 +184,18 @@ struct Reader {
     return std::nullopt;
   }
 
+  /// Reads native map entries without converting keys to field names.
+  template <class MapReader>
+  std::optional<Error> read_typed_map(
+      const MapReader& _reader, const InputObjectType& _map) const noexcept {
+    for (uint32_t i = 0; i < _map.size; ++i) {
+      if (const auto err = _reader.read(_map.ptr[i].key, _map.ptr[i].val)) {
+        return err;
+      }
+    }
+    return std::nullopt;
+  }
+
   /**
    * @brief Reads all fields of a msgpack object using a provided reader.
    * @tparam ObjectReader The type of the object reader.

include/rfl/msgpack/Writer.hpp

@@ -26,6 +26,31 @@ class RFL_API Writer {
   using OutputObjectType = MsgpackOutputObject;
   using OutputVarType = MsgpackOutputVar;
 
+  /// A context for writing a typed map entry's key, then its value.
+  struct OutputMapEntryType {};
+
+  OutputMapEntryType map_entry(OutputObjectType*) const { return {}; }
+
+  OutputArrayType add_array_to_map_entry(const size_t _size,
+                                         OutputMapEntryType*) const {
+    return array_as_root(_size);
+  }
+
+  OutputObjectType add_object_to_map_entry(const size_t _size,
+                                           OutputMapEntryType*) const {
+    return object_as_root(_size);
+  }
+
+  template <class T>
+  OutputVarType add_value_to_map_entry(const T& _var,
+                                       OutputMapEntryType*) const {
+    return new_value(_var);
+  }
+
+  OutputVarType add_null_to_map_entry(OutputMapEntryType*) const {
+    return null_as_root();
+  }
+
   Writer(msgpack_packer* _pk);
 
   ~Writer();

include/rfl/parsing/MapParser.hpp

@@ -11,6 +11,9 @@
 #include "MapReader.hpp"
 #include "Parent.hpp"
 #include "Parser_base.hpp"
+#include "SupportsTypedMapReader.hpp"
+#include "SupportsTypedMapWriter.hpp"
+#include "TypedMapReader.hpp"
 #include "schema/Type.hpp"
 #include "schemaful/IsSchemafulReader.hpp"
 #include "schemaful/IsSchemafulWriter.hpp"
@@ -60,7 +63,9 @@ struct MapParser {
    */
   template <class P>
   static void write(const W& _w, const MapType& _m, const P& _parent) {
-    if constexpr (schemaful::IsSchemafulWriter<W>) {
+    if constexpr (SupportsTypedMapWriter<W>) {
+      write_typed_map(_w, _m, _parent);
+    } else if constexpr (schemaful::IsSchemafulWriter<W>) {
       write_map(_w, _m, _parent);
     } else {
       write_object(_w, _m, _parent);
@@ -82,24 +87,57 @@ struct MapParser {
  private:
   static Result<MapType> make_map(const R& _r, const auto& _obj_or_map) {
     MapType map;
-    std::vector<std::string> errors;
-    const auto map_reader =
-        MapReader<R, W, MapType, ProcessorsType>(&_r, &map, &errors);
-    if constexpr (schemaful::IsSchemafulReader<R>) {
-      const auto err = _r.read_map(map_reader, _obj_or_map);
-      if (err) {
+    if constexpr (SupportsTypedMapReader<
+                      R, std::remove_cvref_t<decltype(_obj_or_map)>>) {
+      const auto reader =
+          TypedMapReader<R, W, MapType, ProcessorsType>{&_r, &map};
+      if (const auto err = _r.read_typed_map(reader, _obj_or_map)) {
         return error(*err);
       }
+      return map;
     } else {
-      const auto err = _r.read_object(map_reader, _obj_or_map);
-      if (err) {
-        return error(*err);
+      std::vector<std::string> errors;
+      const auto map_reader =
+          MapReader<R, W, MapType, ProcessorsType>(&_r, &map, &errors);
+      if constexpr (schemaful::IsSchemafulReader<R>) {
+        const auto err = _r.read_map(map_reader, _obj_or_map);
+        if (err) {
+          return error(*err);
+        }
+      } else {
+        const auto err = _r.read_object(map_reader, _obj_or_map);
+        if (err) {
+          return error(*err);
+        }
+      }
+      if (errors.size() != 0) {
+        return error(to_single_error_message(errors));
       }
+      return map;
     }
-    if (errors.size() != 0) {
-      return error(to_single_error_message(errors));
+  }
+
+  template <class P>
+  static void write_typed_map(const W& _w, const MapType& _m,
+                              const P& _parent) {
+    const auto write_entries = [&](auto* map) {
+      for (const auto& [key, value] : _m) {
+        auto entry = _w.map_entry(map);
+        const auto parent = typename ParentType::template MapEntry<
+            typename W::OutputMapEntryType>{&entry};
+        Parser<R, W, KeyType, ProcessorsType>::write(_w, key, parent);
+        Parser<R, W, ValueType, ProcessorsType>::write(_w, value, parent);
+      }
+    };
+    if constexpr (schemaful::IsSchemafulWriter<W>) {
+      auto map = ParentType::add_map(_w, _m.size(), _parent);
+      write_entries(&map);
+      _w.end_map(&map);
+    } else {
+      auto map = ParentType::add_object(_w, _m.size(), _parent);
+      write_entries(&map);
+      _w.end_object(&map);
     }
-    return map;
   }
 
   template <class P>