Hexi is a new, header-only C++ library for network binary serialization. It focuses on modern C++ features, aiming for ease of use, safety, and performance. Hexi supports user-defined types, standard containers, and common data structures out-of-the-box, minimizing boilerplate. It leverages compile-time reflection and constexpr processing to achieve efficiency comparable to hand-written serialization code, while providing a more concise and maintainable solution.
The Arroyo blog post details a significant performance improvement in decoding columnar JSON data using the Rust-based arrow-rs library. By leveraging lazy decoding and SIMD intrinsics, they achieved a substantial speedup, particularly for nested data and lists, compared to existing methods like serde_json and even Python's pyarrow. This optimization focuses on performance-critical scenarios where large JSON datasets are processed, like data engineering and analytics. The improvement stems from strategically decoding only necessary data elements and employing efficient vectorized operations, minimizing overhead and maximizing CPU utilization. This approach promises faster data loading and processing for applications built on the Apache Arrow ecosystem.
Hacker News users discussed the performance benefits and trade-offs of using Apache Arrow for JSON decoding, as presented in the linked blog post. Several commenters pointed out that the benchmarks lacked real-world complexity and that deserialization often isn't the bottleneck in data processing pipelines. Some questioned the focus on columnar format for single JSON objects, suggesting its advantages are better realized with arrays of objects. Others highlighted the importance of SIMD and memory access patterns in achieving performance gains, while some suggested alternative libraries like simd-json for simpler use cases. A few commenters appreciated the detailed explanation and clear benchmarks provided in the blog post, while acknowledging the specific niche this optimization targets.
Latacora's blog post "How (not) to sign a JSON object" cautions against signing JSON by stringifying it before applying a signature. This approach is vulnerable to attacks that modify whitespace or key ordering, which changes the string representation without altering the JSON's semantic meaning. The correct method involves canonicalizing the JSON object first – transforming it into a standardized, consistent byte representation – before signing. This ensures the signature validates only identical JSON objects, regardless of superficial formatting differences. The post uses examples to demonstrate the vulnerabilities of naive stringification and advocates using established JSON Canonicalization Schemes (JCS) for robust and secure signing.
HN commenters largely agree with the author's points about the complexities and pitfalls of signing JSON objects. Several highlighted the importance of canonicalization before signing, with some mentioning specific libraries like JWS and json-canonicalize to ensure consistent formatting. The discussion also touches upon alternatives like JWT (JSON Web Tokens) and COSE (CBOR Object Signing and Encryption) as potentially better solutions, particularly JWT for its ease of use in web contexts. Some commenters delve into the nuances of JSON's flexibility, which can make secure signing difficult, such as varying key order and whitespace handling. A few also caution against rolling your own cryptographic solutions and advocate for using established libraries where possible.
The blog post details the reverse engineering process of Apple's proprietary Typed Stream format used in various macOS features like Spotlight search indexing and QuickLook previews. The author, motivated by the lack of public documentation, utilizes a combination of tools and techniques including analyzing generated Typed Stream files, using class-dump on relevant system frameworks, and examining open-source components like CoreFoundation, to decipher the format. They ultimately discover that Typed Streams are essentially serialized property lists with a specific header and optional compression, allowing for efficient storage and retrieval of typed data. This reverse engineering effort provides valuable insight into the inner workings of macOS and potentially enables interoperability with other systems.
HN users generally praised the author's reverse-engineering effort, calling it "impressive" and "well-documented." Some discussed the implications of Apple using a custom format, speculating about potential performance benefits or tighter integration with their hardware. One commenter noted the similarity to Google's Protocol Buffers, suggesting Apple might have chosen this route to avoid dependencies. Others pointed out the difficulty in reverse-engineering these formats, highlighting the value of such work for interoperability. A few users discussed potential use cases for the information, including debugging and data recovery. Some also questioned the long-term viability of relying on undocumented formats.
Sparrow is a new C++ library designed for efficiently working with the Apache Arrow columnar format. It prioritizes compile times and runtime performance by minimizing dependencies and utilizing modern C++ features like compile-time reflection. Sparrow offers zero-copy reads and writes, enabling high-throughput data processing. It differs from other Arrow C++ implementations by focusing on a minimal and performant core, intentionally omitting features like computation kernels to reduce complexity and compile times. This approach aims to make Sparrow a building block for higher-level libraries and applications that require efficient data manipulation based on the Arrow format.
Hacker News users generally expressed enthusiasm for Sparrow's performance improvements over Apache Arrow's C++ implementation. Several commenters highlighted the importance of memory management and zero-copy operations in achieving these gains. Some discussed the potential benefits for data-intensive applications and integration with other libraries like Pandas. One commenter raised a question about SIMD utilization, while others praised the project's clear benchmarks and documentation. Several users expressed interest in contributing to or experimenting with Sparrow. A few comments also touched on the broader implications for C++ development and the evolution of data processing frameworks.
Keon is a new serialization/deserialization (serde) format designed for human readability and writability, drawing heavy inspiration from Rust's syntax. It aims to be a simple and efficient alternative to formats like JSON and TOML, offering features like strongly typed data structures, enums, and tagged unions. Keon emphasizes being easy to learn and use, particularly for those familiar with Rust, and focuses on providing a compact and clear representation of data. The project is actively being developed and explores potential use cases like configuration files, data exchange, and data persistence.
Hacker News users discuss KEON, a human-readable serialization format resembling Rust. Several commenters express interest, praising its readability and potential as a configuration language. Some compare it favorably to TOML and JSON, highlighting its expressiveness and Rust-like syntax. Concerns arise regarding its verbosity compared to more established formats, particularly for simple data structures, and the potential niche appeal due to the Rust syntax. A few suggest potential improvements, including a more formal specification, tools for generating parsers in other languages, and exploring the benefits over existing formats like Serde. The overall sentiment leans towards cautious optimism, acknowledging the project's potential but questioning its practical advantages and broader adoption prospects.
Summary of Comments ( 38 )
https://news.ycombinator.com/item?id=43508061
HN commenters generally praised Hexi for its simplicity and ease of use, particularly its header-only nature and intuitive syntax. Some compared it favorably to other serialization libraries like Protobuf and Cap'n Proto, highlighting its potential for better performance in certain scenarios due to its zero-copy deserialization. Concerns were raised about potential compile-time impact due to the header-only design and the lack of documentation beyond basic examples. One commenter suggested incorporating compile-time reflection to further enhance the library's capabilities and reduce boilerplate. Others questioned the long-term viability of the project, expressing a desire to see more real-world use cases and benchmarking data. The lack of support for optional fields was also mentioned as a potential drawback.
The Hacker News post about Hexi, a header-only network binary serialization library for C++, generated several comments discussing its merits and drawbacks compared to existing solutions.
One commenter expressed skepticism about the value proposition of Hexi, questioning the need for yet another serialization library in C++. They pointed out the maturity and wide adoption of Protobuf and Cap'n Proto, suggesting that unless Hexi offered significant performance or usability advantages, it would struggle to gain traction. This commenter also highlighted the importance of schema evolution in real-world applications and inquired about Hexi's capabilities in this area.
Another user echoed this sentiment, mentioning FlatBuffers and Cereal as additional alternatives already available. They specifically mentioned the complexity of handling schema evolution and backward compatibility, implying that these are crucial considerations for any serialization library. They also raised the issue of handling untrusted input, emphasizing the importance of security and robust error handling when deserializing data from potentially malicious sources.
A different commenter focused on the potential benefits of Hexi's header-only nature, suggesting that it could simplify integration and reduce build times compared to libraries requiring separate compilation and linking steps. However, they also acknowledged that this advantage might be offset by increased compile times due to the inclusion of the entire library in every translation unit.
Another comment discussed the importance of zero-copy deserialization for performance-sensitive applications, asking whether Hexi supports this feature. Zero-copy deserialization allows data to be used directly from the serialized buffer without requiring a separate copying step, which can significantly improve efficiency.
Several commenters inquired about specific features and capabilities of Hexi, such as support for optional fields, default values, and different data types. They also discussed the library's API design and ease of use, comparing it to other serialization libraries.
One commenter provided a link to a benchmark comparing various serialization libraries, including Protobuf, Cap'n Proto, and FlatBuffers. This benchmark could be useful for evaluating Hexi's performance relative to its competitors.
Finally, the author of Hexi actively participated in the discussion, responding to questions and clarifying various aspects of the library's design and functionality. They addressed concerns about schema evolution, security, and performance, providing additional context and insights into the library's development. They also expressed openness to feedback and suggestions for improvement.