Stories with Tag hexadecimal

• Recommendations for designing magic numbers of binary file formats

  permalink

  Posted: 2025-03-14 20:05:53

  Verbose tl;dr

  To minimize the risks of file format ambiguity, choose magic numbers for binary files that are uncommon and easily distinguishable. Favor longer magic numbers (at least 4 bytes) and incorporate asymmetry and randomness while avoiding printable ASCII characters. Consider including a version number within the magic to facilitate future evolution and potentially embedding the magic at both the beginning and end of the file for enhanced validation. This approach helps differentiate your file format from existing ones, reducing the likelihood of misidentification and improving long-term compatibility.

  The post "Recommendations for designing magic numbers of binary file formats" discusses best practices for choosing and implementing magic numbers—the identifying byte sequences at the beginning of files that signal their type. The author emphasizes the importance of carefully selecting these magic numbers to minimize the risk of misidentification, ensuring robust and reliable software behavior.

  The core recommendation revolves around incorporating human-readable ASCII characters within the magic number. This strategy makes it easier for developers and users to recognize the file type when inspecting the file's raw bytes, aiding in debugging and preventing accidental misinterpretation. This human-readable component should ideally be unique and relevant to the file format's purpose, clearly indicating its nature. The author suggests using a relevant abbreviation or acronym related to the file format, converted into ASCII characters.

  Beyond the human-readable aspect, the author advises including non-ASCII bytes within the magic number to further reduce the chance of collision with other file formats or random data sequences. These non-printable characters increase the entropy of the magic number, making it more statistically distinct. The specific recommended non-ASCII bytes are 0x00 (null byte) and bytes with values above 0x7F (the highest ASCII value). These particular choices minimize the likelihood of accidental matches with common text files or other structured data.

  Furthermore, the author recommends using a magic number of at least four bytes in length. This length provides a good balance between robust identification and minimizing overhead. Longer magic numbers offer stronger guarantees against collisions but can slightly increase processing time. Four bytes are generally considered a sweet spot, providing sufficient uniqueness without undue burden.

  Finally, the post briefly touches on the practical implementation. It advises checking the entire magic number sequence before definitively identifying a file, avoiding partial matches that could lead to false positives. This rigorous checking ensures reliable file type identification, even in the presence of corrupted or incomplete data. In summary, the post provides a clear and concise set of guidelines for designing robust and easily identifiable magic numbers, advocating for a blend of human-readable ASCII and distinguishing non-ASCII bytes for optimal file format identification.

  • file formats
  • binary files
  • magic numbers
  • file signatures
  • design
  • recommendations
  • Software Development
  • programming
  • data structures
  • file identification
  • hexadecimal
  • Reverse Engineering

  Summary of Comments ( 18 )
  https://news.ycombinator.com/item?id=43366671

  Verbose tl;dr

  HN users discussed various strategies for handling magic numbers in binary file formats. Several commenters emphasized using longer, more unique magic numbers to minimize the chance of collisions with other file types. Suggestions included incorporating version numbers, checksums, or even reserved bytes within the magic number sequence. The use of human-readable ASCII characters within the magic number was debated, with some advocating for it for easier identification in hex dumps, while others prioritized maximizing entropy for more robust collision resistance. Using an initial "container" format with metadata and a secondary magic number for the embedded data was also proposed as a way to handle versioning and complex file structures. Finally, the discussion touched on the importance of registering new magic numbers to avoid conflicts and the practical reality that collisions can often be resolved contextually, even with shorter magic numbers.

  The Hacker News post "Recommendations for designing magic numbers of binary file formats" sparked a discussion with several insightful comments focusing on practicality and real-world considerations when choosing magic numbers for file formats.

  One of the most compelling comments highlights the importance of considering the encoding of the file when choosing a magic number. Specifically, it points out that using a UTF-8 BOM (Byte Order Mark) as a magic number can be problematic because it's valid UTF-8 and might appear within the data itself. This could lead to false positives when trying to identify the file type. The commenter suggests prioritizing human readability over relying solely on a BOM and proposes incorporating version numbers within the magic number for better future compatibility.

  Another commenter expands on this idea by recommending a hybrid approach, combining a short magic number with a separate version field shortly thereafter. This approach balances quick identification with the ability to handle future format revisions. They further suggest using ASCII characters for the magic number to ensure straightforward identification and avoid encoding issues.

  Several comments delve into the practical challenges of dealing with corrupted or truncated files. One user suggests incorporating checksums or other integrity checks alongside the magic number to avoid misinterpreting partial files. This preventative measure adds an extra layer of confidence when working with potentially damaged data.

  Adding to the discussion of human readability, one commenter underscores its importance, especially for debugging. Being able to quickly recognize a file type by looking at its first few bytes in a hex editor can significantly speed up the debugging process. They suggest using memorable ASCII strings that clearly indicate the file's purpose.

  Finally, a commenter reflects on the historical context of magic numbers, recalling how they were used in older systems for quick identification. They mention that, despite advancements in file systems, magic numbers still hold relevance, especially for low-level tools and when dealing with data from diverse sources. This historical perspective provides a valuable reminder of the enduring utility of magic numbers.

  The overall sentiment in the comments leans toward practicality and robustness. The discussion emphasizes the need for clear, human-readable magic numbers, combined with versioning and integrity checks to ensure reliable file identification even in less-than-ideal circumstances.

• Bypass DeepSeek censorship by speaking in hex

  permalink

  Posted: 2025-01-31 19:41:49

  Verbose tl;dr

  The Substack post details how DeepSeek, a video search engine with content filtering, can be circumvented by encoding potentially censored keywords as hexadecimal strings. Because DeepSeek decodes hex before applying its filters, a search for "0x736578" (hex for "sex") will return results that a direct search for "sex" might block. The post argues this reveals a flaw in DeepSeek's censorship implementation, demonstrating that filtering based purely on keyword matching is easily bypassed with simple encoding techniques. This highlights the limitations of automated content moderation and the potential for unintended consequences when relying on simplistic filtering methods.

  The Substack post titled "Bypass DeepSeek censorship by speaking in hex" elaborates on a potential method to circumvent content moderation systems, specifically referencing a hypothetical system named "DeepSeek." The author posits that these advanced censorship mechanisms, which likely employ sophisticated natural language processing and machine learning algorithms, can be tricked by encoding messages in hexadecimal format. The core argument revolves around the idea that while these systems are adept at understanding and flagging problematic phrases or keywords in plain text, they might not be equipped to interpret the same information when presented as a string of hexadecimal characters. Essentially, the hexadecimal representation acts as a camouflage, obscuring the true meaning of the message from the censorship algorithm.

  The author suggests that a user could encode their desired message into hexadecimal, transmit this encoded message, and then rely on the recipient to decode it back into readable text on their end. This process effectively bypasses the content moderation system because the system only "sees" the hexadecimal string, which, in and of itself, is not inherently problematic. The true meaning of the message remains hidden until it is decoded by the intended recipient.

  The post further illustrates this concept with a specific example, encoding the phrase "DeepSeek is watching you" into its hexadecimal equivalent. This example serves to demonstrate the practical application of the proposed method. While the author acknowledges the potential limitations and the possibility that sufficiently advanced systems might eventually adapt to this tactic, they present it as a currently viable workaround for bypassing content moderation. The implication is that this method adds a layer of obfuscation that could be sufficient to evade detection, at least temporarily.

  • Censorship
  • circumvention
  • deepseek
  • hexadecimal
  • Encoding
  • decoding
  • online censorship
  • Free Speech
  • internet freedom
  • privacy
  • Security
  • text encoding
  • digital rights
  • information access
  • Data Encoding

  Summary of Comments ( 320 )
  https://news.ycombinator.com/item?id=42891042

  Verbose tl;dr

  Hacker News users discuss potential censorship evasion techniques, prompted by an article detailing how DeepSeek, a coder-focused search engine, appears to suppress results related to specific topics. Several commenters explore the idea of encoding sensitive queries in hexadecimal format as a workaround. However, skepticism arises regarding the long-term effectiveness of such a tactic, predicting that DeepSeek would likely adapt and detect such encoding methods. The discussion also touches upon the broader implications of censorship in code search engines, with some arguing that DeepSeek's approach might hinder access to valuable information while others emphasize the platform's right to curate its content. The efficacy and ethics of censorship are debated, with no clear consensus emerging. A few comments delve into alternative evasion strategies and the general limitations of censorship in a determined community.

  The Hacker News post titled "Bypass DeepSeek censorship by speaking in hex" with the ID 42891042 has several comments discussing the practicality and implications of bypassing censorship using hexadecimal representation of text.

  Several commenters point out that this method is not a robust solution for bypassing censorship. They argue that any sophisticated censorship system would easily detect and block such obvious encoding. One commenter specifically mentions that converting to hex is a trivial transformation and easily reversible, making it a poor choice for evading censorship. This sentiment is echoed by others who suggest that such a simple encoding would be quickly identified and added to the censorship criteria.

  Another line of discussion revolves around the concept of security through obscurity. Commenters debate whether this method could be considered a form of security through obscurity, and generally agree that it is. They highlight the weakness of such an approach, emphasizing that relying on the censor's ignorance of a simple encoding is not a reliable strategy.

  The discussion also touches upon the broader implications of censorship and the cat-and-mouse game between censors and those trying to circumvent them. One commenter suggests that this highlights the futility of trying to censor information in the digital age, as new methods of bypassing restrictions will continually emerge.

  Some commenters explore alternative, more robust methods of bypassing censorship, such as using strong encryption or steganography. They point out that these techniques are significantly more difficult to detect and block than simple hex encoding.

  A few comments delve into the technical aspects of encoding and decoding hexadecimal strings, including mentioning specific programming languages and libraries that can be used for this purpose.

  Finally, some comments express a degree of amusement at the simplicity of the proposed method, with one commenter ironically suggesting speaking in binary as an even more "secure" alternative. This underscores the general consensus that while encoding text in hex might be a clever workaround in a very limited context, it is not a practical or reliable solution for bypassing sophisticated censorship mechanisms.