Stories with Tag Content Extraction

• Show HN: Defuddle, an HTML-to-Markdown alternative to Readability

  permalink

  Posted: 2025-05-22 21:40:54

  Verbose tl;dr

  Defuddle is an open-source command-line tool that converts HTML to Markdown, aiming to be a simpler and more robust alternative to Readability. It focuses on extracting the main content from web pages while preserving basic formatting like headings, lists, and code blocks, outputting clean Markdown suitable for archiving, note-taking, or further processing. Unlike Readability, which primarily targets article-like content, Defuddle attempts to handle a wider variety of HTML structures. It's written in Go and prioritizes speed and predictable output.

  Introducing Defuddle, a novel command-line tool presented on Hacker News as an alternative to Readability for converting HTML content into Markdown. Unlike Readability, which focuses on extracting the main readable content of a webpage for a cleaner reading experience, Defuddle prioritizes faithfully reproducing the structure and formatting of the original HTML document in Markdown format. This makes it particularly suitable for archiving web pages and preserving their original layout, as close to the original HTML as possible within the constraints of Markdown.

  Defuddle is written in Go and leverages the power of the Goldmark Markdown parser. It operates by parsing the provided HTML input and then systematically transforming it into Markdown elements. This includes converting HTML headings (h1, h2, etc.) into their Markdown equivalents (#, ##, etc.), paragraphs into Markdown paragraphs, lists (ordered and unordered) into their Markdown counterparts, and links into Markdown link syntax. The tool aims to handle a wide range of HTML elements and attributes, striving to retain the original document's structure and semantic meaning within the Markdown output.

  While Readability excels at creating a distilled reading experience by removing clutter and focusing on core content, Defuddle fills a different niche. Its primary objective is not readability optimization, but rather accurate HTML-to-Markdown conversion for purposes such as archiving, documentation, or any situation where preserving the original document's structure is paramount. This approach offers a distinct advantage for users who need a reliable method to convert HTML to Markdown while maintaining the original formatting as accurately as possible, offering a more comprehensive representation of the source material than a readability-focused tool.

  • HTML to Markdown
  • Markdown Converter
  • Readability Alternative
  • web scraping
  • Content Extraction
  • HTML Parsing
  • text processing
  • Defuddle
  • Open Source
  • GitHub
  • Python
  • command-line tool
  • cli

  Summary of Comments ( 55 )
  https://news.ycombinator.com/item?id=44067409

  Verbose tl;dr

  HN commenters generally praised Defuddle for its simplicity and effectiveness in converting HTML to Markdown, particularly for archiving web pages. Several appreciated its focus on content extraction over perfect formatting, finding the resulting Markdown more usable. Some suggested improvements like better image handling, code block formatting, and handling of certain HTML elements. One commenter highlighted its usefulness for researchers and academics, while others compared it favorably to other similar tools, noting Defuddle's speed and accuracy. The project's open-source nature and reliance on a single Go binary were also lauded.

  The Hacker News post about "Defuddle, an HTML-to-Markdown alternative to Readability" generated a moderate number of comments, mostly focused on comparing Defuddle to existing tools, discussing potential use cases, and exploring technical aspects.

  Several commenters compared Defuddle to Readability, noting that while Readability aims to create a clean reading experience, Defuddle focuses on preserving the original structure and converting it to Markdown. This distinction was highlighted as potentially useful for archiving web pages and making them easily editable. One user specifically mentioned preferring Markdown over the output of Readability for archiving purposes.

  The discussion also touched upon alternative tools like pandoc and its limitations with complex HTML. Some commenters suggested that Defuddle might be a better choice for certain websites where pandoc struggles. Another user proposed combining lynx (a text-based web browser) with pandoc as a potential alternative workflow.

  The technical implementation of Defuddle was also a topic of interest. One commenter inquired about the choice of Python over Javascript for the project, to which the author (kepano) responded by explaining their preference for Python's ecosystem and the availability of robust HTML parsing libraries. The author also highlighted their choice of Beautiful Soup 4 for HTML parsing and addressed questions regarding the handling of specific elements like <pre> tags and code blocks.

  One commenter explored the potential use case of integrating Defuddle into a note-taking workflow, envisioning a scenario where web content could be easily converted to Markdown and incorporated into notes. They also suggested exploring the use of Readability's API to improve the cleaning process, while acknowledging potential cost implications.

  Finally, some users shared their positive experiences with Defuddle, praising its simplicity and effectiveness. One commenter even reported successful usage on a challenging website where other tools had failed.

  In summary, the comments section offered a valuable discussion around Defuddle, comparing it to existing tools, exploring its potential uses, and delving into some of its technical aspects. The comments generally highlighted the potential of Defuddle as a useful tool for converting HTML to Markdown, especially for archiving and editing web content.

• Extracting content from an LCP "protected" ePub

  permalink

  Posted: 2025-03-16 12:50:56

  Verbose tl;dr

  The blog post explores methods to extract content from an LCP-protected ePub file, primarily for archiving or personal use. It details the challenges posed by LCP's encryption and license validation, and walks through a technical process involving inspecting the ePub's structure, locating the encrypted content, and ultimately decrypting it using the user's own credentials and a modified version of Adobe's Digital Editions library. The author emphasizes this is for educational purposes only and discourages any copyright infringement. While acknowledging potential legal and ethical concerns, the post frames the process as a way to reclaim control over purchased digital content and ensure future accessibility.

  This blog post by Keith Packard details his exploration into decrypting an ePub protected by the Readium LCP (Library Content Protection) Digital Rights Management (DRM) scheme. He begins by outlining his motivations, driven by the desire to access his legally purchased ebooks on any device without vendor lock-in, emphasizing the importance of interoperability and the right to personal use of purchased digital content.

  Packard meticulously describes his step-by-step process. He initiates his investigation by examining the structure of the protected ePub, which is essentially a ZIP archive. He notes the presence of an META-INF/license.lcpl file, indicative of LCP protection, and a .lcpenc file extension on the encrypted XHTML content files within the ePub. He then delves into analyzing the license.lcpl file, which is an XML document containing the license information and a crucial encrypted key. He identifies the encryption algorithm used as AES-128-CBC and highlights the presence of a user key, content key, and a hint to derive the latter from the former.

  The post goes on to describe how Packard reverse-engineered the decryption process. He leverages the open-source Readium LCP client code, specifically focusing on a JavaScript implementation. By carefully examining the JavaScript functions, particularly those involved in key derivation and decryption, he manages to understand the necessary cryptographic operations. He explains his understanding of how the user key, obtained from the license acquisition process, is used to derive the content key via a key derivation function (KDF). He further explains how this content key is subsequently used to decrypt the individual XHTML files comprising the ePub’s content.

  While acknowledging successful decryption of the content key, Packard emphasizes that his work is not yet complete. The post concludes by stating that he has yet to implement the actual decryption of the XHTML content files, leaving this as a future step. He also mentions the need to understand how to acquire the user key legitimately through a proper LCP license server interaction, which he has not yet explored in this particular post. He expresses optimism about the feasibility of achieving full decryption and emphasizes his commitment to documenting his findings for educational purposes and to advocate for open access to legally purchased content.

  • EPUB
  • LCP
  • digital rights management
  • DRM
  • Content Extraction
  • Ebook
  • Readium LCP
  • Digital Content Protection

  Summary of Comments ( 3 )
  https://news.ycombinator.com/item?id=43378627

  Verbose tl;dr

  HN commenters generally express skepticism towards the robustness of LCP "protection," viewing it as a minor speedbump rather than a genuine barrier. Several point out that determined users can always access content through methods like disabling JavaScript or using developer tools. One commenter mentions DeDRM tools as an existing solution for bypassing such restrictions, while others suggest that the real protection lies in social pressure and legal consequences, not technical measures. The feasibility of converting ePubs to PDF and then extracting text is also discussed. Overall, the sentiment is that DRM ultimately harms accessibility and legitimate users more than pirates.

  The Hacker News post "Extracting content from an LCP "protected" ePub" has generated several comments discussing the effectiveness and ethics of LCP (Limited Content Protection) for ebooks.

  One commenter points out the inherent weakness of DRM, stating that if a device can render the content, it can be captured. They argue that DRM only inconveniences legitimate users while dedicated pirates will always find a way around it. This comment highlights the common sentiment that DRM is a futile effort in the long run.

  Another commenter dives into the technicalities, explaining that LCP isn't designed to prevent copying, but rather to associate a specific decryption key with a user or device. They describe how LCP uses a server to deliver keys, and how the described method intercepts this communication to obtain the key. This clarifies the actual function of LCP and how the exploit bypasses it.

  A further comment expands on the limitations of LCP, mentioning that it doesn't protect against screen scraping or even simply printing to PDF. This emphasizes the vulnerability of the protection scheme and how easily it can be circumvented by relatively simple methods.

  The discussion also touches on the legal aspects, with one commenter noting that circumventing DRM, even poorly implemented DRM like LCP, might still be illegal depending on jurisdiction. This provides a counterpoint to the purely technical discussion and highlights the legal ramifications of such actions.

  Another commenter criticizes Adobe's implementation of LCP, calling it "pathetic" and suggesting that it provides a false sense of security to publishers. They express frustration with the apparent lack of effort in implementing a robust protection scheme.

  Finally, some comments focus on the underlying issue of control versus access. One user argues that publishers fear losing control over their content, while another contends that what publishers really want is the ability to track and control access to the content. This highlights a philosophical difference in approaches to digital content distribution.

  In summary, the comments express a general skepticism towards LCP's effectiveness as a DRM solution. They highlight the inherent limitations of DRM, discuss the technical aspects of the exploit, and touch upon the legal and philosophical implications of digital content protection. Several commenters express frustration with the perceived weakness of LCP and the seeming futility of DRM in general.