The author dramatically improved the debug build speed of their C++ project, achieving up to 100x faster execution. The primary culprit was excessive logging, specifically the use of a logging library with a slow formatting implementation, exacerbated by unnecessary string formatting even when logs weren't being written. By switching to a faster logging library (spdlog), deferring string formatting until after log level checks, and optimizing other minor inefficiencies, they brought their debug build performance to a usable level, allowing for significantly faster iteration times during development.
Gaultier, the author, details their experience drastically improving the debug build performance of a C++ project. They begin by highlighting the common problem of slow debug builds, which often force developers to choose between faster but less informative release builds or slow but debuggable debug builds. This dilemma led to a situation where the debug build of their specific project, a language server for the Zig programming language (zls), was so slow it was practically unusable, taking up to 5 minutes to perform simple actions.
Gaultier's primary performance bottleneck stemmed from excessive logging in debug mode. The logging library, spdlog, while generally performant, was being used extensively, generating and formatting copious amounts of log messages. This intensive I/O operation significantly slowed down the build, especially considering the server's architecture, which relied on asynchronous operations and frequent context switching. Furthermore, the format string construction itself added overhead due to the way it interacted with variadic templates.
The author explored various avenues for optimization. Initially, they attempted to leverage the preprocessor to conditionally disable logging in release mode, but this proved insufficient as the format strings were still being constructed even if the logging itself was bypassed. They then experimented with a more nuanced preprocessor approach, using if constexpr to completely eliminate format string creation for disabled log levels. While this yielded some improvement, it wasn't enough to solve the core issue.
The breakthrough came from employing a more radical solution: completely removing spdlog from the debug build and replacing it with a minimal, custom logging solution. This bespoke logging system avoided format string construction entirely by directly outputting raw strings using std::cerr, which is unbuffered. This switch eliminated the overhead of formatting and significantly reduced the I/O burden.
The author opted to retain spdlog for release builds, allowing for configurable logging levels and formatted output in production. This strategy provided the best of both worlds: fast debug builds without logging overhead and flexible logging in release builds. The result of this optimization was a dramatic 100x speed improvement in the debug build, making it finally practical for development and debugging. The author concludes by emphasizing the importance of fast feedback cycles during development and how this optimization significantly enhanced their workflow. They also briefly discuss the limitations of their new debug logging system, such as the lack of log levels, acknowledging it as a trade-off for the achieved performance gain.
Summary of Comments ( 16 )
https://news.ycombinator.com/item?id=43087482
Commenters on Hacker News largely praised the author's approach to optimizing debug builds, emphasizing the significant impact build times have on developer productivity. Several highlighted the importance of the described techniques, like using link-time optimization (LTO) and profile-guided optimization (PGO) even in debug builds, challenging the common trade-off between debuggability and speed. Some shared similar experiences and alternative optimization strategies, such as using pre-compiled headers (PCH) and unity builds, or employing tools like ccache. A few also pointed out potential downsides, like increased memory usage with LTO, and the need to balance optimization with the ability to effectively debug. The overall sentiment was that the author's detailed breakdown offered valuable insights and practical solutions for a common developer pain point.
The Hacker News post "Making my debug build run 100x faster so that it is finally usable" generated a lively discussion with several compelling comments. Many commenters shared their own experiences and insights related to debug build performance.
A recurring theme was the importance of build optimization and the significant impact it can have on developer productivity. One commenter highlighted the frustration of slow debug builds, stating that it disrupts the flow of development and makes debugging a painful process. They praised the author of the original article for sharing their optimization techniques, emphasizing the value of such knowledge in the developer community.
Several commenters discussed specific strategies for improving debug build times. Suggestions included disabling link-time optimization (LTO), using pre-compiled headers (PCH), and minimizing the use of debug symbols. One commenter pointed out that the choice of build system can also significantly affect build times, with some systems being inherently faster than others. Another commenter shared their experience with incremental builds, noting that they can dramatically reduce build times when implemented correctly.
The discussion also touched upon the trade-offs between debug build speed and debugging capabilities. While faster builds are generally desirable, some commenters cautioned against sacrificing essential debugging information for the sake of speed. They argued that a balance must be struck between build performance and the ability to effectively debug code. One commenter suggested using different build configurations for different stages of development, with faster builds optimized for rapid iteration and slower, more comprehensive builds reserved for in-depth debugging.
Some commenters expressed skepticism about the author's claim of a 100x speedup, suggesting that such a dramatic improvement might be specific to the author's particular project or environment. They encouraged others to try the author's techniques and share their own results, emphasizing the importance of empirical evidence.
Overall, the comments on the Hacker News post reflect a shared concern among developers about the performance of debug builds and a desire for effective strategies to improve them. The discussion provided valuable insights into various optimization techniques and sparked a productive exchange of ideas and experiences.