Stories with Tag Regression

• When eBPF pt_regs reads return garbage on the latest Linux kernels, blame Fred

  permalink

  Posted: 2025-03-01 01:37:26

  Verbose tl;dr

  A recent Linux kernel change inadvertently broke eBPF programs relying on PT_REGS_RC(regs). Intended to optimize register access for x86, this change accidentally cleared the return value register before eBPF programs using kprobe and kretprobe could access it. This resulted in eBPF tools like bpftrace and bcc showing garbage data instead of expected return values. The issue primarily affects x86 systems running kernel versions 6.5 and later and has already been fixed in 6.5.1, 6.4.12, and 6.1.38. Users of affected kernels should update to receive the fix.

  Tanel Poder's blog post, "When eBPF pt_regs reads return garbage on the latest Linux kernels, blame Fred," discusses a perplexing issue encountered while using extended Berkeley Packet Filter (eBPF) programs to trace system calls on recent Linux kernels. The problem manifested as seemingly random garbage data being read from the pt_regs structure, which holds CPU register values at the time of a system call. This structure is crucial for eBPF programs to understand the context of the call and access arguments passed to it.

  Poder meticulously details his troubleshooting process, beginning with the observation of inconsistent data when attempting to read the system call number from pt_regs->ax. He suspected a kernel bug, initially focusing on potential issues with the relatively new instruction pointer value caching mechanism introduced to enhance performance. To isolate the problem, Poder employed several debugging techniques, including:

  • kprobe tracing: He used kprobes, another kernel tracing facility, to directly examine the contents of pt_regs inside the kernel, confirming the corruption wasn't occurring within the eBPF program itself but rather in the data being provided to it.
  • Kernel debugging with printk: He added print statements within the kernel code to track the values of pt_regs at various points, helping him pinpoint the location where the corruption occurred.
  • Examining kernel source code: Poder delved into the kernel source code, meticulously tracing the flow of execution related to system call entry and the handling of pt_regs, ultimately identifying a suspicious code path.

  His investigation ultimately revealed that the culprit wasn't the instruction pointer caching but rather a seemingly innocuous optimization introduced by a developer named "Fred." This optimization involved reusing a stack variable previously used for the system call number within the __sysvec_tail function, which is part of the system call handling logic. This reuse inadvertently corrupted the pt_regs structure because the stack variable was not properly cleared or reinitialized before being reused for a different purpose.

  The consequence of this optimization was that the original system call number within pt_regs was overwritten, leading to the "garbage" data observed by Poder. He explains that this issue was particularly tricky to identify due to its timing sensitivity and dependency on the specific path taken through the optimized code. The problem didn't always manifest, making it appear intermittent and further complicating the debugging process.

  The post concludes with Poder highlighting the importance of thorough testing, even for seemingly minor optimizations, and emphasizes the complexity of modern kernel development. He also notes the value of persistent debugging and the use of various tools and techniques to pinpoint the root cause of elusive bugs. He applauds the responsiveness of the kernel developers, who acknowledged and swiftly addressed the issue once identified.

  • eBPF
  • pt_regs
  • Linux
  • Kernel
  • Debugging
  • Tracing
  • Performance Analysis
  • software error
  • Regression
  • Fred
  • BCC
  • BPF Compiler Collection

  Summary of Comments ( 9 )
  https://news.ycombinator.com/item?id=43214576

  Verbose tl;dr

  The Hacker News comments discuss the complexities and nuances of the issue presented in the article about pt_regs returning garbage in recent Linux kernels due to changes introduced by "Fred." Several commenters express sympathy for Fred, highlighting the challenging trade-offs inherent in kernel development, especially when balancing performance optimizations with backward compatibility. Some point out the difficulties of maintaining eBPF programs across kernel versions and the lack of clear documentation or warnings about these breaking changes. Others delve into the technical specifics, discussing register context, stack unwinding, and the implications for debuggers and profiling tools. The overall sentiment seems to be one of acknowledging the difficulty of the situation and the need for better communication and tooling to navigate such kernel-level changes. A few users also suggest potential workarounds and debugging strategies.

  The Hacker News post titled "When eBPF pt_regs reads return garbage on the latest Linux kernels, blame Fred" has generated a moderate number of comments, most of which delve into the technical details of the issue and offer further insights or related experiences.

  Several commenters discuss the complexities of the pt_regs structure and its usage within the eBPF (extended Berkeley Packet Filter) context. One user highlights the inherent fragility of relying on the layout of pt_regs, as it is architecture-specific and subject to change. They point out that accessing pt_regs directly from eBPF programs is essentially working with a "private, unstable ABI" and that a more robust solution would involve explicitly passing the needed register values to the eBPF program. This echoes the sentiment expressed in the original article about the need for a more stable interface for eBPF programs to access register data.

  Another comment chain focuses on the challenges of maintaining compatibility in the Linux kernel, especially when dealing with low-level structures like pt_regs. One commenter mentions the difficulty of keeping track of all the implicit dependencies and the potential for unintended side effects when making changes to core kernel components. They express sympathy for the developers involved, acknowledging the difficulty of balancing performance optimization with maintaining stable ABIs.

  A couple of commenters share their own experiences with similar issues related to kernel updates and ABI compatibility. One recounts a story of encountering unexpected behavior after a kernel upgrade, which ultimately traced back to changes in internal kernel structures. This anecdote reinforces the point about the inherent risks associated with relying on undocumented or unstable interfaces.

  One commenter questions the use of "blame" in the title, suggesting that it is perhaps too strong a word, given that the change was likely unintentional and a consequence of complex system interactions. They advocate for a more understanding approach, acknowledging the difficulty of maintaining such a large and intricate project as the Linux kernel.

  The comments also touch upon related topics such as the use of kernel tracing tools, the benefits and drawbacks of eBPF technology, and the trade-offs between performance and stability. While not directly related to the core issue, these comments provide additional context and enrich the discussion.

  Overall, the comments on Hacker News provide valuable insights into the complexities of kernel development, the challenges of maintaining ABI compatibility, and the delicate balance between performance and stability. They also offer practical advice for developers working with eBPF and highlight the importance of using stable interfaces whenever possible.

• A Clang regression related to switch statements and inlining

  permalink

  Posted: 2025-02-18 12:38:08

  Verbose tl;dr

  A recent Clang optimization introduced in version 17 regressed performance when compiling code containing large switch statements within inlined functions. This regression manifested as significantly increased compile times, sometimes by orders of magnitude, and occasionally resulted in internal compiler errors. The issue stems from Clang's attempt to optimize switch lowering by transforming it into a series of conditional moves based on jump tables. This optimization, while beneficial in some cases, interacts poorly with inlining, exploding the complexity of the generated intermediate representation (IR) when a function with a large switch is inlined multiple times. This ultimately overwhelms the compiler's later optimization passes. A workaround involves disabling the problematic optimization via a compiler flag (-mllvm -switch-to-lookup-table-threshold=0) until a proper fix is implemented in a future Clang release.

  This blog post details a performance regression discovered by the author, Adrian Nicula, in Clang versions 15 and 16 concerning the compilation of C++ code containing large switch statements within inline functions. The issue arises specifically when these switch statements are located inside inline functions that are called repeatedly within a hot loop. Prior to Clang 15, the compiler effectively optimized these scenarios, resulting in efficient code execution. However, in Clang 15 and 16, the optimization strategy changed, leading to a significant performance degradation in specific circumstances.

  The core problem stems from how Clang handles jump tables, a common optimization technique for switch statements. Previously, when an inline function with a large switch was called repeatedly, Clang would generate a single jump table for the switch statement and reuse it across all call sites. This approach minimized code size and maximized performance.

  Beginning with Clang 15, the compiler seemingly changed its inlining heuristics. Instead of creating a single shared jump table, Clang now generates a separate jump table for each instance of the inlined function within the loop. This duplication significantly increases the code size, particularly for large switch statements with numerous cases. The larger code size negatively impacts instruction cache performance, leading to the observed performance regression.

  Nicula demonstrates the issue with a concise example involving a benchmarking program that measures the execution time of code containing a large switch statement within an inline function. He provides performance measurements across different Clang versions, clearly showing the performance drop in versions 15 and 16. The benchmark also highlights that the issue only manifests when the inline function is called a substantial number of times within a loop.

  The author further investigates the generated assembly code, confirming the proliferation of jump tables in Clang 15 and 16 compared to earlier versions. This analysis solidifies the hypothesis that the change in jump table generation is the root cause of the performance problem.

  While Nicula did not pinpoint the exact commit introducing this regression, he suspects it may be related to modifications in Clang's inlining or jump table generation logic around the time of Clang 15's release. The author concludes by recommending users experiencing similar performance issues to revert to Clang 14 or explore compiler flags related to inlining and optimization to potentially mitigate the problem. He also expresses hope that the Clang community will address this regression in future releases.

  • Clang
  • compiler
  • C++
  • C
  • Regression
  • Switch Statement
  • Inlining
  • optimization
  • Code Generation
  • LLVM
  • bug
  • performance

  Summary of Comments ( 2 )
  https://news.ycombinator.com/item?id=43088797

  Verbose tl;dr

  The Hacker News comments discuss a performance regression in Clang involving large switch statements and inlining. Several commenters confirm experiencing similar issues, particularly when compiling large codebases. Some suggest the regression might be related to changes in the inlining heuristics or the way Clang handles jump tables. One commenter points out that using a constexpr hash table for large switches can be a faster alternative. Another suggests profiling and selective inlining as a workaround. The lack of clear identification of the root cause and the potential impact on compile times and performance are highlighted as concerning. Some users express frustration with the frequency of such regressions in Clang.

  The Hacker News post discussing the Clang regression related to switch statements and inlining sparked a conversation revolving primarily around compiler optimization, code generation, and debugging challenges. Several commenters delved into the technical intricacies of the issue.

  One commenter highlighted the complexities involved in compiler optimization, specifically mentioning the difficulty in striking a balance between performance gains and potential code bloat. They pointed out that aggressive inlining, while often beneficial, can sometimes lead to larger binaries and potentially slower execution in certain scenarios, as was seemingly the case with the Clang regression described in the article. This commenter also touched upon the trade-offs compilers must make and how these decisions can sometimes have unforeseen consequences.

  Another commenter focused on the debugging challenges introduced by such optimizations. They argued that overly aggressive inlining can obscure the relationship between the original source code and the generated assembly, making it harder to debug issues. This difficulty stems from the fact that the inlined code is effectively "merged" into the calling function, making it harder to trace back to the original source location when stepping through a debugger.

  The discussion also touched upon the specifics of switch statement optimization. One commenter explained how compilers often transform switch statements into various forms, such as jump tables or binary search trees, depending on the density and distribution of the cases. They suggested that the Clang regression might be related to a suboptimal choice of switch implementation in specific scenarios.

  Furthermore, a commenter mentioned the importance of profiling and benchmarking in identifying and addressing such performance regressions. They emphasized that relying solely on theoretical analysis of code transformations can be misleading and that empirical data is crucial for understanding the actual impact of compiler optimizations.

  Finally, some commenters discussed potential workarounds and suggested exploring compiler flags to fine-tune inlining behavior or to disable specific optimizations. This highlighted the importance of having granular control over the compiler's optimization strategies to mitigate potential performance issues.

  Overall, the comments on Hacker News provided valuable insights into the technical nuances of the Clang regression, focusing on the challenges related to compiler optimization, debugging, and the importance of profiling and benchmarking. The discussion demonstrated a deep understanding of compiler internals and offered practical suggestions for dealing with similar issues.