Stories with Tag Tracing

• When eBPF pt_regs reads return garbage on the latest Linux kernels, blame Fred

  permalink

  Posted: 2025-03-01 01:37:26

  Verbose tl;dr

  A recent Linux kernel change inadvertently broke eBPF programs relying on PT_REGS_RC(regs). Intended to optimize register access for x86, this change accidentally cleared the return value register before eBPF programs using kprobe and kretprobe could access it. This resulted in eBPF tools like bpftrace and bcc showing garbage data instead of expected return values. The issue primarily affects x86 systems running kernel versions 6.5 and later and has already been fixed in 6.5.1, 6.4.12, and 6.1.38. Users of affected kernels should update to receive the fix.

  Tanel Poder's blog post, "When eBPF pt_regs reads return garbage on the latest Linux kernels, blame Fred," discusses a perplexing issue encountered while using extended Berkeley Packet Filter (eBPF) programs to trace system calls on recent Linux kernels. The problem manifested as seemingly random garbage data being read from the pt_regs structure, which holds CPU register values at the time of a system call. This structure is crucial for eBPF programs to understand the context of the call and access arguments passed to it.

  Poder meticulously details his troubleshooting process, beginning with the observation of inconsistent data when attempting to read the system call number from pt_regs->ax. He suspected a kernel bug, initially focusing on potential issues with the relatively new instruction pointer value caching mechanism introduced to enhance performance. To isolate the problem, Poder employed several debugging techniques, including:

  • kprobe tracing: He used kprobes, another kernel tracing facility, to directly examine the contents of pt_regs inside the kernel, confirming the corruption wasn't occurring within the eBPF program itself but rather in the data being provided to it.
  • Kernel debugging with printk: He added print statements within the kernel code to track the values of pt_regs at various points, helping him pinpoint the location where the corruption occurred.
  • Examining kernel source code: Poder delved into the kernel source code, meticulously tracing the flow of execution related to system call entry and the handling of pt_regs, ultimately identifying a suspicious code path.

  His investigation ultimately revealed that the culprit wasn't the instruction pointer caching but rather a seemingly innocuous optimization introduced by a developer named "Fred." This optimization involved reusing a stack variable previously used for the system call number within the __sysvec_tail function, which is part of the system call handling logic. This reuse inadvertently corrupted the pt_regs structure because the stack variable was not properly cleared or reinitialized before being reused for a different purpose.

  The consequence of this optimization was that the original system call number within pt_regs was overwritten, leading to the "garbage" data observed by Poder. He explains that this issue was particularly tricky to identify due to its timing sensitivity and dependency on the specific path taken through the optimized code. The problem didn't always manifest, making it appear intermittent and further complicating the debugging process.

  The post concludes with Poder highlighting the importance of thorough testing, even for seemingly minor optimizations, and emphasizes the complexity of modern kernel development. He also notes the value of persistent debugging and the use of various tools and techniques to pinpoint the root cause of elusive bugs. He applauds the responsiveness of the kernel developers, who acknowledged and swiftly addressed the issue once identified.

  • eBPF
  • pt_regs
  • Linux
  • Kernel
  • Debugging
  • Tracing
  • Performance Analysis
  • software error
  • Regression
  • Fred
  • BCC
  • BPF Compiler Collection

  Summary of Comments ( 9 )
  https://news.ycombinator.com/item?id=43214576

  Verbose tl;dr

  The Hacker News comments discuss the complexities and nuances of the issue presented in the article about pt_regs returning garbage in recent Linux kernels due to changes introduced by "Fred." Several commenters express sympathy for Fred, highlighting the challenging trade-offs inherent in kernel development, especially when balancing performance optimizations with backward compatibility. Some point out the difficulties of maintaining eBPF programs across kernel versions and the lack of clear documentation or warnings about these breaking changes. Others delve into the technical specifics, discussing register context, stack unwinding, and the implications for debuggers and profiling tools. The overall sentiment seems to be one of acknowledging the difficulty of the situation and the need for better communication and tooling to navigate such kernel-level changes. A few users also suggest potential workarounds and debugging strategies.

  The Hacker News post titled "When eBPF pt_regs reads return garbage on the latest Linux kernels, blame Fred" has generated a moderate number of comments, most of which delve into the technical details of the issue and offer further insights or related experiences.

  Several commenters discuss the complexities of the pt_regs structure and its usage within the eBPF (extended Berkeley Packet Filter) context. One user highlights the inherent fragility of relying on the layout of pt_regs, as it is architecture-specific and subject to change. They point out that accessing pt_regs directly from eBPF programs is essentially working with a "private, unstable ABI" and that a more robust solution would involve explicitly passing the needed register values to the eBPF program. This echoes the sentiment expressed in the original article about the need for a more stable interface for eBPF programs to access register data.

  Another comment chain focuses on the challenges of maintaining compatibility in the Linux kernel, especially when dealing with low-level structures like pt_regs. One commenter mentions the difficulty of keeping track of all the implicit dependencies and the potential for unintended side effects when making changes to core kernel components. They express sympathy for the developers involved, acknowledging the difficulty of balancing performance optimization with maintaining stable ABIs.

  A couple of commenters share their own experiences with similar issues related to kernel updates and ABI compatibility. One recounts a story of encountering unexpected behavior after a kernel upgrade, which ultimately traced back to changes in internal kernel structures. This anecdote reinforces the point about the inherent risks associated with relying on undocumented or unstable interfaces.

  One commenter questions the use of "blame" in the title, suggesting that it is perhaps too strong a word, given that the change was likely unintentional and a consequence of complex system interactions. They advocate for a more understanding approach, acknowledging the difficulty of maintaining such a large and intricate project as the Linux kernel.

  The comments also touch upon related topics such as the use of kernel tracing tools, the benefits and drawbacks of eBPF technology, and the trade-offs between performance and stability. While not directly related to the core issue, these comments provide additional context and enrich the discussion.

  Overall, the comments on Hacker News provide valuable insights into the complexities of kernel development, the challenges of maintaining ABI compatibility, and the delicate balance between performance and stability. They also offer practical advice for developers working with eBPF and highlight the importance of using stable interfaces whenever possible.

• Case Study: ByteDance Uses eBPF to Enhance Networking Performance

  permalink

  Posted: 2025-01-29 15:58:20

  Verbose tl;dr

  ByteDance, facing challenges with high connection counts and complex network topologies across its global services, leveraged eBPF to significantly improve networking performance. They developed several in-house eBPF-based tools, including a high-performance load balancer and a connection management system, to optimize resource utilization and reduce latency. These tools allowed for more efficient traffic distribution, connection concurrency control, and real-time performance monitoring, leading to improved stability and resource efficiency in their data centers. The adoption of eBPF enabled ByteDance to overcome limitations of traditional kernel-based networking solutions and achieve greater scalability and control over their network infrastructure.

  This case study details how ByteDance, the parent company of popular social media platforms like TikTok and Douyin, leveraged extended Berkeley Packet Filter (eBPF) technology to significantly improve their network performance and observability. ByteDance operates a massive, globally distributed network infrastructure handling immense traffic volumes, necessitating highly optimized and efficient network operations. Traditional network monitoring and troubleshooting methods proved inadequate for their scale and complexity, often involving complex deployments and limited visibility.

  eBPF presented a compelling solution due to its ability to dynamically attach custom programs to various kernel hooks without requiring kernel recompilation or module loading. This flexibility allows for real-time performance analysis and targeted modifications to network behavior. ByteDance utilized eBPF in several key areas:

  1. Gateway Load Balancing: By implementing an eBPF-based load balancer at their gateway layer, ByteDance optimized traffic distribution across multiple backend servers. This approach bypassed the limitations of traditional load balancing methods, enabling more granular control and improved resource utilization. The eBPF program dynamically adjusted traffic flow based on real-time network conditions, ensuring optimal performance even under fluctuating loads. This directly addressed issues with connection stickiness experienced with traditional layer-4 load balancing, achieving more effective distribution across backend servers.

  2. Network Namespace Isolation: ByteDance employs network namespaces to isolate different services and applications. Managing inter-namespace communication efficiently is crucial. They utilized eBPF to optimize traffic forwarding between namespaces, significantly reducing latency and overhead associated with virtual network interfaces. This facilitated smoother and faster communication between services.

  3. Short-lived Connection Optimization: Short-lived connections, common in microservice architectures and high-volume applications, create significant overhead in connection establishment and teardown. ByteDance used eBPF to optimize the handling of these connections, specifically TCP short-lived connections within data centers, by optimizing the TCP stack behavior within the kernel. This optimization reduced the computational burden on servers and improved the efficiency of these transient connections, especially benefiting applications like online gaming and live streaming that rely heavily on quick, short bursts of communication. By offloading connection management to the kernel via eBPF, they bypassed userspace context switching and system calls, resulting in substantial latency reduction.

  4. Network Performance Monitoring and Troubleshooting: eBPF provided enhanced visibility into network traffic, allowing ByteDance to identify and diagnose performance bottlenecks quickly. By attaching eBPF programs to specific points in the network stack, they gathered detailed metrics on packet flow, latency, and errors. This real-time data enabled proactive identification and resolution of performance issues, contributing to improved overall system stability and reduced downtime. Specifically, they gained insight into traffic distribution across servers, latency between services, and other critical performance indicators, enabling them to pinpoint and address bottlenecks proactively.

  Overall, the adoption of eBPF empowered ByteDance to achieve significant improvements in network performance, scalability, and observability. The dynamic nature and flexibility of eBPF enabled them to tailor network operations precisely to their specific needs, resulting in more efficient resource utilization, reduced latency, and improved user experience. This case study demonstrates the potential of eBPF as a powerful tool for optimizing complex, high-traffic network infrastructures.

  • eBPF
  • ByteDance
  • networking
  • performance
  • Case Study
  • Network Optimization
  • Kernel Programming
  • Observability
  • Tracing
  • Linux
  • System Performance
  • Technology

  Summary of Comments ( 7 )
  https://news.ycombinator.com/item?id=42866572

  Verbose tl;dr

  Hacker News users discussed ByteDance's use of eBPF for network performance, focusing on the challenges of deploying such a complex system. Several commenters questioned the actual performance gains, highlighting the lack of quantifiable data in the case study. Some expressed skepticism about the complexity introduced by eBPF, arguing that simpler solutions might be more effective. The discussion also touched on the benefits of XDP for DDoS mitigation and the potential for eBPF to revolutionize networking, while acknowledging the steep learning curve. Several users pointed out the missing details in the case study, such as specific implementations and comparative benchmarks, making it difficult to assess the true impact of ByteDance's approach.

  The Hacker News post titled "Case Study: ByteDance Uses eBPF to Enhance Networking Performance" has generated a moderate discussion with several insightful comments. Many commenters focus on the practical implications and broader trends surrounding eBPF adoption.

  Several comments highlight the growing significance of eBPF for performance optimization, echoing the case study's findings. One commenter emphasizes how eBPF allows bypassing the kernel's general-purpose networking stack, enabling tailored optimizations for specific applications. This aligns with another comment pointing out the power of shifting complex logic from userspace into the kernel using eBPF, improving efficiency without requiring kernel modifications. The inherent flexibility and safety of eBPF are also lauded, with one user mentioning how these attributes make it a compelling alternative to traditional kernel modules.

  The discussion also touches on the expanding use cases of eBPF beyond networking. One commenter notes the growing adoption of eBPF for security and observability, showcasing its versatility. Another comment mentions its use in tracing and profiling, furthering the narrative of eBPF as a powerful tool for diverse performance-related tasks.

  A recurring theme is the potential of eBPF to reshape the networking landscape. One commenter speculates on the possibility of eBPF programs becoming the primary way to interact with the network stack in the future, suggesting a shift away from traditional methods. Another comment emphasizes the rising importance of eBPF expertise, predicting a surge in demand for skilled professionals in this area.

  Some comments provide context and further information related to the case study. One user mentions Cilium, an eBPF-based networking project, and its relevance to service mesh implementations. Another user notes the increasing popularity of eBPF among large organizations and points to Meta (Facebook) as another prominent adopter.

  While expressing enthusiasm for eBPF, some comments also acknowledge its complexities. One user mentions the challenges associated with debugging and managing eBPF programs, hinting at the potential learning curve involved.

  Overall, the comments on the Hacker News post paint a picture of eBPF as a rapidly maturing technology with significant potential for performance enhancement across various domains. The discussion reflects the growing excitement surrounding eBPF and its potential to revolutionize networking and other areas of system optimization.

• Testtrim: A testing tool that couldn't test itself (until now)

  permalink

  Posted: 2025-01-25 20:24:55

  Verbose tl;dr

  Testtrim, a tool designed to reduce the size of test suites while maintaining coverage, ironically struggled to effectively test itself due to its reliance on ptrace for syscall tracing. This limitation prevented Testtrim from analyzing nested calls, leading to incomplete coverage data and hindering its ability to confidently trim its own test suite. A recent update introduces a novel approach using eBPF, enabling Testtrim to accurately trace nested syscalls. This breakthrough allows Testtrim to thoroughly analyze its own behavior and finally optimize its test suite, demonstrating its newfound self-testing capability and reinforcing its effectiveness as a test suite reduction tool.

  Mathieu Fenniak's blog post, "Testtrim: A testing tool that couldn't test itself (until now)," details the intricate journey of enhancing Testtrim, a sophisticated testing tool specifically designed for file descriptor usage in system calls within the Linux kernel. Initially, Testtrim faced a significant limitation: it couldn't effectively test itself. This self-testing deficiency stemmed from its reliance on ptrace for syscall tracing, which presented a fundamental conflict when attempting to trace syscalls generated by the tool itself while it was already utilizing ptrace for its testing operations. This created a recursive ptrace scenario, which the Linux kernel explicitly prohibits to prevent deadlocks and other complications.

  The blog post meticulously outlines the technical complexities involved in overcoming this hurdle. The core of the solution involved leveraging a nested tracing mechanism. Instead of relying solely on ptrace, Testtrim was modified to employ a combination of ptrace(PTRACE_SEIZE) and seccomp(SECCOMP_MODE_FILTER) for syscall interception. This allowed Testtrim to trace the initial set of system calls. For the critical nested layer, where Testtrim needed to analyze its own syscall behavior while already engaged in a tracing operation, the blog post describes the implementation of a custom kernel module. This module intercepted the necessary syscalls specifically within the Testtrim process, providing the required information without resorting to the problematic recursive ptrace.

  Fenniak elaborates on the technical challenges encountered during this implementation. The initial approach involved using kprobes, which proved insufficient due to their inability to access specific register values necessary for comprehensive syscall analysis. Subsequently, the implementation shifted to utilize tracepoints, offering the granular access required for accurate data collection. The blog post delves into the specifics of interacting with the trace_pipe mechanism to retrieve the captured syscall data from the kernel module. It also highlights the importance of carefully managing the synchronization and buffering aspects of this inter-process communication to ensure data integrity and prevent race conditions.

  Finally, the blog post concludes by celebrating the successful implementation of this nested tracing approach. This advancement allows Testtrim to thoroughly test its own intricate syscall interactions, significantly bolstering its reliability and robustness. This achievement marks a substantial improvement in Testtrim's capabilities, solidifying its position as a valuable tool for rigorous testing of file descriptor management within the Linux kernel. The nuanced description of the solution underscores the depth of technical expertise required to navigate the complexities of kernel-level tracing and highlights the innovative approach taken to overcome the inherent limitations of traditional ptrace-based methods.

  • software testing
  • testing tools
  • test coverage
  • metacircular testing
  • self-testing
  • Recursion
  • system calls
  • Tracing
  • Testtrim
  • Debugging

  Summary of Comments ( 0 )
  https://news.ycombinator.com/item?id=42824526

  Verbose tl;dr

  The Hacker News comments discuss the complexity of testing tools like Testtrim, which aim to provide comprehensive syscall tracing. Several commenters appreciate the author's deep dive into the technical challenges and the clever solution involving a VM and intercepting the vmexit instruction. Some highlight the inherent difficulties in testing tools that operate at such a low level, where the very act of observation can alter the behavior of the system. One commenter questions the practical applications, suggesting that existing tools like strace and ptrace might be sufficient in most scenarios. Others point out that Testtrim's targeted approach, specifically focusing on nested virtualization, addresses a niche but important use case not covered by traditional tools. The discussion also touches on the value of learning obscure assembly instructions and the excitement of low-level debugging.

  The Hacker News post titled "Testtrim: A testing tool that couldn't test itself (until now)" sparked a brief but insightful discussion with a few key comments.

  One commenter highlights the core issue presented in the article: the difficulty of testing system call tracing tools due to their reliance on ptrace. They explain that these tools essentially operate by "sitting underneath" the target process, making it challenging to trace themselves without creating a confusing and possibly conflicting hierarchy of tracing. The commenter then expresses appreciation for the clear explanation of the problem and solution provided in the article.

  Another commenter points out the specific challenge related to the "observer effect" in such situations, where the act of observing (tracing) the system calls inherently alters the behavior of the system being observed, making self-testing problematic. They mention the difficulty of using existing tools like strace, further emphasizing the uniqueness of the problem faced by the testtrim developer. This comment adds to the discussion by providing another perspective on the inherent complexity involved.

  A third comment adds a humorous touch, referencing the paradoxical nature of self-reference and using the example of a barber who shaves everyone in town who doesn't shave themselves, posing the classic question of who shaves the barber. This lighthearted comment, while not directly addressing the technical details, captures the essence of the self-referential challenge present in testing a system call tracing tool.

  Finally, one commenter focuses on the solution implemented, which involves conditionally disabling syscall tracing if the process being traced is also testtrim. They applaud the elegance and simplicity of this solution, seeing it as a testament to good design and a clear understanding of the problem.

  While the discussion is not extensive, these comments provide valuable insights into the complexities of testing system call tracing tools, the specific challenges related to self-referential testing, and the appreciation for the elegant solution presented by the author of the original article.

• Bpftune uses BPF to auto-tune Linux systems

  permalink

  Posted: 2024-11-17 11:38:35

  Verbose tl;dr

  bpftune is a new open-source tool from Oracle that leverages eBPF (extended Berkeley Packet Filter) to automatically tune Linux system parameters. It dynamically adjusts settings related to networking, memory management, and other kernel subsystems based on real-time workload characteristics and system performance. The goal is to optimize performance and resource utilization without requiring manual intervention or system-specific expertise, making it easier to adapt to changing workloads and achieve optimal system behavior.

  The project bpftune, hosted on GitHub by Oracle, introduces a novel approach to automatically tuning Linux systems using Berkeley Packet Filter (BPF) technology. This tool aims to dynamically optimize system parameters in real-time based on observed system behavior, rather than relying on static configurations or manual adjustments.

  bpftune leverages the power and flexibility of eBPF to monitor various system metrics and resource utilization. By hooking into critical kernel functions, it gathers data on CPU usage, memory allocation, I/O operations, network traffic, and other relevant performance indicators. This data is then analyzed to identify potential bottlenecks and areas for improvement.

  The core functionality of bpftune revolves around its ability to automatically adjust system parameters based on the insights derived from the collected data. This dynamic tuning mechanism allows the system to adapt to changing workloads and optimize its performance accordingly. For instance, if bpftune detects high network latency, it might adjust TCP buffer sizes or other network parameters to mitigate the issue. Similarly, if it observes excessive disk I/O, it could modify scheduler settings or I/O queue depths to improve throughput.

  The project emphasizes a safe and controlled approach to system tuning. Changes to system parameters are implemented incrementally and cautiously to avoid unintended consequences or instability. Furthermore, bpftune provides mechanisms for reverting changes and monitoring the impact of adjustments, allowing administrators to maintain control over the tuning process.

  bpftune is designed to be extensible and adaptable to various workloads and environments. Users can customize the tool's behavior by configuring the specific metrics to monitor, the tuning algorithms to employ, and the thresholds for triggering adjustments. This flexibility makes it suitable for a wide range of applications, from optimizing server performance in data centers to enhancing the responsiveness of desktop systems. The project aims to simplify the complex task of system tuning, making it more accessible to a broader audience and enabling users to achieve optimal performance without requiring in-depth technical expertise. By using BPF, it aims to offer a low-overhead, high-performance solution for dynamic system optimization.

  • BPF
  • Linux
  • Performance Tuning
  • System Tuning
  • Auto-tuning
  • Kernel Tuning
  • Observability
  • Tracing
  • eBPF
  • optimization
  • Resource Management
  • Automation
  • Profiling
  • System Performance
  • Linux Kernel

  Summary of Comments ( 73 )
  https://news.ycombinator.com/item?id=42163597

  Verbose tl;dr

  Hacker News commenters generally expressed interest in bpftune and its potential. Some questioned the overhead of constantly monitoring and tuning, while others highlighted the benefits for dynamic workloads. A few users pointed out existing tools like tuned-adm, expressing curiosity about bpftune's advantages over them. The project's novelty and use of eBPF were appreciated, with some anticipating its integration into existing performance tuning workflows. A desire for clear documentation and examples of real-world usage was also expressed. Several commenters were specifically intrigued by the network latency use case, hoping for more details and benchmarks.

  The Hacker News post titled "Bpftune uses BPF to auto-tune Linux systems" (https://news.ycombinator.com/item?id=42163597) has several comments discussing the project and its implications.

  Several commenters express excitement and interest in the project, seeing it as a valuable tool for system administrators and developers seeking performance optimization. The use of BPF is praised for its efficiency and ability to dynamically adjust system parameters. One commenter highlights the potential of bpftune to simplify complex tuning tasks, suggesting it could be particularly helpful for those less experienced in performance optimization.

  Some discussion revolves around the specific parameters bpftune adjusts. One commenter asks for clarification on which parameters are targeted, while another expresses concern about the potential for unintended side effects when automatically modifying system settings. This leads to a brief exchange about the importance of understanding the implications of any changes made and the need for careful monitoring.

  A few comments delve into the technical aspects of the project. One commenter inquires about the learning algorithms employed by bpftune and how it determines the optimal parameter values. Another discusses the possibility of integrating bpftune with existing monitoring tools and automation frameworks. The maintainability of the BPF programs used by the tool is also raised as a potential concern.

  The practical applications of bpftune are also a topic of conversation. Commenters mention potential use cases in various environments, including cloud deployments, high-performance computing, and database systems. The ability to dynamically adapt to changing workloads is seen as a key advantage.

  Some skepticism is expressed regarding the project's long-term viability and the potential for over-reliance on automated tuning tools. One commenter cautions against blindly trusting automated solutions and emphasizes the importance of human oversight. The potential for unforeseen interactions with other system components and the need for thorough testing are also highlighted.

  Overall, the comments on the Hacker News post reflect a generally positive reception of bpftune while also acknowledging the complexities and potential challenges associated with automated system tuning. The commenters express interest in the project's development and its potential to simplify performance optimization, but also emphasize the need for careful consideration of its implications and the importance of ongoing monitoring and evaluation.