The blog post details troubleshooting high CPU usage attributed to the writeback process in a Linux kernel. After initial investigations pointed towards cgroups and specifically the cpu.cfs_period_us parameter, the author traced the issue to a tight loop within the cgroup writeback mechanism. This loop was triggered by a large number of cgroups combined with a specific workload pattern. Ultimately, increasing the dirty_expire_centisecs kernel parameter, which controls how long dirty data stays in memory before being written to disk, provided the solution by significantly reducing the writeback activity and lowering CPU usage.
The blog post "Debugging our new Linux kernel" details a performance investigation centered around high CPU utilization stemming from the writeback process within Linux control groups (cgroups). The author, facing sluggish system performance after a kernel upgrade, noticed that a significant portion of CPU cycles were being consumed by writeback threads associated with specific cgroups. This suggested a problem related to how the kernel was managing data flushing to disk within these isolated resource groups.
The initial suspicion fell upon the storage layer, prompting checks for disk I/O bottlenecks. However, analysis of disk metrics revealed normal operation, indicating the issue resided elsewhere. This redirected the focus towards the kernel's memory management and its interaction with cgroups.
The investigation proceeded by leveraging kernel tracing tools like ftrace and perf. These utilities allowed the author to inspect the kernel's execution path and pinpoint the functions involved in the excessive writeback activity. The tracing data highlighted frequent calls related to memory reclamation and page cache flushing within the affected cgroups.
Through careful examination of the trace output, the author observed a pattern of repeated scanning of inactive file pages. This led to the hypothesis that the kernel was unnecessarily triggering writeback operations for pages that hadn't been modified or accessed recently. The excessive scanning and subsequent flushing contributed to the observed high CPU load.
Further scrutiny pointed towards a recent change in the kernel's memory management subsystem, specifically a modification to the kswapd daemon's behavior within cgroups. This change, intended to improve memory management efficiency, appeared to have inadvertently introduced a regression causing excessive scanning and flushing of inactive pages within specific cgroups.
The author concluded that the high CPU usage by writeback was a direct consequence of this unintended side-effect of the kernel upgrade. While a definitive fix within the kernel itself wasn't immediately available, the post concludes with the author implementing a temporary workaround by adjusting the dirty_ratio and dirty_background_ratio cgroup parameters. This modification effectively controlled the aggressiveness of the kernel's writeback mechanism within the affected cgroups, alleviating the high CPU utilization and restoring acceptable system performance. The author acknowledges this is a temporary solution and looks forward to a proper kernel patch addressing the root cause.
Summary of Comments ( 15 )
https://news.ycombinator.com/item?id=43046174
Commenters on Hacker News largely discuss practical troubleshooting steps and potential causes of the high CPU usage related to cgroups writeback described in the linked blog post. Several suggest using tools like
perfto profile the kernel and pinpoint the exact function causing the issue. Some discuss potential problems with the storage layer, like slow I/O or a misconfigured RAID, while others consider the possibility of a kernel bug or an interaction with specific hardware or drivers. One commenter shares a similar experience with NFS and high CPU usage related to writeback, suggesting a potential commonality in networked filesystems. Several users emphasize the importance of systematic debugging and isolation of the problem, starting with simpler checks before diving into complex kernel analysis.The Hacker News post titled "Linux kernel cgroups writeback high CPU troubleshooting" sparked a discussion with several insightful comments.
One commenter shared a similar experience, highlighting how an increased
vm.dirty_ratiosetting led to performance improvements in a database workload. They also emphasized the importance of settingvm.dirty_background_ratioappropriately to avoid performance hiccups due to sudden writeback flushes.Another commenter delved into the technical details of writeback, explaining how the Linux kernel manages dirty pages and the role of
pdflush(now replaced byflush-x:ykernel threads). They noted how an incorrectly configuredvm.dirty_ratiocan lead to excessive CPU usage by these threads, precisely the issue faced by the original author. This commenter also suggested checking thebdi(backing device information) statistics to pinpoint the specific device causing the writeback bottleneck.A third commenter provided a practical tip: using
iostat -x 1to monitor disk activity during periods of high CPU usage attributed to writeback. This command helps identify whether the disk itself is the bottleneck or if the issue lies within the kernel's writeback mechanisms.Another commenter pointed out the importance of considering the underlying storage hardware when tuning
vm.dirty_ratio. They advised caution when dealing with SSDs, as aggressive writeback settings could negatively impact their lifespan. This advice underscored the need for a holistic approach to performance tuning, considering both software and hardware limitations.Furthermore, a user shared their personal anecdote of encountering similar issues with NFS shares. They suggested investigating NFS-specific settings and configurations as potential culprits for high CPU usage related to writeback when working with network file systems.
Several other comments provided additional context and resources. One user linked to a kernel documentation page explaining the
dirty_ratioanddirty_background_ratioparameters, offering further reading for those interested in understanding the intricacies of the Linux kernel's memory management. Another commenter mentioned the potential impact of memory pressure on writeback activity, suggesting checking memory usage metrics alongside disk I/O statistics.Overall, the comments on the Hacker News post offered a valuable collection of practical advice, technical explanations, and real-world experiences, providing a comprehensive perspective on troubleshooting high CPU usage related to writeback in the Linux kernel.