GitHub Actions workflows, especially those involving Node.js projects, can suffer from significant disk I/O bottlenecks, primarily during dependency installation (npm install). These bottlenecks stem from the limited I/O performance of the virtual machines used by GitHub Actions runners. This leads to dramatically slower execution times compared to local machines with faster disks. The blog post explores this issue by benchmarking npm install operations across various runner types and demonstrates substantial performance improvements when using self-hosted runners or alternative CI/CD platforms with better I/O capabilities. Ultimately, developers should be aware of these potential bottlenecks and consider optimizing their workflows, exploring different runner options, or utilizing caching strategies to mitigate the performance impact.
This blog post by Depot details the author's experience troubleshooting and resolving performance bottlenecks stemming from disk I/O limitations within their GitHub Actions CI/CD pipelines. The author initially observed inexplicably slow build times for their Rust project, specifically during the cargo build phase. Suspecting resource constraints within the GitHub Actions virtual environment, they began investigating various possibilities, including CPU, memory, and network limitations. However, through systematic experimentation and profiling using tools like iostat, they pinpointed the root cause to be sluggish disk I/O performance.
The author meticulously describes their investigation process, showcasing the data they collected and the reasoning behind their conclusions. They initially ruled out CPU and memory bottlenecks as the primary culprits due to consistently low utilization during the slow builds. Network limitations were also discounted after observing consistent network performance. This led them to focus on disk I/O, where iostat revealed exceptionally high "await" times, indicating that processes were spending significant time waiting for disk operations to complete.
Having identified disk I/O as the bottleneck, the author explored several mitigation strategies. They experimented with utilizing tmpfs, a RAM-based file system, to hold parts of the build process, effectively bypassing the slower physical disk. Mounting the project's target directory (where build artifacts are stored) within tmpfs yielded significant performance improvements, drastically reducing build times.
Further investigation revealed that the performance discrepancy was primarily due to the differing I/O characteristics between the self-hosted runner used for local testing and the GitHub-hosted runner used for CI. The self-hosted runner likely utilized an SSD, providing significantly faster random read/write speeds compared to the potentially slower storage used by the GitHub-hosted runner. The author emphasizes the importance of considering these environmental differences when optimizing CI pipelines.
The blog post concludes with a recommendation to consider tmpfs as a valuable tool for addressing I/O bottlenecks in CI environments, particularly for scenarios involving frequent disk access, such as compilation processes. It emphasizes the importance of profiling and understanding resource utilization to pinpoint performance bottlenecks accurately. The author also acknowledges that tmpfs may not be a universal solution, particularly for very large projects where RAM capacity might become a limiting factor. However, they suggest it as a valuable optimization technique for many projects running in constrained CI environments.
Summary of Comments ( 16 )
https://news.ycombinator.com/item?id=43506574
HN users discussed the surprising performance disparity between GitHub-hosted and self-hosted runners, with several suggesting network latency as a significant factor beyond raw disk I/O. Some pointed out the potential impact of ephemeral runner environments and the overhead of network file systems. Others highlighted the benefits of using actions/cache or alternative CI providers with better I/O performance for specific workloads. A few users shared their experiences, with one noting significant improvements from self-hosting and another mentioning the challenges of optimizing build processes within GitHub Actions. The general consensus leaned towards self-hosting for I/O-bound tasks, while acknowledging the convenience of GitHub's hosted runners for less demanding workflows.
The Hacker News post titled "Disk I/O bottlenecks in GitHub Actions" (https://news.ycombinator.com/item?id=43506574) has generated a moderate number of comments, discussing various aspects of the linked blog post about disk I/O performance issues in GitHub Actions.
Several commenters corroborate the author's findings, sharing their own experiences with slow disk I/O in GitHub Actions. One user mentions observing significantly improved performance after switching to self-hosted runners, highlighting the potential benefits of having more control over the execution environment. They specifically mention the use of tmpfs for build directories as a contributing factor to the improved speeds.
Another commenter points out that the observed I/O bottlenecks are likely not unique to GitHub Actions, suggesting that similar issues might exist in other CI/CD environments that rely on virtualized or containerized runners. They argue that understanding the underlying hardware and storage configurations is crucial for optimizing performance in any CI/CD pipeline.
A more technically inclined commenter discusses the potential impact of different filesystem layers and virtualization technologies on I/O performance. They suggest that the choice of filesystem within the runner's container, as well as the virtualization technology used by the underlying infrastructure, could play a significant role in the observed performance differences.
One commenter questions the methodology used in the original blog post, specifically regarding the use of
ddfor benchmarking. They argue thatddmight not accurately reflect real-world I/O patterns encountered in typical CI/CD workloads. They propose alternative benchmarking tools and techniques that might provide more relevant insights into the performance characteristics of the storage system.Finally, some commenters discuss potential workarounds and mitigation strategies for dealing with slow disk I/O in GitHub Actions, including using RAM disks, optimizing build processes to minimize disk access, and leveraging caching mechanisms to reduce the amount of data that needs to be read from or written to disk. They also discuss the trade-offs associated with each of these approaches, such as the limited size of RAM disks and the potential complexity of implementing custom caching solutions.