Stories with Tag Workflows

• Disk I/O bottlenecks in GitHub Actions

  permalink

  Posted: 2025-03-28 15:22:36

  Verbose tl;dr

  GitHub Actions workflows, especially those involving Node.js projects, can suffer from significant disk I/O bottlenecks, primarily during dependency installation (npm install). These bottlenecks stem from the limited I/O performance of the virtual machines used by GitHub Actions runners. This leads to dramatically slower execution times compared to local machines with faster disks. The blog post explores this issue by benchmarking npm install operations across various runner types and demonstrates substantial performance improvements when using self-hosted runners or alternative CI/CD platforms with better I/O capabilities. Ultimately, developers should be aware of these potential bottlenecks and consider optimizing their workflows, exploring different runner options, or utilizing caching strategies to mitigate the performance impact.

  This blog post by Depot details the author's experience troubleshooting and resolving performance bottlenecks stemming from disk I/O limitations within their GitHub Actions CI/CD pipelines. The author initially observed inexplicably slow build times for their Rust project, specifically during the cargo build phase. Suspecting resource constraints within the GitHub Actions virtual environment, they began investigating various possibilities, including CPU, memory, and network limitations. However, through systematic experimentation and profiling using tools like iostat, they pinpointed the root cause to be sluggish disk I/O performance.

  The author meticulously describes their investigation process, showcasing the data they collected and the reasoning behind their conclusions. They initially ruled out CPU and memory bottlenecks as the primary culprits due to consistently low utilization during the slow builds. Network limitations were also discounted after observing consistent network performance. This led them to focus on disk I/O, where iostat revealed exceptionally high "await" times, indicating that processes were spending significant time waiting for disk operations to complete.

  Having identified disk I/O as the bottleneck, the author explored several mitigation strategies. They experimented with utilizing tmpfs, a RAM-based file system, to hold parts of the build process, effectively bypassing the slower physical disk. Mounting the project's target directory (where build artifacts are stored) within tmpfs yielded significant performance improvements, drastically reducing build times.

  Further investigation revealed that the performance discrepancy was primarily due to the differing I/O characteristics between the self-hosted runner used for local testing and the GitHub-hosted runner used for CI. The self-hosted runner likely utilized an SSD, providing significantly faster random read/write speeds compared to the potentially slower storage used by the GitHub-hosted runner. The author emphasizes the importance of considering these environmental differences when optimizing CI pipelines.

  The blog post concludes with a recommendation to consider tmpfs as a valuable tool for addressing I/O bottlenecks in CI environments, particularly for scenarios involving frequent disk access, such as compilation processes. It emphasizes the importance of profiling and understanding resource utilization to pinpoint performance bottlenecks accurately. The author also acknowledges that tmpfs may not be a universal solution, particularly for very large projects where RAM capacity might become a limiting factor. However, they suggest it as a valuable optimization technique for many projects running in constrained CI environments.

  • GitHub Actions
  • disk i/o
  • bottlenecks
  • CI/CD
  • performance
  • optimization
  • Workflows
  • Continuous Integration
  • continuous delivery
  • Virtual Machines
  • io performance
  • disk performance

  Summary of Comments ( 16 )
  https://news.ycombinator.com/item?id=43506574

  Verbose tl;dr

  HN users discussed the surprising performance disparity between GitHub-hosted and self-hosted runners, with several suggesting network latency as a significant factor beyond raw disk I/O. Some pointed out the potential impact of ephemeral runner environments and the overhead of network file systems. Others highlighted the benefits of using actions/cache or alternative CI providers with better I/O performance for specific workloads. A few users shared their experiences, with one noting significant improvements from self-hosting and another mentioning the challenges of optimizing build processes within GitHub Actions. The general consensus leaned towards self-hosting for I/O-bound tasks, while acknowledging the convenience of GitHub's hosted runners for less demanding workflows.

  The Hacker News post titled "Disk I/O bottlenecks in GitHub Actions" (https://news.ycombinator.com/item?id=43506574) has generated a moderate number of comments, discussing various aspects of the linked blog post about disk I/O performance issues in GitHub Actions.

  Several commenters corroborate the author's findings, sharing their own experiences with slow disk I/O in GitHub Actions. One user mentions observing significantly improved performance after switching to self-hosted runners, highlighting the potential benefits of having more control over the execution environment. They specifically mention the use of tmpfs for build directories as a contributing factor to the improved speeds.

  Another commenter points out that the observed I/O bottlenecks are likely not unique to GitHub Actions, suggesting that similar issues might exist in other CI/CD environments that rely on virtualized or containerized runners. They argue that understanding the underlying hardware and storage configurations is crucial for optimizing performance in any CI/CD pipeline.

  A more technically inclined commenter discusses the potential impact of different filesystem layers and virtualization technologies on I/O performance. They suggest that the choice of filesystem within the runner's container, as well as the virtualization technology used by the underlying infrastructure, could play a significant role in the observed performance differences.

  One commenter questions the methodology used in the original blog post, specifically regarding the use of dd for benchmarking. They argue that dd might not accurately reflect real-world I/O patterns encountered in typical CI/CD workloads. They propose alternative benchmarking tools and techniques that might provide more relevant insights into the performance characteristics of the storage system.

  Finally, some commenters discuss potential workarounds and mitigation strategies for dealing with slow disk I/O in GitHub Actions, including using RAM disks, optimizing build processes to minimize disk access, and leveraging caching mechanisms to reduce the amount of data that needs to be read from or written to disk. They also discuss the trade-offs associated with each of these approaches, such as the limited size of RAM disks and the potential complexity of implementing custom caching solutions.

• Whose code am I running in GitHub Actions?

  permalink

  Posted: 2025-03-25 17:17:05

  Verbose tl;dr

  GitHub Actions' opaque nature makes it difficult to verify the provenance of the code being executed in your workflows. While Actions marketplace listings link to source code, the actual runner environment often uses pre-built distributions hosted by GitHub, with no guarantee they precisely match the public repository. This discrepancy creates a potential security risk, as malicious actors could alter the distributed code without updating the public source. Therefore, auditing the integrity of Actions is crucial, but currently complex. The post advocates for reproducible builds and improved transparency from GitHub to enhance trust and security within the Actions ecosystem.

  Alex Chan's blog post, "Whose code am I running in GitHub Actions?", delves into the critical issue of supply chain security within the context of GitHub Actions, a popular CI/CD platform. The central question posed is how much trust users implicitly place in the various actions they integrate into their workflows, and what mechanisms exist to verify the integrity and provenance of these actions.

  The post begins by highlighting the convenience and extensibility offered by GitHub Actions' marketplace, enabling developers to incorporate pre-built functionalities into their workflows with minimal effort. However, this convenience comes with an inherent security risk. By incorporating third-party actions, developers essentially grant those actions access to their codebase and potentially sensitive secrets, opening up avenues for malicious actors.

  Chan emphasizes the potential vulnerability stemming from compromised accounts of action maintainers. If an attacker gains access to an action maintainer's account, they could modify the action's code to perform malicious activities, impacting all repositories utilizing that action. Even seemingly innocuous actions could be weaponized to exfiltrate data or inject vulnerabilities into the software being built.

  The blog post then explores various strategies for mitigating these risks. One approach discussed is pinning actions to specific commit SHAs. This ensures that a known, vetted version of the action is used, preventing automatic updates that might introduce malicious code. However, this approach introduces the overhead of manually updating actions and potentially missing out on beneficial updates and bug fixes.

  Another method is using a private registry for actions. This allows organizations to host and control the actions used within their workflows, providing greater assurance over their security and provenance. While offering increased control, this approach requires more setup and maintenance.

  Furthermore, the post discusses leveraging OpenID Connect (OIDC) to establish trust between GitHub Actions and cloud providers. This allows actions to access cloud resources without needing long-lived secrets, thereby minimizing the potential damage from compromised actions.

  Chan also touches on the importance of auditing the actions used in workflows, including understanding their dependencies and scrutinizing their code for potential security flaws. This involves actively reviewing the action's source code, understanding its permissions, and considering the reputation and trustworthiness of the action maintainer.

  The post concludes by emphasizing the need for a multi-layered approach to security in GitHub Actions workflows. This includes combining various mitigation strategies, such as pinning actions, using private registries, employing OIDC, and performing regular audits, to minimize the risk of running potentially malicious code. The ultimate goal is to establish a robust security posture that balances the convenience of using third-party actions with the critical need to protect sensitive data and maintain the integrity of the software development lifecycle.

  • GitHub Actions
  • Security
  • Auditing
  • Code Provenance
  • software supply chain
  • CI/CD
  • DevOps
  • software engineering
  • best practices
  • Workflows
  • Permissions
  • GitHub

  Summary of Comments ( 19 )
  https://news.ycombinator.com/item?id=43473623

  Verbose tl;dr

  HN users largely agreed with the author's concerns about the opacity of third-party GitHub Actions. Several highlighted the potential security risks of blindly trusting external code, with some suggesting that reviewing the source of each action should be standard practice, despite the impracticality. Some argued for better tooling or built-in mechanisms within GitHub Actions to improve transparency and security. The potential for malicious actors to introduce vulnerabilities through seemingly benign actions was also a recurring theme, with users pointing to the risk of supply chain attacks and the difficulty in auditing complex dependencies. Some suggested using self-hosted runners or creating internal action libraries for sensitive projects, although this introduces its own management overhead. A few users countered that similar trust issues exist with any third-party library and that the benefits of using pre-built actions often outweigh the risks.

  The Hacker News post "Whose code am I running in GitHub Actions?" (linking to an article about auditing GitHub Actions for security risks) generated a moderate amount of discussion with several compelling points raised.

  Several commenters focused on the inherent trust issues with third-party actions. One commenter highlighted the risk of malicious actors gaining control of popular actions and injecting malicious code, potentially impacting numerous repositories. They underscored the importance of auditing dependencies, even within trusted actions, as they can pull in other less-vetted actions.

  Another thread discussed the difficulty of thoroughly auditing actions. Even simple actions can be complex under the hood, and reviewing them requires significant time and expertise. The analogy to npm packages was drawn, with the observation that security issues in widely used packages can have cascading effects. The point was made that a comprehensive audit of GitHub Actions is a non-trivial task.

  A commenter mentioned a tool called actionlint, which helps in catching potential security vulnerabilities in GitHub Actions workflows. This provided a concrete solution for users looking to improve the security posture of their CI/CD pipelines.

  The trade-off between convenience and security was also a recurring theme. While pre-built actions streamline workflows, they come with inherent risks. One commenter advocated for building custom actions for critical tasks whenever feasible, despite the increased overhead, to maintain greater control over the code being executed.

  The feasibility of self-hosting runners was discussed, presenting it as a method to mitigate some of the security concerns around third-party actions. However, commenters acknowledged the added complexity and maintenance overhead associated with this approach, suggesting it's not a universally applicable solution.

  One user suggested using a tool like act for local testing, which allows developers to run their workflows locally before pushing them to GitHub, offering an additional layer of security.

  Finally, the importance of pinning action versions was emphasized to prevent unexpected updates from introducing breaking changes or vulnerabilities. This practice allows for more controlled and predictable CI/CD execution.

  Overall, the comments paint a picture of a complex ecosystem where convenience often comes at the cost of security. While tools and strategies exist to mitigate risks, the responsibility ultimately falls on developers to carefully consider the implications of the actions they use.