The blog post argues for an intermediate representation (IR) layer in query compilers between the logical plan and the physical plan, called the "relational algebra IR." This layer would represent queries in a standardized, relational algebra form, enabling greater portability and reusability of optimization rules across different physical execution engines. Currently, optimization logic is often tightly coupled to specific physical plans, making it difficult to adapt to new engines or hardware. By introducing this standardized relational algebra IR, query compilers can achieve better modularity and extensibility, simplifying development and allowing for easier experimentation with new optimization strategies without needing to rewrite code for each backend. This ultimately leads to more efficient query execution across diverse environments.
Yasser is developing "Tilde," a new compiler infrastructure designed as a simpler, more modular alternative to LLVM. Frustrated with LLVM's complexity and monolithic nature, he's building Tilde with a focus on ease of use, extensibility, and better diagnostics. The project is in its early stages, currently capable of compiling a subset of C and targeting x86-64 Linux. Key differentiating features include a novel intermediate representation (IR) designed for efficient analysis and transformation, a pipeline architecture that facilitates experimentation and customization, and a commitment to clear documentation and a welcoming community. While performance isn't the primary focus initially, the long-term goal is to be competitive with LLVM.
Hacker News users discuss the author's approach to building a compiler, "Tilde," positioned as an LLVM alternative. Several commenters express skepticism about the project's practicality and scope, questioning the rationale behind reinventing LLVM, especially given its maturity and extensive community. Some doubt the performance claims and suggest benchmarks are needed. Others appreciate the author's ambition and the technical details shared, seeing value in exploring alternative compiler designs even if Tilde doesn't replace LLVM. A few users offer constructive feedback on specific aspects of the compiler's architecture and potential improvements. The overall sentiment leans towards cautious interest with a dose of pragmatism regarding the challenges of competing with an established project like LLVM.
Summary of Comments ( 8 )
https://news.ycombinator.com/item?id=42996656
HN commenters generally agree with the author's premise that a middle tier is missing in query compilers, sitting between logical optimization and physical optimization. This tier would handle "cross-physical plan" optimizations, allowing for better cost-based decisions that consider different physical plan choices holistically rather than sequentially. Some discuss the challenges in implementing this, particularly the explosion of search space and the difficulty in accurately costing plans. Others offer specific examples where such a tier would be beneficial, such as selecting join algorithms based on data distribution or optimizing for specific hardware like GPUs. A few commenters mention existing systems that implement similar concepts, though not necessarily as a distinct tier, suggesting the idea is already being explored in practice. Some debate the practicality of the proposed solution, suggesting alternative approaches like adaptive query execution or learned optimizers.
The Hacker News post titled "The missing tier for query compilers," linking to an article on scattered-thoughts.net, has generated a modest discussion with a few interesting points.
One commenter highlights the value of the proposed "IR optimizer" tier, agreeing that it sits logically between the logical plan optimization and the physical plan generation. They point out the challenge of optimizations that are neither purely logical nor physical, citing predicate pushdown as a prime example. This commenter further emphasizes the importance of cost-based optimization at this intermediate stage, suggesting it allows for more informed decisions.
Another commenter focuses on the practical difficulties of building such a system. They mention the considerable effort involved in accurately estimating costs without generating a full physical plan, suggesting this might diminish the potential benefits. They also highlight the complexities introduced by supporting diverse execution backends, each with unique performance characteristics.
A third commenter draws a parallel to LLVM, noting its similar tiered architecture and how it effectively bridges the gap between higher-level representations and target-specific optimizations. They propose that adopting a similar approach in query compilers could lead to significant improvements.
A brief comment concurs with the author's premise, mentioning that current query optimizers often struggle with certain types of optimizations. They agree that an intermediate representation could address these shortcomings.
Another commenter makes a more abstract observation, likening the concept to the "no free lunch" theorem. They suggest that while the proposed approach has merit, there will always be trade-offs and challenges associated with building truly universal optimization strategies.
The discussion, while not extensive, provides valuable perspectives on the challenges and potential benefits of introducing an intermediate representation in query compilers. The comments generally agree on the theoretical value but also acknowledge the practical difficulties of implementation and cost estimation. The comparison to LLVM's architecture offers an intriguing potential direction for future research in this area.