Make Kyuubi As An Unified SQL Gateway

[packyan]

Kyuubi is an SQL Gateway that integrates multiple SQL query engines, but the syntax of each query engine varies. When users need to migrate HiveQL to Kyuubi's Spark SQL engine, they need to modify queries with incompatible syntax. If Kyuubi can unify the syntax of various query engines, it will greatly improve Kyuubi's usability.

And if Kyuubi can sense the load of each engine, it can route SQL to the optimal query engine based on the load.

[pan3793]

Personally, I think it's a tough way, and if we do that, we will find there are some functionality gaps in different engines, and Kyuubi must write some computing logic to make up. Eventually, Kyuubi will evolve into a computing engine like Trino. Then why just use Trino directly?

[yaooqinn]

It always sounds like a good idea but never actually is.

Speaking of HiveQL, Spark SQL, or other SQL dialects, the fact is that they are different for now. They don't stay where they are now and are evolving, with 50% being consistent/compliant aligning with ANSI Standard, and 50% being much more different from each other aligning with the fast-growing technology and business. To put it simply, a consistent syntax is not achievable within the same SQL engine across different versions.

In order to harmonize the two dialects, it is not an option to create another dialect to cover them, which would only increase the number of the other dialects. It might be a good option to increase the first 50% I mentioned above by the efforts of both communities.

And if Kyuubi can sense the load of each engine, it can route SQL to the optimal query engine based on the load.

This is an improvement we can make.

[Hamburger66]

I was curious if there has been any progress on this topic lately? How can Kyuubi sense the load of each engine and route based on the load?