Trino "extendability"

[kassett]

I think this is my first discussion topic here so I'd like to preface by saying that I've been advocating for Trino for years. I think it's a great product, I've used it in various degrees in every company I've worked at.

Recently I've been spending significant time getting into the nitty-gritty of MySQL/MariaDB, mostly trying to build really robust systems for zero-downtime deployments. At some point I had a pretty good idea for how to do this but it required building a UDF -- if you don't know, a UDF is a function / procedure written in C and then loaded as a .so file. The functionality was pretty simple but was prohibitively difficult given the MySQL development environment. Looking into it more, I discovered that almost nobody writes this UDFs anymore, I think because the development environment is so bad.

Right now I am working on a Trino connector for DynamoDB. It is slow-going work, mostly because my Java is well rusted and the project takes so long to build -- I hate to say it, but I've had similar experiences to working on MySQL. My question is, what does the future of extendability look like for Trino? Obviously for much smaller use-cases, Lua is a great option for adding extendability. But is there something that could be done to allow for dynamically loading connectors. This seems an especially helpful design given the build times for Trino -- the more connectors you add, the longer it takes to build. I've never seen this pattern for anything even resembling a database, but gRPC is frequently used for systems with robust plugin architecture. I understand that this is a major change that would have to happen, but I think fostering a rich plugin environment makes for a stronger developer community and a better product overall. Let me know your thoughts?

[martint]

Welcome to the Trino community forum!

You don’t need to build your plugins inside the Trino codebase. Many connectors are developed as separate Maven projects. This speeds up development significantly since you’re only building your own module. Take a look at https://github.com/nineinchnick/trino-git, for example.

Trino also supports dynamically resolved functions (see ConnectorMetadata and FunctionProvider), so you can expose functions from your connector however you like -- Java, external systems, etc. In addition, you can define functions directly in SQL (https://trino.io/docs/current/udf/sql.html) or Python (https://trino.io/docs/current/udf/python.html)

@nineinchnick, can you share your experience with building plugins outside of the Trino source tree?

[nineinchnick]

I can recommend building new plugins in separate repositories, to avoid having to learn how to only build selected modules in a Maven project. But if you do work on new connectors in Trino, you can activate the gib Maven profile to only build impacted modules. It's used in the CI workflow run for all pull requests.