[Docs]: Reorder query engines based on PMC suggested ordering

[bitsondatadev]

@Fokko @RussellSpitzer @nastra @ajantha-bhat, is there any reason we shouldn't just make the query engines in the docs page alphanumeric?

Potentially give them their own order as @flyrain suggests?

I believe Spark, Flink, and Trino have the most support and why they came first initially, but as this project grows, it's likely better we just follow alphanumeric everywhere. Unless we want to group engines by some metric of Iceberg support and then alphanumeric.

Let me know what you all think and if this should be a topic at the next sync.

Originally posted by @bitsondatadev in #9669 (comment)

[manuzhang]

We have dedicated docs for Spark, Flink and Hive while those for other engines are links to their own sites. I think our own docs should come first.

[ajantha-bhat]

+1 for reordering,

maybe moving query engines under a section "engines" similar to "integrations" can also help in declutter the contents.

And we may also need a section called "catalogs" with a page of each catalog.

[nastra]

I agree with @manuzhang. Given that the community supports for Spark / Flink / Hive, those docs should come first IMO. In order to de-clutter the left-side menu, maybe we should have Community-maintained engine support and Externally-maintained engine support (or something along those lines) to better indicate what's supported by the community inside the Iceberg repo vs what's supported within other projects/vendors.

[bitsondatadev]

I like that split. And within these splits would we apply alphanumeric sorting then?

Also do we have a separate vendor list? I like actually listing out the vendors as products vs having a vendor "pitch page". Feels more dev focused.

Iceberg engines

• Flink
• Hive
• Spark

Community engines

• Doris
• Impala
• Starrocks
• Trino

Enterprise engines

• AWS Athena
• AWS EMR
• AWS Redshift
• Big Query
• Cloudera
• Snowflake
• Starburst

Catalogs

• DynamoDB
• Hive Metastore
• JDBC
• Nessie
• REST

Oh and this begs the question about Enterprise catalogs like Glue and Tabular.

I'm missing a bunch but that would be the idea.

[manuzhang]

I don't think vendors like AWS EMR or Cloudera should be listed here. They may provide engines, catalogs or compaction services that integrate with Iceberg.

[scottteal]

Would you take the same approach for catalogs with "Community Catalogs" (Hive Metastore, Nessie, etc.) and "Enterprise Catalogs" (Amazon DynamoDB, AWS Glue Data Catalog, Dremio Arctic, GCP BigLake Metastore, Tabular, etc) ?

[bitsondatadev]

I don't think vendors like AWS EMR or Cloudera should be listed here. They may provide engines, catalogs or compaction services that integrate with Iceberg.

Perhaps those could be categorized into a final bucket of Integrations?

I think building up a page akin to Trino's ecosystem page is kind of what I have in mind, and I think these categories help a lot.

[bitsondatadev]

Would you take the same approach for catalogs with "Community Catalogs" (Hive Metastore, Nessie, etc.) and "Enterprise Catalogs" (Amazon DynamoDB, AWS Glue Data Catalog, Dremio Arctic, GCP BigLake Metastore, Tabular, etc) ?

Yeah, and I guess the same could be said for engines as well. So maybe making too heavy of splits isn't valuable, but what we could do to maintain some delineation of what's vendor vs what is not could be done with tags. Which makes search and creating dynamic pages much easier.

[monimiller]

Hi there - Starburst devrel gal here. Was looking at the iceberg documentation recently and saw Dremio & Athena listed on the side of the documentation. Any chance we can get Starburst added as suggested above? Think this would be a great place to link to: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html. LMK if you have any Q's. Thanks!

[bitsondatadev]

Hi there - Starburst devrel gal here. Was looking at the iceberg documentation recently and saw Dremio & Athena listed on the side of the documentation. Any chance we can get Starburst added as suggested above? Think this would be a great place to link to: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html. LMK if you have any Q's. Thanks!

Hey Mon,

Thanks for bringing this up. We're missing a bunch of engines here and once we add them all, we will need to reorganize how our docs are laid out. As you can see, I've modeled some suggestions of the Trino docs, but ultimately this will be an Apache Iceberg PMC decision.

Let me run this by the PMC and see if we should make this a mailing list thread or if the PMC have already seen this and have some ideas already.

For the weight my opinion holds, I think we should at least do some split of OSS and Enterprise, and then do some consistent heuristic like alphanumeric or like most features implemented, highest adoption etc... and then sort based on that heuristic within the two categories.

We'll see, since doing better than alphanumeric requires a bit of work.

Thanks :)