[Website] Add Blog about Comet Donation #479

alamb · 2024-02-27T10:56:34Z

Rationale: We should tell the world about the Comet donation.

See background and more details on the mailing list: https://lists.apache.org/thread/0q1rb11jtpopc7vt1ffdzro0omblsh0s

alamb · 2024-02-27T10:57:05Z

_posts/2024-02-27-comet-donation.md

+┃       (Native Code)       ┃
+┗━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
+```
+


I felt we needed some sort of image, but this is pretty basic ASCII art... If anyone has other ideas that would be appreciated

The image in comet's README is a good alternative, but we need to add the driver part.

That is a good image, but it is quite detailed

What I was hoping was to have an image that visually depicts the relationship between Spark and Comet

Something similar maybe to https://github.com/oap-project/gluten?tab=readme-ov-file#2-architecture

I don't know enough about how spark is implemented internally to make such a picture now, though

I'm thinking a figure like:

Spark Frontend / Optimizer (JVM Based) -> Comet Execution Engine (Native Code)

-> Arrow DataFusion

-> Spark compatible expression/operator

I took a shot at updating it in 70ff88d

What do you think?

Looks good to me. Thanks @alamb

alamb · 2024-02-27T10:57:39Z

_posts/2024-02-27-comet-donation.md

+[Gluten project]: https://incubator.apache.org/projects/gluten.html
+
+# Get Involved
+Comet is still in the early stages of development and we would love to have you


It would be good to double check / make sure this is the right way to ask people to get involved

This looks good enough for now. Maybe we can consider opening the "Discussions" tab for people to ask questions & raise discussions, but that can be done later.

_posts/2024-02-27-comet-donation.md

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Andy Grove <andygrove73@gmail.com>

alamb · 2024-02-28T11:52:06Z

@sunchao / @viirya do you know of any benchmark results that we can share about comet (e.g. "it ran the same query 30% faster with 50% the resources")? I realize we may not have numbers that can be shared at this time, but if we did I think it would make this post more compelling

sunchao · 2024-02-28T16:43:19Z

@alamb we don't have numbers with Comet yet, but we are working on this and hopefully it will be available in the 1-2 months.

sunchao · 2024-02-28T16:44:38Z

_posts/2024-02-27-comet-donation.md

+replacement for Spark's JVM based SQL execution engine and offers significant
+performance improvements for some workloads as shown below.
+
+```text


instead of text, maybe we can create some diagrams using tools like Lucidchart? I can help to translate this.

That would be great if you can translate the diagram to an image. Maybe we could also (re)use such an image in the main comet readme.

How does this look? Let me know if you have ideas to improve it.

I swapped the blog to use the image and I think it is looking pretty good

Also I liked the image so much I propose putting it in the main comet repo: apache/datafusion-comet#148

sunchao · 2024-02-28T16:50:21Z

_posts/2024-02-27-comet-donation.md

+[Gluten project]: https://incubator.apache.org/projects/gluten.html
+
+# Get Involved
+Comet is still in the early stages of development and we would love to have you


This looks good enough for now. Maybe we can consider opening the "Discussions" tab for people to ask questions & raise discussions, but that can be done later.

sunchao · 2024-02-28T16:51:15Z

I'm hoping we can have another blog post right after the initial release, with some benchmark numbers.

_posts/2024-02-27-comet-donation.md

alamb · 2024-02-28T20:03:44Z

Here is some third-party coverage: https://thenewstack.io/apple-comet-brings-fast-vector-processing-to-apache-spark/

alamb · 2024-02-29T14:31:25Z

I'm hoping we can have another blog post right after the initial release, with some benchmark numbers.

That makes a lot of sense. I put a note about working on an initial release and another update after that in b3d17ce

SChakravorti21 · 2024-02-29T16:26:11Z

_posts/2024-02-27-comet-donation.md

+Comet is one of a growing class of projects that aim to accelerate Spark using
+native columnar engines such as the proprietary [Databricks Photon Engine] and
+the open source [Gluten project] and [Spark RAPIDS].


Would it help to give some background as to why these initiatives have been popping up in the first place? I know the article alludes to columnar execution and lower resource usage, but (imo) it would be more approachable if it were framed from the perspective of the struggles Spark users face today, and then explaining how these initiatives address those problems.

Relatedly, I vaguely remember hearing about an effort to bring columnar execution into mainline Spark (was it Project Tungsten?). I would be curious to hear about the differences between that effort and Comet's approach.

Thank you @SChakravorti21 for this suggestion. I agree such background would help -- maybe some of the other reviewers (some of whom are spark committers I believe) can offer more specifics and how this relates to mainline spark.

The Velox Paper basically says the JVM implementation of spark is slow so native columnar execution is better, but I don't recall it delving into any more detail

SChakravorti21 · 2024-02-29T16:30:01Z

_posts/2024-02-27-comet-donation.md

+rich query optimizer and planner. However, the execution is delegated to Comet,
+which is significantly faster and more resource efficient than the  JVM based
+implementation.


I assume you might already be planning to add more details, but I'd love to hear about any Comet-specific optimizations that were implemented. Similarly, I'd be interested to hear about DataFusion-specific features that made it a particularly good fit for Comet's implementation (DataFusion's focus on extensibility comes to mind).

I defer to @sunchao / @viirya or one of the other comet authors for this. Maybe it would be a good topic to address in the initial release blog

SChakravorti21 · 2024-02-29T16:32:19Z

_posts/2024-02-27-comet-donation.md

+Comet into the Apache Software Foundation will accelerate its development and 
+grow its community of contributors and users.
+
+[Comet project]: https://github.com/apache/arrow-datafusion-comet


I learned that there is a similar effort to build a Spark-native accelerator using DataFusion: Blaze. Wondering if it deserves a mention as well.

@richox or @yjshen, who worked on Blaze in the past, do you have any suggestions or thoughts on if/how we should mention Blaze in this post?

The last time I checked the Blaze project hadn't had any updates, but it appears that in the last 6 months @richox has been working on it (BTW @richox contributed some excellent improvements to DataFusion as well such a a loser tree for the sorting implementation 🙏 )

Added a link in 919c934

I heard the current Blaze team has proposed donating it to Apache in the past, but I am not sure what the status is.

It is probably unrealistic, but perhaps the projects could converge over time

+1. It would be great if Blaze and Comet can join forces in some way!

alamb · 2024-03-04T07:59:46Z

I plan to publish this blog tomorrow, March 5, unless anyone would like more time to review or has other thoughts. Thank you all for the feedback so far

alamb · 2024-03-06T07:52:21Z

Sorry I forgot to hit the button yesterday. I guess it will be published today March 6

alamb · 2024-03-06T07:52:37Z

Thanks everyone for the help

alamb · 2024-03-06T07:58:36Z

Published at https://arrow.apache.org/blog/2024/03/06/comet-donation/

viirya · 2024-03-06T08:03:45Z

Thank you @alamb !

sunchao · 2024-03-06T17:51:22Z

Thanks @alamb !

alamb added 2 commits February 27, 2024 05:54

[Website]: Add Blog about Comet Donation

e737e84

update wording

1515c4e

alamb commented Feb 27, 2024

View reviewed changes

alamb requested review from viirya and sunchao February 27, 2024 11:00

alamb mentioned this pull request Feb 27, 2024

Write a blog post about the comet donation apache/datafusion-comet#6

Closed

alamb requested a review from andygrove February 27, 2024 11:00

andygrove reviewed Feb 27, 2024

View reviewed changes

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

andygrove reviewed Feb 27, 2024

View reviewed changes

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

andygrove reviewed Feb 27, 2024

View reviewed changes

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

viirya reviewed Feb 27, 2024

View reviewed changes

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

viirya reviewed Feb 27, 2024

View reviewed changes

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

viirya reviewed Feb 27, 2024

View reviewed changes

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

alamb and others added 2 commits February 28, 2024 06:30

Apply suggestions from code review

8ae61a6

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Andy Grove <andygrove73@gmail.com>

Improve figure

70ff88d

sunchao reviewed Feb 28, 2024

View reviewed changes

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

_posts/2024-02-27-comet-donation.md Outdated Show resolved Hide resolved

alamb added 3 commits February 29, 2024 09:22

Implement suggestions from Chao Sun

c512a68

Fix spelling of DataBricks --> Databricks

e58fba0

Add note about working on an initial release

b3d17ce

SChakravorti21 reviewed Feb 29, 2024

View reviewed changes

alamb added 2 commits March 1, 2024 15:25

Swap ascii art for actual image

64bf139

Add reference to Blaze

919c934

alamb mentioned this pull request Mar 1, 2024

minor: Update README.md with system diagram apache/datafusion-comet#148

Merged

kou changed the title ~~[Website]: Add Blog about Comet Donation~~ [Website] Add Blog about Comet Donation Mar 3, 2024

alamb mentioned this pull request Mar 4, 2024

DataFusion weekly project plan (Andrew Lamb) - March 4, 2024 apache/datafusion#9453

Closed

5 tasks

Update date to 2024-03-06

8d7e76e

alamb merged commit d54b3be into apache:main Mar 6, 2024
1 check passed

alamb deleted the alamb/comet branch March 6, 2024 07:52

[Website] Add Blog about Comet Donation #479

[Website] Add Blog about Comet Donation #479

Conversation

alamb commented Feb 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Feb 28, 2024

sunchao commented Feb 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunchao commented Feb 28, 2024 • edited Loading

alamb commented Feb 28, 2024

alamb commented Feb 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Mar 4, 2024

alamb commented Mar 6, 2024

alamb commented Mar 6, 2024

alamb commented Mar 6, 2024

viirya commented Mar 6, 2024

sunchao commented Mar 6, 2024

sunchao commented Feb 28, 2024 •

edited

Loading