Benchmarks available on premise ?

[chberti]

Hi there !

I saw a Medium blog post on your project and I find it great.
I would like to use it to benchmark an on premise Spark cluster specifically. You mention Azure Lakehouse in the documentation and in the config parameters (bass).

If I use my own datalake URL and initiate with a already running SparkSession, would it work on my cluster ?

Thank you in advance 😊

[mwc360]

@chberti - there's some parallel work going on right now to make LakeBench more cloud/storage agnostic to support exactly these types of scenarios. It might work but I'd guess something will fail as it was initially released as being very cloud service dependent. It should be another week or two before there's broader support. I'll keep you posted.
#45

[mwc360]

@chberti PR #45 has been merged which removes the dependency on Fabric. If you are using an existing spark session with spark.sql.warehouse.dir configured it should work. If a spark session already exists it will use that, otherwise it will create one with single node config.

LMK as outside of Fabric and a handful of Azure platforms, I've only tested locally in single node config against my local filesystem. thx!

[mwc360]

@chberti LakeBench 0.12.0 is out which may provide what you need. lmk if we can close this!

[mwc360]

Closing as there's now local support and I haven't heard back.

[chberti]

First, I would like to apologize for not answering. Seems like I missed the notifications.

Thank’s a lot for this improvement, it’s a great thing to have simple benchmarks tools available on local stacks.

Have a good day.

Regards,
Charles