Can't compile and run on Macpro M1 ?

[opvexe]

a large number of interfaces are not implemented:

# github.com/SnellerInc/sneller/usock
usock/conn_other.go:28:7: Implemented redeclared in this block
        usock/conn.go:32:7: other declaration of Implemented
usock/conn_other.go:34:6: Fd redeclared in this block

[ramondeklein]

Note that Sneller relies on AVX-512 instructions and the M1 doesn't emulate these when running in Rosetta 2. So even if you're able to compile it, you won't be able to run it.

[dvaldivia]

I think it would be super useful for development to have sneller compile for the newer apple hardware that uses Apple Silicon, this way developers will be able to test and adopt sneller. Perhaps introducing support for NEON instructions?

[ramondeklein]

There are several reasons why we don't support Apple Silicon:

1. SIMD on Apple Silicon is very limited compared to AVX-512 and we are not able to get the same performance benefits.
2. Rewriting all hand-crafted AVX-512 code to another SIMD instruction doubles the amount of engineering efforts, while only providing very small benefits.
3. Testing Sneller can be done on an AWS with little effort. Other clouds that offer AVX-512 machines can be used as well (i.e. Azure).
4. Using Apple Silicon hardware isn't a realistic platform for running Sneller in production environments.

We are planning to provide an environment where users can test-drive Sneller. It will allow to ingest data into our cloud-based solution and you will be able to query it for some time. Data will be purged automatically after some days. The most difficult part of this is to prevent abuse of this test-environment.

[gedw99]

I really wanted to try this out too. Bummer that the SIMD for other CPU ISA is not happening.

@ramondeklein @philhofer
Would you be open to contributions ?

The golang team have an issue for portable SIMD: golang/go#58610

[philhofer]

Would you be open to contributions?

Certainly!

The bulk of the porting that needs to happen is translation of about 36,000 lines of x86 assembly in the bytecode engine.

[philhofer]

I should also mention we're busing working on launching a "playground" interface that would let you upload data and run queries against our cloud infra for free in order to get a sense of our capabilities; hopefully I can tell you guys more in a few days!

[gedw99]

That sounds awesome . Thanks for the heads-ups

[gedw99]

Would it be possible to run locally without any SIMD acceleration then ? So I can easily test without the playground

[frank-sneller]

Regarding ARM support, the graviton3 servers on AWS now offer ARM SVE (Scalable Vector Extention) support which is pretty similar architecturally to AVX512, but a complete different syntax (and with varying capabilities).

A longer term target would be to support this as well, but with over 300 primitives right now, that is a significant effort (although we can probably perform part of the "translation" automatically).

[gedw99]

You read my mind .. I was wondering about that too .

still would be cool to just run without any acceleration for testing and contributing

[frank-sneller]

Yeah, a portable golang (non-assembly) implementation would be nice to have -- it would run at much lower speeds of course which would be tolerable for GB-sized workloads but unbearable for TB-sized workloads.

[gedw99]

It’s just for testing / contributions. So perf can be sucky. I can do perf testing in the playground.

Idea.. make easy for a developer to test locally and then test via the playground. Maybe a CLI with a Context argument that is passed in. Would then be nice on ramp for devs maybe

[dvaldivia]

Maybe even force to pass a flag locally to convey to developers they are not getting any acceleration

[gedw99]

Good idea. So it’s clearly intentional

[frank-sneller]

Idea.. make easy for a developer to test locally and then test via the playground. Maybe a CLI with a Context argument that is passed in. Would then be nice on ramp for devs maybe

This time you read our minds ... 😄 as a quick preview checkout the following for a (revamped) local demo on some sample JSON data (but this does require local AVX512 support, eg. an c6i instance on AWS): https://asciinema.org/a/eOjVUwlA7ZYXTGtC6PpsupR2O

Also, as mentioned by @philhofer we are close to launching a playground where you can, amongst others, create a table in no time from a set of presigned S3 URLs on some existing objects in S3 -- stay tuned for that !

[gedw99]

hey @frank-sneller

zion is Ion :)

Will def try it out and feedback.

really great stuff !!

[frank-sneller]

Correct, under the hood it is all standard ion stuff (and our SQL dialect is based on PartiQL, which is a nice fit with ion).

[philhofer]

Hi all,

Just a (belated) follow up to this issue: you can try out sneller for free in your browser on our playground.