ExShards

This is a wrapper on top of ETS and Shards.

Shards is a simple library to scale-out ETS tables, which implements the same ETS API. Taking advantage of this, what ExShards does is provides a wrapper to use either ets or shards totally transparent.

Additionally, ExShards provides an extended API, with a fresh and fluent interface – more Elixir-friendly. For more information, check out the Extended API section.

Installation and Usage

To start playing with ex_shards you just have to follow these simple steps:

1. Add ex_shards to your list of dependencies in mix.exs:

def deps do
  [{:ex_shards, "~> 0.2"}]
end

2. Since ex_shards uses shards, make sure that shards is started before your application:

def application do
  [applications: [:shards]]
end

Build

$ git clone https://github.com/cabol/ex_shards.git
$ cd ex_shards
$ mix deps.get && mix compile

Getting Started!

Start an Elixir console:

$ iex -S mix

Once into the Elixir console:

# create a table with default options
iex> ExShards.new :mytab
:mytab

iex> ExShards.insert :mytab, [k1: 1, k2: 2, k3: 3]
true

iex> for k <- [:k1, :k2, :k3] do
       [{_, v}] = ExShards.lookup(:mytab, k)
       v
     end
[1, 2, 3]

# let's query all values using select
# we need to require Ex2ms to build match specs
iex> require Ex2ms
Ex2ms
iex> ms = Ex2ms.fun do {_, v} -> v end
[{{:_, :"$1"}, [], [:"$1"]}]
iex> ExShards.select :mytab, ms
[1, 2, 3]

iex> ExShards.delete :mytab, :k3
true
iex> ExShards.lookup :mytab, :k3
[]

# let's create another table
iex> ExShards.new :mytab2, [{:n_shards, 4}]
:mytab2

# start the observer so you can see how shards behaves
iex> :observer.start
:ok

As you might have noticed, it's extremely easy, such as you were using ETS API directly.

Extended API

As you probably have noticed, most of the Elixir APIs are designed to be Fluent, they allow us to take advantage of the pipe operator, making the code more readable and elegant of course.

Because shards implements the same ets API, most of the functions follows the old-traditional Erlang-style, so it is not possible to pipe them. Here is where the extended API comes in!

ExShards.Ext is the module that implements the extended API, and provides a fluent API with a set of nicer and fresh functions, based on the Elixir.Map API. No more words, let's play a bit:

iex> :t |> ExShards.new |> ExShards.set(a: 1, b: 2) |> ExShards.put(:c, 3) |> ExShards.update!(:a, &(&1 * 2))
:t

iex> for k <- [:a, :b, :c, :d], do: ExShards.get(:t, k)
[2, 2, 3, nil]

iex> :t |> ExShards.remove(:c) |> ExShards.fetch!(:c)
** (KeyError) key :c not found in: :t

iex> :t |> ExShards.drop([:a, :b, :x]) |> ExShards.put(:y, "new!") |> ExShards.keys
[:y]

ExShards.Ext is well documented, and you can find the documentation in the next links:

• ExShards.Ext
• API Reference

Distributed ExShards

Let's see how ExShards works in distributed fashion.

1. Let's start 3 Elixir consoles running ExShards:

Node a:

$ iex --name a@127.0.0.1 -S mix

Node b:

$ iex --name b@127.0.0.1 -S mix

Node c:

$ iex --name c@127.0.0.1 -S mix

2. Create a table with global scope (scope: :g) on each node and then join them.

iex> ExShards.new :mytab, scope: :g, nodes: [:"b@127.0.0.1", :"c@127.0.0.1"]
:mytab

# or if you somehow have the nodes clustered already

iex> ExShards.new :mytab, scope: :g, nodes: Node.list
:mytab

# then

iex> ExShards.get_nodes :mytab
[:"a@127.0.0.1", :"b@127.0.0.1", :"c@127.0.0.1"]

3. Now ExShards cluster is ready, let's do some basic operations:

From node a:

iex> ExShards.insert :mytab, k1: 1, k2: 2
true

From node b:

iex> ExShards.insert :mytab, k3: 3, k4: 4
true

From node c:

iex> ExShards.insert :mytab, k5: 5, k6: 6
true

Now, from any of previous nodes:

iex> for k <- [:k1, :k2, :k3, :k4, :k5, :k6] do
       [{_, v}] = ExShards.lookup(:mytab, k)
       v
     end
[1, 2, 3, 4, 5, 6]

All nodes should return the same result.

Let's do some deletions, from any node:

iex> ExShards.delete :mytab, :k6
true

From any node:

iex> ExShards.lookup :mytab, :k6
[]

Let's check again all:

iex> for k <- [:k1, :k2, :k3, :k4, :k5] do
       [{_, v}] = ExShards.lookup(:mytab, k)
       v
     end
[1, 2, 3, 4, 5]

References

• ExShards API Reference: ExShards Docs.
• Shards API Reference: Shards API Reference.
• Blog Post about Shards.

Copyright and License

Copyright (c) 2016 Carlos Andres Bolaños R.A.

ExShards source code is licensed under the MIT License.