Skip to content

introduce Ecto.Adapters.SQLite3.Extension #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 24, 2025

Conversation

aseigo
Copy link
Contributor

@aseigo aseigo commented Jun 18, 2025

This allows type extensions to be registered at runtime via the :ecto_sqlite3, extensions: configuration key, which takes a list of modules that implement the Extension behaviour.

See #166

This allows for an extension as simple as this to support storing and retrieving all the types in the geo library:

defmodule GeoSQL.SpatialLite.TypesExtension do
  @behaviour Ecto.Adapters.SQLite3.Extension
  @geo_types [
    Geo.GeometryCollection,
    Geo.LineString,
    Geo.LineStringZ,
    Geo.LineStringZM,
    Geo.MultiLineString,
    Geo.MultiLineStringZ,
    Geo.MultiLineStringZM,
    Geo.MultiPoint,
    Geo.MultiPointZ,
    Geo.MultiPolygon,
    Geo.MultiPolygonZ,
    Geo.Point,
    Geo.PointZ,
    Geo.PointM,
    Geo.PointZM,
    Geo.Polygon,
    Geo.PolygonZ
  ]

  @impl true
  def loaders(:geometry, ecto_type) do
    [&__MODULE__.decode_geometry/1, ecto_type]
  end

  def loaders(_primitive_type, _ecto_type) do
    nil
  end

  @impl true
  def dumpers(:geometry, ecto_type) do
    [ecto_type, &__MODULE__.encode_geometry/1]
  end

  def dumpers(_ecto_type, _primitive_type) do
    nil
  end

  def encode_geometry(%x{} = geometry) when x in @geo_types do
    case Geo.WKB.dec,encode(geometry) do
      {:ok, data} -> {:ok, data}
      {:error, _reason} -> :error
    end
  end

  #   def decode_geometry(<<len::integer-size(32), wkb::binary-size(len)>> = _data) do
  def decode_geometry(wkb) do
    case Geo.WKB.dec,ode(wkb) do
      {:ok, data} -> {:ok, data}
      {:error, _reason} -> :error
    end
  end
end

On a related note: this is still not sufficient to use with the SQL/MM GIS functions (the ST_* ones in SpatialLite), as while storing WKB works fine, Spatialite has it's own almost-but-not-quite-standard-wkb storage format (a truly daft idea) and has a few other minor incompatibilities such as seemingly not supporting SRID:<number> values in WKT/WKB ... so there's still some work to do to leverage this into full, proper Spatialite support ... but I'm working on that as a separate issue in another repo.

This is sufficient for serialization/deserialization of custom data types, however.

this allows type extensions to be registered at runtime via the
`:ecto_sqlite3, extensions:` configuration key, which takes a list
of modules that implement the `Extension` behaviour.
@aseigo
Copy link
Contributor Author

aseigo commented Jun 18, 2025

It would have been nice to have this configurable on a per-Repo basis, but I was unable to find a (non-hacky) way to get at the repo configuration in the loaders/dumpers functions. If anyone has a bright idea there, please let me know!

@aseigo
Copy link
Contributor Author

aseigo commented Jun 20, 2025

Update: I now have a working SpatialLite implementation using this PR :)

@aseigo
Copy link
Contributor Author

aseigo commented Jun 20, 2025

On further investigation, this is not quite enough ... it works very nicely with Ecto, but any structs that are directly added to a query end up being passed directly to the exqlite driver. It seems it will also need to be instrumented, which may make this change moot .. or not. Will investigate further and follow up then.

@aseigo
Copy link
Contributor Author

aseigo commented Jun 20, 2025

exqlite will require similar modifications. These changes are still required as well, but a similar set is needed in exqlite. So, this is back on the menu! ;) I will be submitting a PR to exqlite in a bit to get that moving as well.

warmwaffles pushed a commit to elixir-sqlite/exqlite that referenced this pull request Jun 20, 2025
…tabase (#333)

Introduces a runtime extension to convert data types into something
that can be serialized into the database.

See elixir-sqlite/ecto_sqlite3#167 and
elixir-sqlite/ecto_sqlite3#166

My primary goal is to be able to use Spatialite from Elixir. With this
PR and the one against `ecto_sqlite3`, it is 95% of the way there. With
these two PRs applied, one can now do things like:

```elixir
defmodule Location do
  use Ecto.Schema

  schema "locations" do
    field(:name, :string)
    field(:geom, GeoSQL.Geometry)
  end
end

MyApp.Repo.all(from(location in Location, select: MM2.distance(location.geom, ^geometry)))
```

.. and it just works, regardless of whether `MyApp.Repo` is point at a
PostgreSQL database with PostGIS enabled, or an SQLite3 database with
Spatialite loaded.

The one remaining annoyance is getting structs back *out* of the
database without going through ecto (which *does* work with the
`ecto_sqlite3` PR!). So, I still have to figure out a nice way for the
user to get back `Geo` structs from raw queries such as:

```
from(location in Location, limit: 1, select: MM2.transform(location.geom, 3452))
```

But otherwise everything Just Works transparently with this change.
@aseigo aseigo marked this pull request as ready for review June 20, 2025 14:32
@aseigo
Copy link
Contributor Author

aseigo commented Jun 20, 2025

Small update: the exqlite PR was merged. With that and this PR, I have a fully operational Spatialite implementation! The same unit tests that test the PostGIS implementation now also work against a Spatialite DB.

If/when this PR is merged, I will publish that library on hex.pm :)

@aseigo
Copy link
Contributor Author

aseigo commented Jun 22, 2025

If you'd like to play around with the Spatialite bits, the repository is here: https://github.com/expothecary/geo_sql

I finally got all the tests working this evening, which I wanted to get done before sharing it around. It needs more unit tests, but they will come. It's pretty neat to be able to use the same code for both PostGIS and Spatialite, so long as one sticks to the MM2, MM3, and Common functions.

But it works rather decently, all things considered.

@warmwaffles warmwaffles merged commit 2fa7fb2 into elixir-sqlite:main Jun 24, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants