Add MariaDB Vector DocumentStore integration by SyedShahmeerAli12 · Pull Request #3565 · deepset-ai/haystack-core-integrations

SyedShahmeerAli12 · 2026-07-08T13:59:38Z

Summary

Closes #2340

Implements a complete MariaDB document store integration using MariaDB 11.7+ native VECTOR support.

MariaDBDocumentStore full DocumentStore protocol (write_documents, filter_documents, delete_documents, count_documents)
Vector similarity search using VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN with MHNSW indexing
Full-text keyword search via MATCH ... AGAINST (IN NATURAL LANGUAGE MODE) on a FULLTEXT index
Haystack metadata filtering converted to JSON_UNQUOTE(JSON_EXTRACT(...)) SQL expressions with parameterized queries
MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy support
DuplicatePolicy support: FAIL (INSERT), OVERWRITE (upsert via ON DUPLICATE KEY UPDATE), SKIP (INSERT IGNORE)
Lazy connection with reconnect on ping failure

Tests

80 tests total all passing:

68 unit tests (mocked DB, no external dependency)
12 integration tests verified against a real MariaDB 11.7 Docker container

CI

Added .github/workflows/mariadb.yml with a MariaDB 11.7 service container, matching the pattern used by pgvector.

How to run locally

docker run -d --name mariadb-haystack \
  -e MARIADB_ROOT_PASSWORD=password \
  -e MARIADB_DATABASE=haystack \
  -p 3306:3306 mariadb:11.7

cd integrations/mariadb
pip install -e .
MARIADB_USER=root MARIADB_PASSWORD=password pytest tests/ -m integration

socket-security · 2026-07-08T14:01:37Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/mariadb@1.1.14

View full report

github-actions · 2026-07-08T14:02:41Z

Coverage report (tavily)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
integrations/tavily/src/haystack_integrations/components/fetchers/tavily
tavily_fetcher.py
Project Total

_{This report was generated by python-coverage-comment-action}

Implements MariaDBDocumentStore backed by MariaDB 11.7+ native VECTOR support with MHNSW indexing and full-text keyword search. - Full DocumentStore protocol: write_documents (FAIL/OVERWRITE/SKIP), filter_documents, delete_documents, count_documents - Vector similarity via VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN - Full-text keyword search via MATCH ... AGAINST (NATURAL LANGUAGE MODE) - Haystack metadata filtering converted to JSON_EXTRACT SQL expressions - MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy - 80 tests: 68 unit + 12 integration (all verified against MariaDB 11.7) - GitHub Actions workflow with MariaDB 11.7 service container Closes deepset-ai#2340

- Make mariadb C extension import lazy so API reference builds without requiring libmariadb-dev in the docs environment - Fix Docker health check to use --su-mysql flag required by MariaDB 11.7 - Add mariadb (LGPL-2.1) to license compliance exclusion list

- Add skip-install=true to default hatch env so docs build does not attempt to compile the mariadb C extension (libmariadb-dev not available in the API reference runner). The pydoc search_path already points to src/ so modules are importable without installation. - Switch service container health check from healthcheck.sh (unreliable in some MariaDB 11.7 images) to mysqladmin ping which is more robust.

…THCHECK The mariadb:11.7 Docker image ships with a HEALTHCHECK instruction. GitHub Actions automatically waits for it when no custom options override it. Custom health-cmd variants (healthcheck.sh, mysqladmin) were failing because the slim runner environment handles them differently.

anakin87

I took a first look and found some points to address and some others to discuss.

anakin87 · 2026-07-21T08:11:41Z

@@ -0,0 +1,10 @@
+# Changelog


this file is auto-generated and updated at release time.
Remove it.

anakin87 · 2026-07-21T08:12:38Z

@@ -0,0 +1,77 @@
+# MariaDB Document Store for Haystack


This should be minimal, like other READMEs in this repo.
See for example https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector

anakin87 · 2026-07-21T08:13:55Z

+# skip-install prevents hatch from installing the project (and thus the mariadb C extension)
+# in the docs environment. The pydoc search_path points to src/ so modules are found directly.


what's the problem here?

mariadb==1.1.14 uses distutils internally, which is removed in Python 3.14 setuptools provides the shim to keep the test env working.

anakin87 · 2026-07-21T08:41:44Z

@@ -0,0 +1,401 @@
+# SPDX-FileCopyrightText: 2023-present deepset GmbH <info@deepset.ai>


For integration tests, we should use classes from haystack.testing.document_store ( DocumentStoreBaseTests and DocumentStoreExtendedTests or their inner classes). Check Pgvector for an example.

anakin87 · 2026-07-21T08:46:14Z

+            except Exception:
+                self._close_connection()
+
+        import mariadb  # noqa: PLC0415


mariadb should be imported at the top of the module

The lazy import is intentional top-level import triggers the C extension which fails on Python 3.14 due to the distutils removal; placing it after the early-return ping check means unit tests with mocked connections never hit it.

anakin87 · 2026-07-21T08:57:19Z

+    blob_mime_type VARCHAR(255),
+    meta JSON,
+    FULLTEXT KEY content_ft_idx (content)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4


We should also set the distance function at table creation.

Also, let's make distance not changeable after table creation:

Declare DISTANCE explicitly. The default is euclidean, and a query using a different distance function than the one the index was built for cannot use the index, it falls back to a full table scan.

See https://mariadb.com/docs/server/reference/sql-structure/vectors/create-table-with-vectors

anakin87 · 2026-07-21T09:02:45Z

+            try:
+                self._cursor.execute(sql, row)
+                written += 1
+            except mariadb.IntegrityError as e:
+                if policy == DuplicatePolicy.FAIL:
+                    msg = f"Document with id '{doc.id}' already exists"
+                    raise DuplicateDocumentError(msg) from e


try: self._connection.begin() self._cursor.executemany(sql, rows) written = self._cursor.rowcount self._connection.commit() except mariadb.IntegrityError as e: self._connection.rollback() msg = "Some documents already exist and policy is FAIL" raise DuplicateDocumentError(msg) from e except mariadb.Error as e: self._connection.rollback() raise DocumentStoreError("Failed to write documents") from e

Doing something like this would ensure both atomic writes and exact numbers for DuplicatePolicy.SKIP

anakin87 · 2026-07-21T09:06:24Z

+        field_name = field.split(".", 1)[1]
+        # JSON_UNQUOTE(JSON_EXTRACT(meta, '$.field')) returns the value as a string.
+        # Cast to numeric types when the comparison value is numeric.
+        base = f"JSON_UNQUOTE(JSON_EXTRACT(meta, '$.{field_name}'))"


this might be prone to SQL injection. Let's make it more robust

anakin87 · 2026-07-21T09:10:48Z

+{where_clause}
+ORDER BY score DESC
+LIMIT ?
+"""


this might also return irrelevant documents if relevant ones are less than top_k
we should also apply matching on WHERE or filtering by score

anakin87 · 2026-07-21T09:14:58Z

+    def __del__(self) -> None:
+        self._close_connection()
+


I'd avoid this. close is sufficient.

SyedShahmeerAli12 · 2026-07-21T11:01:00Z

Addressed all the straightforward review comments. The HNSW index design (create_vector_index flag) and the lazy import approach are still open for discussion

SyedShahmeerAli12 requested a review from a team as a code owner July 8, 2026 13:59

SyedShahmeerAli12 requested review from anakin87 and removed request for a team July 8, 2026 13:59

github-actions Bot added topic:CI integration:tavily type:documentation Improvements or additions to documentation labels Jul 8, 2026

SyedShahmeerAli12 added 2 commits July 8, 2026 19:07

SyedShahmeerAli12 force-pushed the feat/mariadb-document-store-2340 branch from f71d1cd to ff12b6a Compare July 8, 2026 14:09

SyedShahmeerAli12 added 6 commits July 8, 2026 19:14

fix(mariadb): lint errors and test env setup

8afc968

fix(mariadb): remaining lint errors and add libmariadb-dev install

7bc8914

fix(mariadb): fix Python 3.14 distutils, S608/EM102/RUF100 lint

90aea75

fix(mariadb): use Any for cursor/connection to fix mypy union-attr

fd1ae83

anakin87 requested changes Jul 21, 2026

View reviewed changes

SyedShahmeerAli12 added 3 commits July 21, 2026 15:29

fix

b0a7f31

fix

b013ead

fix

0eca31c

SyedShahmeerAli12 requested a review from anakin87 July 21, 2026 11:01

add create_vector_index flag

269eeca

		# skip-install prevents hatch from installing the project (and thus the mariadb C extension)
		# in the docs environment. The pydoc search_path points to src/ so modules are found directly.

		@@ -0,0 +1,401 @@
		# SPDX-FileCopyrightText: 2023-present deepset GmbH <info@deepset.ai>

Uh oh!

Conversation

SyedShahmeerAli12 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

CI

How to run locally

Uh oh!

socket-security Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

Coverage report (tavily)

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SyedShahmeerAli12 commented Jul 8, 2026 •

edited

Loading

socket-security Bot commented Jul 8, 2026 •

edited

Loading

SyedShahmeerAli12 Jul 21, 2026 •

edited

Loading

SyedShahmeerAli12 Jul 21, 2026 •

edited

Loading