INTPYTHON-527 Add Queryable Encryption support

[aclark4life]

Previous attempts and additional context here:

• INTPYTHON-527 Add Queryable Encryption config #318

  • Branched from transaction support and includes only helpers

• INTPYTHON-527 Add queryable encryption support #319

  • In which this bug in libmongocrypt was discovered
  • A fix for that bug in PyMongo is being discussed here.

• INTPYTHON-527 Add Queryable Encryption support #323

  • Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

• Add test for "Encrypted fields found" error (ensure this exception still happens)

• Add check for model schema not matching encrypted fields

• Document key_vault_namespace must be encrypted db

• Document that fields within EmbeddedModelArrayField can't be encrypted

• Document workflow:

  • Develop and test local
  • Deploy and run migrate
  • Run showencryptedfieldsmap and set encrypted_fields_map in auto_encryption_opts
  • Deploy again

[timgraham]

What was your thinking here? Were these steps copied from another project? Wouldn't the start server / mongocryptd commands fail with a suitable exit code if they didn't start?

[aclark4life]

POC developed here: https://github.com/aclark4life/test-supercharge-action. The verify steps can probably come out.

[timgraham]

I guess we should have a separate job for Atlas 8 so we still test with Atlas 7, although it's useful to keep it like this for now so we can see the modifications.

[aclark4life]

Do we care about 7 outside of QE? If not, let's just test 8. Still thinking about going the other direction and supporting query, queryPreview, etc. But in this one case I think it's reasonable to cling to non-preview and versions that support query.

[timgraham]

Yes. In fact, based on the driver policy, we have to support MongoDB 6 until July 2028. I've argued that since Django 5.2 is supported until April 2028, we could make Django 5.2 the last version to support MongoDB 6. This is similar to Django's version support for its built in databases. In any case, it would be nice to finalize and document a MongoDB version support policy.

[aclark4life]

OK in that case to remove as much ambiguity as possible, and to support as much MongoDB as we can, how about if we go back to 7 for QE and document rangePreview. I haven't thought about the MongoDB support policy in the context of this project, but what you propose sounds fine. Maybe open a PR to the docs with that policy defined so we can discuss there and merge.

[timgraham]

Can we use a variable for 8.0.15 so it's not hardcoded in so many places? (I have to research to answer.) Same for ubuntu2204, probably. This action uses ubuntu-latest which is 24.04 I believe.

[aclark4life]

Yes, probably via env https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-variables

[timgraham]

Users may wonder why it says 8.0 when the MongoDB documentation says 7.0. Even if we only test with 8 because of the range/rangePreview issue, I think our implementation should work on 7.

[aclark4life]

True, but I'm still inclined to steer away from Preview and only declare support when a query type makes it out of preview.

[timgraham]

Using os.urandom(96) generates a different key each time Python starts which is problematic in a local project, except when running tests. Need to explain this or use a different approach.

[aclark4life]

There is an admonition about it here: https://django-mongodb-backend--329.org.readthedocs.build/en/329/howto/queryable-encryption/#configuring-the-databases-setting

[timgraham]

My point here (as we've discussed) is that it's not appropriate to use a random key that changes each time the web server restarts, a management command, etc. Should we hardcode a particular bytestring that os.urandom(96) generates?

[aclark4life]

Yes, e.g. https://github.com/mongodb-labs/django-mongodb-cli/blob/284ce655970c9b58149e7469f03fecf3a75882d5/django_mongodb_cli/templates/project_template/project_name/settings/project_name.py#L21 although maybe with line breaks.

[aclark4life]

@timgraham FYI mongocryptd POC updated to include crypt shared: https://github.com/aclark4life/test-supercharge-action/actions/runs/19072288362

[aclark4life]

I would suggest just using the POC:

      # 🧩 Install Mongo Crypt Shared Library
      - name: Install Mongo Crypt Shared Library
        run: |
          curl -sSL -o mongo_crypt_shared_v1.tgz https://downloads.mongodb.com/linux/mongo_crypt_shared_v1-linux-x86_64-enterprise-ubuntu2404-8.2.1.tgz
          tar -xzf mongo_crypt_shared_v1.tgz
          sudo mkdir -p /usr/local/lib/mongo_crypt_v1
          sudo cp lib/mongo_crypt_v1.so /usr/local/lib/mongo_crypt_v1/
          echo "✅ Installed Mongo Crypt Shared Library to /usr/local/lib/mongo_crypt_v1"

[aclark4life]

Probably want to use 8.2.1

[timgraham]

MongoDB 8.2 is a rapid release with EOL March 30, 2026. Isn't it better to test with MongoDB 8.0 which enterprise are likely to be using?

[aclark4life]

That's fine, there was a test failure that "failed less" (libmongocrypt wire version error vs . aggregation pipeline "let" command error) when I upgraded but IIUC those tests are expected to fail, so when those failures are asserted then 8.0 is probably OK (or 7.0 if you want to support "preview" query types or maybe add a matrix as you suggested somewhere.)

[timgraham]

Perhaps the old message could be improved by adding quotes around "auto_encryption_opts" and calling it a parameter. Would that address your concern? I wouldn't call it a setting. OPTIONS is described in Django's documentation as "Extra parameters to use when connecting to the database."

Referencing DATABASES['{self.connection.alias}']['OPTIONS'] in error messages has a precedent https://github.com/django/django/blob/2501958b5127020411df6271445ccfd0906df70e/django/db/backends/sqlite3/base.py#L189-L193

Incidentally, I've coincidentally added a test for this message in a local commit.

[aclark4life]

That's fine. We should print the alias though, because the name will indicate "default" may not be the encrypted database.

[timgraham]

If not sure I understand what you mean by printing the alias (it does appear in both the old message and your suggested message as far as I can tell), but does something like this address your concern: "Tried to create model {app_label.Model} in '{self.connection.alias}' database. The model has encrypted fields but DATABASES['{self.connection.alias}']['OPTIONS'] is missing the "auto_encryption_opts" parameter. If this model should not be created in this database, add or fix a database router."

[aclark4life]

That works, thanks. I was saying that the f" syntax wasn't working and the error message contained self.connection.alias instead of default.