feat: add findex impl - client and server sides #15

Manuthor · 2024-10-18T08:00:30Z

Prerequisites:

cloudproof_findex: add a basic REST client Findex implementation (old Findex Cloud not reusable): feat(findex): add basic REST client instanciation - without Findex Cloud cloudproof_rust#72
- Adapt cloudproof_rust for Findex v7.
  - A POC has been done for Findex v7 slightly changing the rest-interface for the new REST basic interface (without the custom Findex Cloud AuthorizationToken). Old work on Findex Cloud is kept for non regression.
- Replace rust macro with classical Rust impl (vscode does not support macros)
- Make Findex REST Backend instantiable (and usable) in other bindings (FFI, Wasm, pyo3)
- ~~Adapt other cloudproof_* langages repos and make the CI OK~~
findex repository:
- ~~Release the findex 7.0.0to be used incloudproof_rust`~~
- [Findex v6] - Add missing serialization impls for structs Tokens, TokenWithEncryptedValueList and TokenToEncryptedValueMap: fix: Add missing serialization impls for structs Tokens, TokenWithEncryptedValueList and TokenToEncryptedValueMap. findex#88
- ~~Eventually release the 6.0.1 to be used in cloudproof_rust~~
Auth0:
- Create a proper findex-server Auth0 tenant
- Configure it, create a test JWT token for user tech@cosmian.com
- Use this token in tests

Client:

Server:

Closes #1
Closes #4
Closes #6
Closes #9
Closes #3

crate/client/src/findex_rest_client.rs

crate/cli/src/lib.rs

crate/client/src/findex_rest_client.rs

.rustfmt.toml

crate/cli/src/actions/findex/mod.rs

tbrezot · 2024-10-22T07:13:16Z

crate/cli/src/actions/findex/index.rs

+        let config = Configuration::Rest(
+            findex_rest_client.client.clone(),
+            findex_rest_client.server_url.clone(),
+            findex_rest_client.server_url.clone(),
+        );


This seems strange to me.

If I follow you well, upon calling a CLI command, you call this function process with the client object (idk what it is). But since I doubt you serialize and store this client upon returning from the CLI, it means it is instantiated each time the CLI is called. Moreover, it seems to me that only one Findex operation can be performed per CLI invocation. Therefore I conclude that this clone is useless since the caller of process will not reuse this FindexClient.

Please, point to me where my reasoning is flawed.

More generally, I do not understand where the necessity of this FindexClient object and this action mechanism comes from. Imho, the ideally simplistic flow is:

user invokes the CLI with some arguments

the CLI parses its conf/ENV variables to findex the server URL. If it cannot be found, it becomes a required argument.

the CLI parses the provided arguments, returning an error upon reading an invalid one. There are 3 (or 4) possible commands: search, insert, delete (additionally compact). The validity of the rest of the argument depends on the identity of the command, and thus the main body of the CLI is just a match on the parsed command.

after parsing all arguments, if no error was thrown, then it means the parsed command will be performed: the CLI therefore instantiates a Findex object.

The CLI runs the appropriate Findex command with its arguments, and returns the result to the user.

As I said, this is rather simplistic and may need to be developed, for example in order to add a non-Findex command like it seems you are planning to. However, I don't think we need dedicated types with their own proceed functions and to pass around some FindexClient object.

The FindexClient is just a HTTPS Client which is initialized once and then pass to clap actions. About this clone, your reasoning is correct, I've removed the shared reference and passed directly the object to be consumed by actions. Yes the CLI instantiates a HTTPS client each time and does not chain Findex operations (that's what we want).
About the CLI flow, since the 4 operations search, add, delete and compact have different arguments, the 4 corresponding process functions (which are basic functions) are used:

to parse those arguments

to give them properly to Findex API

to display to stdout the Findex output

tbrezot · 2024-10-22T07:23:45Z

crate/client/src/findex_rest_client.rs

@@ -21,11 +21,13 @@ use crate::{
 #[derive(Clone)]
 pub struct FindexClient {
    pub server_url: String,
-    client: Client,
+    pub client: Client,


It seems this is just an HTTPS client. I would suggest not to store it in some FindexClient. This would make everything simpler to read (this HTTPS client has nothing to do with Findex, and could be used by any other protocol built over HTTPS).

FindexClient could be renamed at least I suppose

crate/server/src/config/command_line/db.rs

crate/server/src/core/implementation.rs

crate/server/src/database/redis/mod.rs

tbrezot · 2024-10-22T08:08:00Z

Prerequisites:

* `cloudproof_findex`: add a basic REST client Findex implementation (old Findex Cloud not reusable): [feat(findex): add basic REST client instanciation - without Findex Cloud cloudproof_rust#72](https://github.com/Cosmian/cloudproof_rust/pull/72)
  
  * [x]  Create the `findex-cloud` feature and keep `rest-interface` for the new REST basic interface (without the custom Findex Cloud `AuthorizationToken`). Old work on Findex Cloud is kept for non regression.
  * [x]  Replace rust macro with classical Rust impl (vscode does not support macros)
  * [x]  Make Findex REST Backend instantiable (and usable) in other bindings (FFI, Wasm, pyo3)
  * [ ]  Adapt other `cloudproof_*` langages repos and make the CI OK
  * [ ]  Release the `cloudproof_findex` 6.1.0`to be used in`findex-server`

* `findex` repository:
  
  * [x]  Add missing serialization impls for structs `Tokens`, `TokenWithEncryptedValueList` and `TokenToEncryptedValueMap`: [fix: Add missing serialization impls for structs `Tokens`, `TokenWithEncryptedValueList` and `TokenToEncryptedValueMap`. findex#88](https://github.com/Cosmian/findex/pull/88)
  * [ ]  Release the 6.0.1 to be used in `cloudproof_rust`

* `Auth0`:
  
  * [x]  Create a proper `findex-server` Auth0 tenant
  * [ ]  Configure it, create a test JWT token for user `tech@cosmian.com`
  * [ ]  Use this token in tests

Client:

* [x]  Authentication JWT + PKCS12. Clap action `login` and `logout`

* [x]  Clap action `server-version`

* Findex
  
  * Indexing using REST Findex client interface:
    
    * [x]  with an arbitrary dataset
    * [ ]  with any dataset on clap input
  * [ ]  Searching words

* Encrypt indexed data using KMS:
  
  * [ ]  hybrid encryption KEM - DEM with:
    KEM: RFC5649 (done by KMS with ckms)
    DEM: AES GCM  (done locally by ckms)

* [ ]  Merge CLI in `ckms`

Server:

* [x]  Authentication JWT + PKCS12. Reuse KMS Auth middleware

* [x]  Add enpoints for indexing/searching with findex (server "just" INSERT and SELECT data, does not know the Findex master key)

* [x]  Database trait for all databases impls

* [ ]  Database support:
  
  * [x]  Redis
  * [ ]  Sqlite

* [ ]  Authorization: Multiple clients to index and search on the same datasets. Adapt endpoints to index and search using an `ID`

* [ ]  Convert TOML conf in JSON

UI

* User interface should allowed at least:
  
  * [ ]  Authenticating to the server (JWT at least)
  * [ ]  Indexing arbitrary datasets
  * [ ]  Searching words
  * [ ]  Save encrypted either on a S3 storage or on a managed database (another Redis, ...)

Great summary. Some questions:

how are you planning to introduce the KMS in the decryption process? It seems to me that it would require forking Findex v6 since it hard-codes the use of AES/SHA3 in the implementation of the EncryptedValue. The v7 is more flexible since the crypto component is abstracted behind a trait and can therefore be selected at compile-time.
do you really want to "merge the cli in ckms"?
do you really want a JSON config file ? JSON is overly complex for the task imho
I didn't get your authorization bullet: is it the separation between different indexes?

HatemMn · 2024-10-24T13:01:54Z

@Manuthor some updates of the cosmian all big :

We will keep the findex v6 of cloudproof for this PR
According to Bruno UI is not a priority

HatemMn

The following points were done with my commits, edit them according to nesecssituy

Create a proper findex-server Auth0 tenant
Configure it, create a test JWT token for user tech@cosmian.com
Use this token in tests

.pre-commit-config.yaml

crate/test_server/src/test_server.rs

Co-authored-by: Hatem M'naouer <19950216+HatemMn@users.noreply.github.com>

Manuthor · 2024-10-27T19:40:30Z

The following points were done with my commits, edit them according to nesecssituy
* Create a proper findex-server Auth0 tenant

* Configure it, create a test JWT token for user [tech@cosmian.com](mailto:tech@cosmian.com)

* Use this token in tests

done

* ci: fix * test: authentication failing * test: ignore auth tests * test: re-enable test server auth tests * test: re-enable client auth tests * test: disable client auth tests * test: do not clear db * chore: rename access to permission * refactor: check_permission * test: try mt tests * chore: remove all lints * test: remove dependency_on_unit_never_type_fallback * test: make default server port to 6666 * fix: lint

* feat: encrypt datasets - add routes and write on db * build: docker image * fix: permissions on datasets (add/del/get) * chore: split database traits * fix: reuse config_utils * feat: add list_permissions * fix: strip error conversions * ci: merge workflows between debug and release * docs: add main README * docs: add mkdocs * docs: small changes in mkdocs * chore: reduce verbosity * ci: include windows build by default * docs: fix menu * fix(config): use only toml format (client and server) * test: index a 3MB dataset * fix(redis): use pipe everywhere Redis needs to read/write * feat: support docker on arm cpu * ci: build ARM docker only on nightly build * fix: remove pandoc files * docs: add CLI usage * fix: use develop branch from http_client_server repo * ci(docker build): remove useless arg * fix: PR review * fix(only docs): PR review * test: add/delete/get datasets * fix: Redis writing concurrency (on permissions) and Windows tests (#19) * fix: windows tests * fix: clean permissions redis writing in server side * fix: make dataset redis writing sync * fix(redis): delete datasets * test: make consistent the Redis writing. Use pipe everywhere except transaction on permissions creation * fix: increase server waiting for redis to start * fix: move cli conf in client * feat: use Serializable trait * docs: add rust doc on database traits * fix: avoid a user to lookup who are the more powerful users * fix: database parameters not optional but with default values * fix: reenable cli auth tests * fix: remove useless r2d2 redis-feature and FindexRestClientResultHelper::context * fix: removing useless clippy::blocks_in_conditions * fix: remove uneeded RUSTSECs from deny.toml * ci: fix Github actions warnings * ci: fix Github actions warnings - download-artifact to v4 * fix: remove some todos * fix: remove ahash feature from redis crate

tbrezot

great work, not much to say

Cargo.toml

crate/cli/src/actions/datasets.rs

tbrezot · 2024-12-05T22:38:16Z

crate/cli/src/actions/datasets.rs

+        let mut encrypted_entries = HashMap::new();
+        for (key, value) in &self.entries {
+            encrypted_entries.insert(*key, general_purpose::STANDARD.decode(value)?);
+        }


try with HashMap::with_capacity(self.entries.len()) (and optionally a fold avoid declaring a mutable variable.

Are these entries already encrypted?

Ok thanks.
These entries are indeed already encrypted

crate/cli/src/actions/datasets.rs

tbrezot · 2024-12-05T23:50:24Z

crate/structs/src/encrypted_entries.rs

+        let serialized = encrypted_entries.serialize()?;
+        let deserialized = EncryptedEntries::deserialize(&serialized)?;


There is some serialization tests you can import from crypto core.

crate/structs/src/permissions.rs

crate/structs/src/uuids.rs

crate/server/src/routes/findex.rs

Manuthor force-pushed the findex_v6 branch 3 times, most recently from 3fbfcb8 to f8a13c0 Compare October 18, 2024 08:19

Manuthor commented Oct 18, 2024

View reviewed changes

crate/client/src/findex_rest_client.rs Outdated Show resolved Hide resolved

Manuthor commented Oct 18, 2024

View reviewed changes

crate/cli/src/lib.rs Outdated Show resolved Hide resolved

feat: add findex impl - client and server sides

9027235

Manuthor force-pushed the findex_v6 branch from f8a13c0 to 9027235 Compare October 18, 2024 08:25

Manuthor commented Oct 18, 2024

View reviewed changes

crate/client/src/findex_rest_client.rs Outdated Show resolved Hide resolved

Manuthor requested review from bgrieder, HatemMn, ccorsin and tbrezot October 18, 2024 10:24

feat: add CSV datasets support in CLI

7302140

tbrezot reviewed Oct 22, 2024

View reviewed changes

HatemMn requested changes Oct 25, 2024

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

crate/test_server/src/test_server.rs Show resolved Hide resolved

crate/test_server/src/test_server.rs Show resolved Hide resolved

Manuthor force-pushed the findex_v6 branch from 2c6f243 to 3271a1a Compare October 27, 2024 04:47

fix: rename IndexAction to AddAction and flatten Findex actions

f45e876

Manuthor force-pushed the findex_v6 branch from 3271a1a to f45e876 Compare October 27, 2024 05:18

Manuthor and others added 2 commits October 27, 2024 06:19

Update .pre-commit-config.yaml

fcf2bed

Co-authored-by: Hatem M'naouer <19950216+HatemMn@users.noreply.github.com>

fix(auth0): use proper findex-server token

fb190a6

test: add JWT authentication tests

7fa5dd2

This comment was marked as off-topic.

Sign in to view

Manuthor added 5 commits October 30, 2024 10:55

feat: add Delete Findex operation

f0afc9d

fix: remove Secret for now

818c300

fix: prepare sqlite impl

248ef34

feat: add authorization access mechanism

98936b5

feat: revoke role

0b367d6

Manuthor and others added 3 commits November 3, 2024 08:01

docs: cli access

147efd1

chore: rename FindexClient to RestClient

e7dc054

tbrezot mentioned this pull request Dec 5, 2024

feat: encrypt datasets - add routes and write on db #17

Merged

Manuthor marked this pull request as ready for review December 5, 2024 12:11

Manuthor added 2 commits December 5, 2024 13:37

fix: remove useless .rustfmt.toml

dd3810a

fix: Cargo.toml cleanup

2e27b85

tbrezot reviewed Dec 6, 2024

View reviewed changes

Manuthor added 5 commits December 6, 2024 10:14

fix: bad links

278fa7e

fix: clean useless comments in rust doc

4e61e9c

fix: in datasets actions, use &[Uuid] instead of Vec<Uuid>

68899c5

fix: uniformize in clap action all process() to run()

58d06bc

fix: remove console

fa8274e

ccorsin removed their request for review December 6, 2024 15:08

fix: PR review

f9fb44e

Manuthor merged commit 4b89c2f into develop Dec 9, 2024
12 checks passed

Manuthor deleted the findex_v6 branch December 9, 2024 14:24

		let serialized = encrypted_entries.serialize()?;
		let deserialized = EncryptedEntries::deserialize(&serialized)?;

feat: add findex impl - client and server sides #15

feat: add findex impl - client and server sides #15

Uh oh!

Conversation

Manuthor commented Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tbrezot Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

Manuthor Oct 27, 2024

Choose a reason for hiding this comment

Uh oh!

tbrezot Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

Manuthor Oct 27, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tbrezot commented Oct 22, 2024

Uh oh!

HatemMn commented Oct 24, 2024

Uh oh!

HatemMn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Manuthor commented Oct 27, 2024

Uh oh!

This comment was marked as off-topic.

Uh oh!

tbrezot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tbrezot Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

Manuthor Dec 6, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tbrezot Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Manuthor commented Oct 18, 2024 •

edited

Loading