feat: indy revocation (prover & verifier)

[JamesKEbert]

This PR adds support for querying the revocation status of credentials and presenting proof of non revocation as the Prover/Holder role in Indy revocation.

For a presentation request, this PR will automatically fetch whether matching credentials are revoked given that the request specifies a non-revocation interval and the credentials are revocable. This is done for display to end users for their decision making or for app machine governance rules.
Additionally, when creating a presentation, the revocation states will be generated for the requestedCredentials automatically as part of the proof creation, if proof of non-revocation is requested and the credentials are revocable.

This work was implemented in accordance with Aries RFC 0441 and Indy HIPE 0011. This PR addresses issues #349 & #382.

This PR is reliant on the merging of an indy-sdk typing fix PR (DefinitelyTyped/DefinitelyTyped#58026) & a corresponding PR for indy-sdk-react-native (hyperledger-archives/indy-sdk-react-native#27).

Additions by @TimoGlastra:

Adds an downloadToFile method that directly download the file from a specified url to a specified path. The implementation is different for Node.JS and React Native, as it is hard to work with binary files directly in JS without touching Node.JS / RN specific API's.

This uses RNFS instead of expo file system. This makes the implementation easier, doesn't require everyone using AFJ to use expo and also doesn't mean we have to switch up any dependencies.

Future outstanding work:

• Addition of revocation tests (fairly problematic as we're only providing the Holder role, but can be supplemented by the AATH).
• In-memory caching of ledger transactions (this specifically will improve performance with revocation).
  • Revocation Registry Definitions
  • Revocation Registry Deltas
• Persistent caching of ledger transactions (this specifically will improve performance with revocation).
• Verification of downloaded tails file hash

A big thanks to other contributors: @TheTreek, @dbluhm, @TimoGlastra, @burdettadam, among many others :)

[TimoGlastra]

Awesome!

Indy SDK React Native 0.1.20 has been released: https://github.com/hyperledger/indy-sdk-react-native/releases/tag/0.1.20

The new Indy SDK types version: https://www.npmjs.com/package/@types/indy-sdk/v/1.16.10

[TimoGlastra]

Really nice work @JamesKEbert. Left a few nits, but overall LGTM.

Only thing is that this addition doesn't include a single test, meaning it's really easy to break something without us knowing. We've had discussions during the WG call on how to approach this. An easy scripts to test revocation with ACA-Py could be one. Another thing I was thinking is whether we can set up revocation in a way where we create a wallet with a revocable credential, export it and then import it everytime we run the tests. We can then mock all ledger responses and run through the flow of revocation. However that may be a bit too much work. It should be trivial to integrate this with the AATH, so maybe that's a good start

[JamesKEbert]

An easy scripts to test revocation with ACA-Py could be one. Another thing I was thinking is whether we can set up revocation in a way where we create a wallet with a revocable credential, export it and then import it everytime we run the tests. We can then mock all ledger responses and run through the flow of revocation. However that may be a bit too much work. It should be trivial to integrate this with the AATH, so maybe that's a good start

Thanks for the review! For testing I'd definitely lean in the direction of first doing the AATH integration, and then following up with scripts for ACA-Py testing or wallet import testing. I'm definitely happy to help with either of those elements, although I admit my Docker-fu is a little lighter than I'd like 😅

[JamesKEbert]

Note, we've opened a PR for the indy-sdk types to resolve the last sneaky type issue that I've found: DefinitelyTyped/DefinitelyTyped#58667

[TimoGlastra]

Seems like revocation is working, but not for all test cases. However most seem related to test cases that test valid formats that do not follow the best practices. We should try to filter those out from running as we don't support them.

To run it yourself:

1. checkout feature/afj-revocation branch out from TimoGlastra/aries-agent-test-harness
2. [optional] If you made changes, update tar files in the aries-backchannels/javascript directory
3. build acapy-main and javascript backchannels
   • ./manage build -a acapy-main -a javascript
4. Run revocation tests
   • LEDGER_URL_CONFIG=http://test.bcovrin.vonx.io TAILS_SERVER_URL_CONFIG=https://tails.vonx.io ./manage run -d acapy-main -b javascript -t @revocation -t ~@wip -t ~@DIDExchangeConnection

I think for all tests that are following RFC 0441 we should add a @RFC0441 tag so we can easily test revocation cases that follow those practices.

Failing tests with unknown errors:

• @T002-HIPE0011, @T013-HIPE0011, @T014-HIPE0011 (very helpful error message)

{"name":"IndySdkError","message":"IndyError(CommonInvalidStructure): CommonInvalidStructure","status":500,"errors":[],"stack":"IndySdkError: IndyError(CommonInvalidStructure): CommonInvalidStructure\n    at IndyHolderService.createProof (/src/node_modules/@aries-framework/core/src/modules/indy/services/IndyHolderService.ts:75:34)"}

• @T006-HIPE0011,@T006.2-HIPE0011 - Proof should be unverified, but is verified
  This one is weird as ACA-Py is giving verified=true. Not sure how that could happen if AFJ is the holder? Is this an issue with ACA-Py?

Output from ACA-Py

{
  "presentation_request_dict": {
    "@type": "did:sov:BzCbsNYhMrjHiqZDTUASHg;spec/present-proof/1.0/request-presentation",
    "@id": "8cd5a7df-0635-4ba8-bcd6-aa7fcae6e737",
    "request_presentations~attach": [
      {
        "@id": "libindy-request-presentation-0",
        "mime-type": "application/json",
        "data": {
          "base64": "eyJuYW1lIjogInRlc3QgcHJvb2YiLCAidmVyc2lvbiI6ICIwLjEuMCIsICJyZXF1ZXN0ZWRfYXR0cmlidXRlcyI6IHsiYWRkcmVzc19hdHRycyI6IHsibmFtZSI6ICJhZGRyZXNzIiwgInJlc3RyaWN0aW9ucyI6IFt7InNjaGVtYV9uYW1lIjogIlNjaGVtYV9Ecml2ZXJzTGljZW5zZV9SZXZvYyIsICJzY2hlbWFfdmVyc2lvbiI6ICIxLjEuMCJ9XX19LCAicmVxdWVzdGVkX3ByZWRpY2F0ZXMiOiB7fSwgIm5vbl9yZXZva2VkIjogeyJmcm9tIjogMTY0NDM1NjE5NCwgInRvIjogMTY0NDM1NjE5NH0sICJub25jZSI6ICI2MzQ1ODM5NDc4ODQzODIyNTI5NDcwNzIifQ=="
        }
      }
    ],
    "comment": "This is a comment for the request for presentation."
  },
  "presentation": {
    "proof": {
      "proofs": [
        {
          "primary_proof": {
            "eq_proof": {
              "revealed_attrs": {
                "address": "21751895109898684064665132381010603351785496039964583518083248344707107214824"
              },
              "a_prime": "35951694140173391187382984975312833053133269398945471140443956866014292443437180157021795855506085234619280222403945504171909454495448538993828950287078921910157120871212232549420607207973326118007426513324799348562424146976063537156906218966399368090477267784614388002291933067090791696790270764774476516047329458255824561370194002372999650164866349325627459102520156614487474466566224604842279287486030546926214884481488293461125152750242215887275481513561875317163951079812241942175232153462987080722844609252508491423173605670561904908328303832529311231483103805738792100983548602445188473322599816788062344895483",
              "e": "182816191046963617115316113252273659784079255429559237547775528711348740757728231488946255140070041184081122940836033415400870903181088610",
              "v": "48942573298623877827018040803166076598059146638876198516324295161079875753053908166945474683207552870190120728284014774283648143323077116351480344970810293185809500344833921949768833959667904351128201631449835011243923986939035589536863143524633301118275951414678187741136397319956236498294420116912190008399873524737566368254336977687374197450102601702839295390696830582691938246145152610347468847009702320766643464570999227099291582226639501500104098879117215077954058429576213722952137561505534608218454253577033949765865465893714450201053623592891792884846237597978471938106483796075750369951519131473531482879205948078901579969295280835810154711679832187415384661663356343354468567361001525634232916797597382022604065028369791359110458000327381778276576326616407537351476936238394176878652914784892291523073454860033486048891209502313559329690266432398882378396003378071448110625478581041610219742075126857660094564",
              "m": {
                "age": "6557787004113693124538755399218385029872350397170895310008224030114218805375297563176921033207841971383804920344741913460763793242426214242693211820398531381336496712161450451744",
                "master_secret": "6435591023206324734353112322772517005078559963369279218329463943267643550477224601965199242031743841443919193635024347873326158195575388024286362997264564940505492382588279602130",
                "dl_number": "2546217642122550478472232298439388459601226041411930259349861674001813118762618443397865340333151956203365606625613317412755644233379468217369795470740931020399698527425348218684",
                "expiry": "6975994867256434384218323298829091107642403219246322267833281640199044585243493218477379952077215003934687513907869878926092511425632310988045583907645650135368106020070640493382"
              },
              "m2": "4350513776990549372192451651957300165486458565819859036512101432784095655711035489224969151103459867910705970963187721051850109591983055171405537667302963837524747177677477510688"
            },
            "ge_proofs": []
          }
        }
      ],
      "aggregated_proof": {
        "c_hash": "85013567076856774462532618607322836223946623513776238687646533288200783680954",
        "c_list": [
          [
            1, 28, 202, 210, 147, 1, 137, 48, 163, 19, 93, 122, 110, 183, 180,
            48, 14, 175, 12, 95, 92, 218, 168, 89, 34, 243, 76, 168, 181, 28,
            181, 122, 2, 20, 7, 214, 16, 227, 3, 193, 112, 13, 117, 145, 21, 28,
            117, 129, 140, 160, 137, 9, 237, 232, 3, 185, 0, 141, 108, 217, 199,
            212, 147, 70, 253, 109, 146, 103, 99, 228, 168, 149, 200, 35, 227,
            154, 76, 255, 169, 27, 217, 30, 128, 190, 86, 56, 148, 192, 230, 87,
            115, 144, 98, 93, 192, 109, 38, 176, 0, 189, 179, 153, 13, 187, 140,
            215, 237, 202, 194, 32, 22, 85, 194, 238, 70, 89, 12, 153, 186, 54,
            133, 163, 238, 250, 110, 214, 180, 221, 142, 5, 198, 51, 181, 153,
            160, 192, 114, 0, 127, 45, 92, 84, 2, 224, 4, 239, 176, 80, 234,
            244, 9, 158, 190, 51, 18, 180, 20, 199, 205, 72, 9, 127, 212, 190,
            59, 243, 242, 131, 115, 36, 239, 222, 61, 2, 220, 237, 119, 205,
            200, 141, 75, 150, 191, 112, 107, 35, 54, 33, 245, 112, 155, 205,
            165, 190, 4, 67, 215, 94, 72, 198, 25, 66, 151, 152, 192, 54, 202,
            65, 246, 169, 46, 71, 142, 170, 229, 249, 75, 219, 98, 64, 194, 82,
            147, 222, 230, 134, 146, 162, 92, 206, 68, 91, 249, 105, 73, 56, 25,
            37, 128, 45, 188, 32, 117, 92, 217, 184, 248, 234, 129, 139, 220,
            10, 231, 18, 20, 219, 251
          ]
        ]
      }
    },
    "requested_proof": {
      "revealed_attrs": {
        "address_attrs": {
          "raw": "9 6 st, Q Ontario Canada, K9O 3R5",
          "encoded": "21751895109898684064665132381010603351785496039964583518083248344707107214824",
          "sub_proof_index": 0
        }
      },
      "self_attested_attrs": {},
      "unrevealed_attrs": {},
      "predicates": {}
    },
    "identifiers": [
      {
        "schema_id": "PMUiB2ovzCTiqYrWrF5Kde:2:Schema_DriversLicense_Revoc:1.1.0",
        "cred_def_id": "PMUiB2ovzCTiqYrWrF5Kde:3:CL:147358:7294",
        "rev_reg_id": "PMUiB2ovzCTiqYrWrF5Kde:4:PMUiB2ovzCTiqYrWrF5Kde:3:CL:147358:7294:CL_ACCUM:d0d039af-ec4e-455c-a481-3a9059967a14"
      }
    ]
  },
  "role": "verifier",
  "connection_id": "4bcfd27a-27b9-4e61-a4d9-d32a4e2867a8",
  "presentation_request": {
    "nonce": "634583947884382252947072",
    "name": "test proof",
    "version": "0.1.0",
    "requested_attributes": {
      "address_attrs": {
        "restrictions": [
          {
            "schema_name": "Schema_DriversLicense_Revoc",
            "schema_version": "1.1.0"
          }
        ],
        "name": "address"
      }
    },
    "requested_predicates": {},
    "non_revoked": { "to": 1644356194, "from": 1644356194 }
  },
  "thread_id": "8cd5a7df-0635-4ba8-bcd6-aa7fcae6e737",
  "updated_at": "2022-02-08T21:36:44.011177Z",
  "initiator": "self",
  "trace": false,
  "presentation_exchange_id": "cedee7a1-fb1a-447f-8fd1-addd73794142",
  "auto_present": false,
  "state": "done",
  "verified": "true",
  "created_at": "2022-02-08T21:36:35.285766Z"
}

 assert not context.credential_verification_dict[context.presentation_thread_id]
      AssertionError

Failing tests due to not adhering to 0441:

• @T005-HIPE0011, @T006.1-HIPE0011

Presentation requests proof of non-revocation with an interval from: '1643750205' that does not match the interval to: '1644959805', as specified in Aries RFC 0441

[TimoGlastra]

Also @types/indy-sdk@1.16.12 has been released

[TimoGlastra]

I'm good with merging this and having it as a experimental feature. Then we can start tracking revocation test in AATH on a daily basis

[JamesKEbert]

I'm good with merging this and having it as an experimental feature. Then we can start tracking revocation test in AATH on a daily basis

Agreed--my only concern is in regards to merging today would be the prototype pollution mentioned, so if that's not something you're able to recreate than I'm in favor of merging.

It's complaining about prototype pollution and that we shouldn't create an object prone to prototype polution

[TimoGlastra]

It's still present: https://github.com/hyperledger/aries-framework-javascript/security/code-scanning/8?query=ref%3Arefs%2Fpull%2F592%2Fmerge+ref%3Arefs%2Fpull%2F592%2Fhead

How to fix:

---

Most JavaScript objects inherit the properties of the built-in Object.prototype object. Prototype pollution is a type of vulnerability in which an attacker is able to modify Object.prototype. Since most objects inherit from the compromised Object.prototype object, the attacker can use this to tamper with the application logic, and often escalate to remote code execution or cross-site scripting.

One way to cause prototype pollution is by modifying an object obtained via a user-controlled property name. Most objects have a special proto property that refers to Object.prototype. An attacker can abuse this special property to trick the application into performing unintended modifications of Object.prototype.

Recommendation
Use an associative data structure that is resilient to untrusted key values, such as a Map. In some cases, a prototype-less object created with Object.create(null) may be preferable.

Alternatively, restrict the computed property name so it can't clash with a built-in property, either by prefixing it with a constant string, or by rejecting inputs that don't conform to the expected format.

Example
In the example below, the untrusted value req.params.id is used as the property name req.session.todos[id]. If a malicious user passes in the ID value proto, the variable todo will then refer to Object.prototype. Finally, the modification of todo then allows the attacker to inject arbitrary properties onto Object.prototype.

let express = require('express');

express.put('/todos/:id', (req, res) => {
    let id = req.params.id;
    let items = req.session.todos[id];
    if (!items) {
        items = req.session.todos[id] = {};
    }
    items[req.query.name] = req.query.text;
    res.end(200);
});

One way to fix this is to use Map objects to associate key/value pairs instead of regular objects, as shown below:

let express = require('express');

express.put('/todos/:id', (req, res) => {
    let id = req.params.id;
    let items = req.session.todos.get(id);
    if (!items) {
        items = new Map();
        req.sessions.todos.set(id, items);
    }
    items.set(req.query.name, req.query.text);
    res.end(200);
});