Nightly Tests: compare2algod in addition to the regular tests

[tzaffi]

Nightly Tests

A test that asserts that relevant new functionality in the unlimited assets feature branch of go-algorand has not been merged. This assertion is important to let us know that certain functionalities, need to be refactored as soon as possible.

algod v. indexer Swagger

make indexer_v_algod

Introduce a nightly job that:

0. Gets triggered at 3:37 am UTC

1. Runs all the usual Circle tests against the develop branch

2. Loads algod's swagger from the latest master branch commit, and indexer's swagger from the latest develop commit into Python dict's

3. Computes the followingjson_diff's between the models in those swaggers:

• a "dropped" diff that shows elements in algod that are no longer in indexer
• a "full" diff that shows all differences

4. Compares these diffs against expected diffs (i.e. computes 2 diffs of diffs)

5. Fails if either of the diff of diffs is non-empty

Additional Changes

1. Slight changes to response type descriptions of TealValue and global-state-schema to align with algod
2. Re-assignment of postgres docker external ports in an attempt to stop clashing with other sandboxes

FAQ

Q: What does this job not do?

It doesn't connect to Slack or anywhere else outside of circle. I don't have the perms to configure that.

Q: How can I see if the test passed or failed?

Look for the circleci_build_and_test_nightly workflows running in Indexer's Circle Pipeline

Q: What does it mean that the nightly test failed because of algod v. indexer swagger?

It means that either some change in indexer/develop::HEAD or some change in the go-algorand/master::HEAD caused an unexpected diff between the swaggers. Typical reasons are:

• a new feature got introduced into algod's API
• a new feature got introduced into indexer's API
• less likely: a change to the expected outputs under parity/reports or change to swagger locations or format

Q: The nightly test failed because of algod v. indexer swagger. How can I fix that test?

In the most likely case that a new feature was added to algod:

1. decide if this feature needs to be supported by indexer. If so? Create a ticket.
2. run the test locally and replace the files parity/reports/*.yml by their newer versions parity/reports/*_OUT.yml
3. commit these new versions and the test will pass the next circle run (assuming there hasn't been an additional breaking change)

In the second most likely case that the feature was added to indexer and it's not a bug, just follow steps 2 and 3 above.

Summary of changes

Circle

• .circleci/config.yml - add Circle workflow circleci_build_and_test_nightly against the developer branch (currently on this branch to prove its working, but will re-point to developer upon final approval)
• Makefile - new commands for Circle
• misc/requirements.txt - pytest, pyaml and gitpython

Swagger Comparison Python Test under misc/parity/

• misc/__init__.py removed imports... TBD if this works
• misc/parity/.gitignore
• misc/parity/__init__.py
• misc/parity/json_diff.py - main logic
• misc/parity/test_json_diff.py - unit tests for json_diff.py
• misc/parity/reports/algod2indexer_dropped.yml - report of all UNEXPECTED algod features which were dropped by indexer
• misc/parity/reports/algod2indexer_full.yml - report of all UNEXPECTED indexer vs. algod discrepancies
• misc/parity/test_indexer_v_algod.py - test that compares indexer vs. algod swagger, outputs a report, and compares that report with pre-computed reports. Fails when there are changes between new report and previous.

Secondary changes

• .gitignore - ignore a common mac file
• api/indexer.oas2.json - small fixes that were suggested by the comparison test
• cmd/block-generator/docker/docker-compose.yml - change external docker postgres port to avoid conflict with other local sandboxes
• docker-compose.yml - change external docker postgres port to avoid conflict with other local sandboxes
• misc/docker-compose.yml - change external docker postgres port to avoid conflict with other local sandboxes

Autogenerated

Anything under api/generated as well as api/indexer.oas3.yml can be ignored

Example Circle Nightly Runs

Passing

NIGHTLY test

Notice above that it ran against the at the time latest go-algorand commit

Failing

NIGHTLY test

Test Plan

Almost everything is just a test, so verify on Circle that the tests are running correctly. Since the only changes in the auto-generated files were triggered by description changes, there should be no functional changes.

[winder]

could you move the parity directory into misc? Other than, and removing future, I'm ready to approve.

[tzaffi]

could you move the parity directory into misc? Other than, and removing future, I'm ready to approve.

👍

In case I ever need some AST parsing thing like this, I've consecrated it all in this gist

[winder]

LGTM