Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove /gov/read and /gov/query endpoints #2442

Merged
merged 26 commits into from
Apr 12, 2021

Conversation

jumaffre
Copy link
Contributor

@jumaffre jumaffre commented Apr 9, 2021

Part of #2411

This PR removes the existing /gov/read and /gov/query endpoints which used the Lua interpreter and were mostly used for testing.

Instead, ccf.ledger.Ledger now has two new members get_transaction(seqno) to return the ccf.ledger.Transaction recorded in the ledger at a specific seqno and get_latest_public_state() to return the latest public state. The former is used to check that a specific table was updated by a proposal (hence tracking the proposal completion seqno) while the latter is used to check the final public state (e.g. to check that JWT refresh works).

@jumaffre jumaffre requested a review from a team as a code owner April 9, 2021 15:03
@@ -312,238 +192,6 @@ struct TestNewMember
crypto::Pem cert;
};

DOCTEST_TEST_CASE("Add new members until there are 7 then reject")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This made use of query and read but I decided to scrap it as it is already covered by end-to-end tests.

for table_name, records in tx.get_public_domain().get_tables().items():
if table_name in public_tables:
public_tables[table_name].update(records)
# Remove deleted keys
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the latest state does not include keys deleted keys (value of None in delta), although we do include empty tables.

python/ccf/ledger.py Outdated Show resolved Hide resolved
Comment on lines +539 to +540
for chunk in self:
for tx in chunk:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-critical: This iteration is going to be very slow. Could we rely on the filenames here to scan to the relevant chunk, and parse it in isolation? And even within that, could we skip over other transactions rather than parsing them? Not needed for this PR or current tests, but will likely bite us if we start using this ledger util in more places/on real service ledgers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I should have mentioned it in the comment. I wanted the fetching to still verify the integrity of the ledger and we currently have no way to do that without starting from the very beginning of the ledger. I'll create a separate ticket and add a comment in the docstring.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eddyashton See first item in #2453 for follow-up improvements.

python/ledger_tutorial.py Outdated Show resolved Hide resolved
jumaffre and others added 3 commits April 9, 2021 17:09
Co-authored-by: Eddy Ashton <ashton.eddy@gmail.com>
Co-authored-by: Eddy Ashton <ashton.eddy@gmail.com>
@ghost
Copy link

ghost commented Apr 9, 2021

remove_gov_read_1@23206 aka 20210412.8 vs main ewma over 20 builds from 22883 to 23191

Click to see table
build_id build_number sb_sgx_cft^ sb_sgx_cft_mem sb_sgx_bft^ sb_sgx_bft_mem sb_ws_sgx_cft^ sb_ws_sgx_cft_mem sb_sig_sgx_cft^ sb_sig_sgx_cft_mem tpcc_sgx_cft^ tpcc_sgx_cft_mem tpcc_sgx_bft^ tpcc_sgx_bft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_ws_sgx_cft^ ls_ws_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem CHAMP put (/s)^ CHAMP get (/s)^
22883 20210406.12 24760.1 1.21939e+07 15056.8 3.91947e+07 26974.8 1.27181e+07 4467.19 1.03588e+07 6614.02 8.5332e+07 3427.07 1.78655e+08 25203.1 1.7961e+07 30349.4 1.74367e+07 4180.87 1.45532e+07 2022.51 9.04813e+06 1410.38 7.99955e+06 1.33385e+06 3.58036e+07
22891 20210406.14 27979.4 1.2456e+07 14113.3 4.07676e+07 33262.9 1.29803e+07 4960.09 1.03588e+07 6606.71 8.61185e+07 3328.56 1.76558e+08 25064.9 1.74367e+07 31088.9 1.74367e+07 4269.92 1.48153e+07 2050.34 9.04813e+06 1402.01 7.99955e+06 1.33724e+06 3.63121e+07
22910 20210406.19 27850.8 1.2456e+07 14232.9 3.91947e+07 33865.1 1.27181e+07 4568.09 1.00967e+07 6361.42 8.5332e+07 3397.92 1.66072e+08 24551.1 1.71746e+07 31202.8 1.74367e+07 4159.7 1.48153e+07 1976.23 8.78598e+06 1407.86 7.99955e+06 1.36079e+06 3.62471e+07
22917 20210406.21 27748.1 1.21939e+07 15059.8 3.86704e+07 32329.7 1.29803e+07 4769.64 1.03588e+07 6449.06 8.50699e+07 3318.05 1.64762e+08 25347.1 1.74367e+07 31644.5 1.76989e+07 4189.12 1.48153e+07 1975.65 9.31027e+06 1372.19 9.31027e+06 1.34789e+06 3.66369e+07
22924 20210407.1 24894.8 1.19317e+07 14625.2 3.86704e+07 31052.6 1.27181e+07 4761.97 1.03588e+07 6749.96 8.37592e+07 3314.31 1.57946e+08 26393.1 1.71746e+07 29513.9 1.71746e+07 4184.71 1.45532e+07 2013.06 1.2456e+07 1456.49 7.99955e+06 1.32289e+06 3.5249e+07
22928 20210407.3 26995.6 1.2456e+07 13586.5 3.68354e+07 32980.8 1.27181e+07 4987.4 1.03588e+07 6132.66 8.42835e+07 3114.25 1.93335e+08 23917.9 1.71746e+07 29400.8 1.74367e+07 4362.8 1.48153e+07 1968.44 1.08831e+07 1359.82 7.99955e+06 1.30702e+06 3.61831e+07
22941 20210407.6 23271.8 1.21939e+07 14952.4 3.86704e+07 33153.9 1.29803e+07 4784.25 1.00967e+07 6149.61 8.50699e+07 3325.17 1.62665e+08 25603.2 1.71746e+07 31967.9 1.74367e+07 4225.78 1.48153e+07 1953.69 8.78598e+06 1407.25 8.2617e+06 1.35126e+06 3.58669e+07
22975 20210407.16 24927.9 1.19317e+07 14691.8 3.81461e+07 32626.6 1.27181e+07 4846.57 1.00967e+07 6506.04 8.63806e+07 3299.17 1.91762e+08 24302.8 1.71746e+07 31355.5 1.74367e+07 4404.51 1.45532e+07 2070.24 1.29803e+07 1447.19 8.2617e+06 1.3375e+06 3.5993e+07
23005 20210408.1 27229.9 1.21939e+07 14256.3 3.70975e+07 32560.8 1.27181e+07 4865.39 1.00967e+07 6830.24 8.48077e+07 3420.04 1.58994e+08 24781.2 1.71746e+07 31687.4 1.74367e+07 4019.78 1.4291e+07 1971.12 1.27181e+07 1380.23 7.99955e+06 1.33099e+06 3.66369e+07
23006 20210408.2 24902.5 1.21939e+07 14528.6 3.84083e+07 31561.7 1.2456e+07 4949.59 1.03588e+07 6324.8 8.42835e+07 3403.96 1.55324e+08 25640.5 1.74367e+07 30056.8 1.76989e+07 4389.86 1.45532e+07 2001.44 8.78598e+06 1440.09 9.31027e+06 1.30153e+06 3.59298e+07
23025 20210408.8 27003.6 1.21939e+07 14342.6 3.84083e+07 33791.5 1.29803e+07 4904.22 1.00967e+07 6502.3 8.45456e+07 3353.36 1.65548e+08 23086 1.69124e+07 28869.3 1.74367e+07 4195.23 1.45532e+07 1870.96 7.99955e+06 1381.17 8.2617e+06 1.34311e+06 3.56168e+07
23050 20210408.16 28117.6 1.2456e+07 14800.9 3.63111e+07 31778.9 1.4291e+07 4978.96 1.03588e+07 6267.46 8.61185e+07 3205.02 1.67645e+08 23256.9 1.69124e+07 31172.8 1.7961e+07 3996.57 1.4291e+07 1895.04 8.78598e+06 1388.83 9.31027e+06 1.33741e+06 3.5993e+07
23069 20210408.22 27412.1 1.2456e+07 13953.1 3.81461e+07 31122.8 1.27181e+07 4847.12 1.00967e+07 6566.49 8.5332e+07 3325.82 1.7682e+08 22045.9 1.74367e+07 30587.1 1.76989e+07 4155.28 1.4291e+07 2017.73 9.04813e+06 1445.6 7.99955e+06 1.34364e+06 3.55549e+07
23079 20210408.25 27847.3 1.2456e+07 14288 3.68354e+07 31087.8 1.27181e+07 4866.98 1.03588e+07 6578.06 8.50699e+07 3228.6 1.9019e+08 25983.1 1.71746e+07 28739.5 1.71746e+07 4003.31 1.4291e+07 1974.56 9.04813e+06 1433.32 7.73741e+06 1.3211e+06 3.61193e+07
23094 20210409.1 26993 1.21939e+07 14504.1 3.99811e+07 33739.2 1.29803e+07 4963.63 1.00967e+07 6700.18 8.50699e+07 3385.08 1.68694e+08 23941.4 1.69124e+07 30881.9 1.74367e+07 3825.47 1.40289e+07 1968.98 8.78598e+06 1427.8 8.2617e+06 1.3627e+06 3.5993e+07
23107 20210409.5 26984 1.2456e+07 15451.5 3.68354e+07 32370.8 1.27181e+07 5417.27 1.03588e+07 6572.9 8.50699e+07 3163.35 2.27152e+08 23913.3 1.69124e+07 32587.3 1.74367e+07 4223.19 1.4291e+07 1999.3 9.04813e+06 1504.13 7.47526e+06 1.32779e+06 3.61193e+07
23117 20210409.8 25904.1 1.21939e+07 14215.2 3.89325e+07 30392.5 1.29803e+07 4922.96 1.0621e+07 6551.68 8.48077e+07 2788.25 2.53366e+08 25015.3 1.71746e+07 30914 1.74367e+07 3853.23 1.40289e+07 2003.12 8.78598e+06 1463.83 7.73741e+06 1.33376e+06 3.62478e+07
23154 20210409.20 25455.3 1.2456e+07 14108.8 3.7884e+07 33597.4 1.29803e+07 4926.18 1.0621e+07 6639.66 8.5332e+07 3302.32 1.58208e+08 23577.4 1.7961e+07 31825 1.76989e+07 4019.73 1.4291e+07 2096.98 9.04813e+06 1441.42 7.99955e+06 1.3561e+06 3.63121e+07
23184 20210412.1 26532.9 1.27181e+07 14780.4 3.76218e+07 31976.2 1.27181e+07 4974.16 1.0621e+07 6596.69 8.58563e+07 3167.78 1.93335e+08 24313.6 1.71746e+07 31434.5 1.9796e+07 4196.96 1.45532e+07 2095.93 9.04813e+06 1395.1 7.99955e+06 1.32935e+06 3.65062e+07
23191 20210412.4 28168.2 1.21939e+07 13578.1 3.81461e+07 31048.7 1.27181e+07 5390.42 1.03588e+07 6693.37 8.50699e+07 3244.67 1.88879e+08 22873 1.76989e+07 31199.8 1.74367e+07 3656.45 1.40289e+07 1989.73 1.11453e+07 1408.86 9.31027e+06 1.3404e+06 3.62478e+07

images

@jumaffre jumaffre merged commit 9e0e908 into microsoft:main Apr 12, 2021
achamayou pushed a commit to achamayou/CCF that referenced this pull request Apr 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants