change(ci): Generate testnet checkpoints in CI

[teor2345]

Motivation

We want to automatically generate checkpoints in CI on testnet.

Closes #6367

Solution

• Run a full sync on testnet on every merge to main, and when the cached state is missing, because it takes about 2 hours to run
  • Also allow it to be run manually
  • This is the same way that the mainnet full sync runs
• Use that cached state to generate checkpoints on testnet
  • Update the cached state for the next job
  • Add a Docker entrypoint for testnet full sync
• Simplify the mainnet and testnet full sync environmental variables
• Update the documentation for the new checkpoint generation and release process

Review

This is a routine change.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

Re-order the jobs so the dependency graph is clearer - full syncs should all be near the top after the build
https://github.com/ZcashFoundation/zebra/actions/runs/4814515102

[teor2345]

Manual testing:

The mainnet checkpoints from CI are:

2023-04-27T06:51:33.324917Z  INFO spawned zebra-checkpoints connected to zebrad, checkpoints should appear here... network=Mainnet zebra_rpc_address=127.0.0.1:36547 zebra_tip_height=Height(2065993) last_checkpoint="2056451"



2056577 0000000001199c2eb1236810fa6879260b0c50a125924d1dc45af9c49d447d3d
2056704 000000000066feff4b9546c951f275f3c3c20a3dc9d14b35ee7954da53bb9baa
2056851 0000000000abed6ba089f09c03beb07e971bd4c633425ca95e4310e2d24dd647
2056988 0000000000ae80af35dc4eb2a6ff9874ff91b6c4c3631e018ee3bdde9f5f303a
2057130 0000000001007b090b605356ced4759600f521348f78d3ca1de390098bbd8ca0
2057290 00000000010156ad40dfef51fb4befaea763b4b1e7e7ca7e34d777574b892de1
2057417 0000000000e477b024f4da7364520f100b609c5d2bef03c9b15e4d00686478b4
2057541 00000000009d0483b526a6541b47addb4c804988557d778e4a4c5af265c09c50
2057680 0000000000e086693f67ee02b7dadba61d20ec9e2d0aa827d9afe1a1e1fdd8c5
2057817 0000000000753963810a57c29545a43c5fd7f064f9741c7f33d790dc14b538ff
2057950 00000000002dbde66c131fafb301839898c211daf1128c98dfc2b47fa48dc9e1
2058080 00000000003e08e0b5898e968733450e69a3887519fd5e81945fc8b295103bf5
2058224 000000000000da50b842f7711d345fb13a4aa76bf43bfb715328afc644ee5c1e
2058457 00000000002fd562ace4d78c55e6b1ea1ff0ea9138a5cebab6ea86ab3ceecc04
2058857 0000000000a1728b5cc7b5e52ed23721cfa904ab461828e607cdd625e2498ea4
2059257 00000000002e9af5b1b2c65df8025b9d3ccd278a9369736bfe82484819f10913
2059602 0000000000053ff3825fc458fe547dc307c0226bb598ff34cfb43ed6491dea6e
2059914 0000000001072835af053aba28cb72e78d6ab1dc5f40666abaf00ada27aebb6d
2060243 00000000010df595ef74005d55cbdc39a73469433fbdda9c05034999aea5245d
2060568 00000000003f3ece1ea2bff0ead37174502d97a7603d36f17ce73da199cc563d
2060886 00000000008d96ce1b1e2d12b8f18d8b23f2024a5ba435e3a22af45036370715
2061143 0000000000d95b089ea220529e2f9e55c9bbaa67b08f17110c277c15e3d2fd66
2061314 00000000007a6ea29131c4680f47e6cb8a38e11a2fdc5b0117dfddaa2f40b9fe
2061485 0000000000dc9fa988d0cb314264bef29e14f13b46c13525c85f617f32f044d8
2061649 00000000009ae4c917d6d0d2f4f45b603a4a89c8488e793657c7d3c111216ed2
2061882 0000000001230cd5eea5b60da086c1733f1ea36afad17d550edf3e63767676a1
2062087 00000000007d706ee89f1edb5ed09f25d96ad784d393fa3b53609c64fb1916d2
2062262 000000000194ab98bf8d7a2d7ebfebc039540077fc74ac9de783171a3b2b4389
2062439 0000000001327f3e10d573b05680c5e3a0b7475048caaa45f1bad58a9cb18e45
2062610 0000000000d77640fade5adf74a5daf435671c143686707254e828e9a74938dc
2062781 0000000001395f9c9d4f865808cbe135e7f198a17bdb7a0170c7fb45f03ba994
2062934 0000000000488ce3d5167ae77aaac6982a4dcd1fdfd216e8ab581a6c6e53077d
2063100 000000000039269c1d411cc0ed1379fc5077f4a1c4ba525dea5e8866decb6b52
2063238 0000000000381f0ecccf04ed5fcf810909c3ff6ca6835c5f756532dadc469718
2063374 00000000008d71e65488fa5642b532010fde17c8642b99d19f6601fc411f1968
2063502 000000000046981b37102dc9eefdd9d510868deedcf91f283dd69c9c617bf4e0
2063639 00000000008e4841931f637058d6fb5121843220ae33d5e6c6c0f02c297fe537
2063745 000000000133247950739ae3d47bd0577fce5445980f32450bf1587065bbb941
2063862 0000000000a33fddcfbc654c485999e40062381ff0a510a9f8359f87f0e1ce73
2063972 0000000000ae6bfc5c42b4a82fad5215f9df51f19360b283fb60fbd49ba27851
2064087 000000000144a1e2a1c29645cbafb60990b7c04f9957ca916d61d8dac6965407
2064212 00000000015dbbfced4b168e2ea2ba6f7d6a8406d32f098d95b24bf9bd451d8a
2064333 0000000001b295913877cdeee5fee31e3840a51346c1d81752f1d3556dd88f2b
2064457 00000000000237a5a7621e5a5be971cb4760ca703f8d20230eba0d1142fd730c
2064583 0000000000642da0b1c4a98e84c8b58979372789adbd9e6ac142b1fb1e2f7db5
2064710 000000000208771fa71d05ea147431ebac0ceb9c7f498205e63a927a22254e74
2064825 00000000001057605d4fa38cb516b86e96591a12612cecf42ad687582cbad0b3
2064970 00000000016c6c7cd920f5849ec3de076d148a0fc10dd5b041a2d0e71822a428
2065080 00000000012e3ff5533b2b51a36d020d837ece8260dca11f934ff4c41463debe
2065275 00000000005189ebdd1b44fcb4fd564b5ddd0dd7f9078c51609e45bef8ba8789
2065527 0000000000df9fae7530382bd02abb9586e56c883d6fd25233b988a5b4bee636
2065766 000000000181d5427966dc6d4a6dcd7934a681b329cbd0b5054c9a5d3efec9a0



2023-04-27T06:54:33.315987Z  INFO finished generating Zebra checkpoints network=Mainnet zebra_tip_height=Height(2065993) last_checkpoint="2056451"

https://github.com/ZcashFoundation/zebra/actions/runs/4816455207/jobs/8576823182#step:8:265

I manually generated mainnet checkpoints using my local zcashd instance:

$ zebra-checkpoints-release --last-checkpoint $(tail -1 zebra-consensus/src/checkpoint/main-checkpoints.txt | cut -d" " -f1)
zebra-checkpoints launched                                                                           
Command-line arguments: Args { backend: Zebrad, transport: Cli, cli: "zcash-cli", addr: None, last_checkpoint: Some(Height(2056451)), zcli_args: [] }
Fetching block info and calculating checkpoints...


2056577 0000000001199c2eb1236810fa6879260b0c50a125924d1dc45af9c49d447d3d
2056704 000000000066feff4b9546c951f275f3c3c20a3dc9d14b35ee7954da53bb9baa
2056851 0000000000abed6ba089f09c03beb07e971bd4c633425ca95e4310e2d24dd647
2056988 0000000000ae80af35dc4eb2a6ff9874ff91b6c4c3631e018ee3bdde9f5f303a
2057130 0000000001007b090b605356ced4759600f521348f78d3ca1de390098bbd8ca0
2057290 00000000010156ad40dfef51fb4befaea763b4b1e7e7ca7e34d777574b892de1
2057417 0000000000e477b024f4da7364520f100b609c5d2bef03c9b15e4d00686478b4
2057541 00000000009d0483b526a6541b47addb4c804988557d778e4a4c5af265c09c50
2057680 0000000000e086693f67ee02b7dadba61d20ec9e2d0aa827d9afe1a1e1fdd8c5
2057817 0000000000753963810a57c29545a43c5fd7f064f9741c7f33d790dc14b538ff
2057950 00000000002dbde66c131fafb301839898c211daf1128c98dfc2b47fa48dc9e1
2058080 00000000003e08e0b5898e968733450e69a3887519fd5e81945fc8b295103bf5
2058224 000000000000da50b842f7711d345fb13a4aa76bf43bfb715328afc644ee5c1e
2058457 00000000002fd562ace4d78c55e6b1ea1ff0ea9138a5cebab6ea86ab3ceecc04
2058857 0000000000a1728b5cc7b5e52ed23721cfa904ab461828e607cdd625e2498ea4
2059257 00000000002e9af5b1b2c65df8025b9d3ccd278a9369736bfe82484819f10913
2059602 0000000000053ff3825fc458fe547dc307c0226bb598ff34cfb43ed6491dea6e
2059914 0000000001072835af053aba28cb72e78d6ab1dc5f40666abaf00ada27aebb6d
2060243 00000000010df595ef74005d55cbdc39a73469433fbdda9c05034999aea5245d
2060568 00000000003f3ece1ea2bff0ead37174502d97a7603d36f17ce73da199cc563d
2060886 00000000008d96ce1b1e2d12b8f18d8b23f2024a5ba435e3a22af45036370715
2061143 0000000000d95b089ea220529e2f9e55c9bbaa67b08f17110c277c15e3d2fd66
2061314 00000000007a6ea29131c4680f47e6cb8a38e11a2fdc5b0117dfddaa2f40b9fe
2061485 0000000000dc9fa988d0cb314264bef29e14f13b46c13525c85f617f32f044d8
2061649 00000000009ae4c917d6d0d2f4f45b603a4a89c8488e793657c7d3c111216ed2
2061882 0000000001230cd5eea5b60da086c1733f1ea36afad17d550edf3e63767676a1
2062087 00000000007d706ee89f1edb5ed09f25d96ad784d393fa3b53609c64fb1916d2
2062262 000000000194ab98bf8d7a2d7ebfebc039540077fc74ac9de783171a3b2b4389
2062439 0000000001327f3e10d573b05680c5e3a0b7475048caaa45f1bad58a9cb18e45
2062610 0000000000d77640fade5adf74a5daf435671c143686707254e828e9a74938dc
2062781 0000000001395f9c9d4f865808cbe135e7f198a17bdb7a0170c7fb45f03ba994
2062934 0000000000488ce3d5167ae77aaac6982a4dcd1fdfd216e8ab581a6c6e53077d
2063100 000000000039269c1d411cc0ed1379fc5077f4a1c4ba525dea5e8866decb6b52
2063238 0000000000381f0ecccf04ed5fcf810909c3ff6ca6835c5f756532dadc469718
2063374 00000000008d71e65488fa5642b532010fde17c8642b99d19f6601fc411f1968
2063502 000000000046981b37102dc9eefdd9d510868deedcf91f283dd69c9c617bf4e0
2063639 00000000008e4841931f637058d6fb5121843220ae33d5e6c6c0f02c297fe537
2063745 000000000133247950739ae3d47bd0577fce5445980f32450bf1587065bbb941
2063862 0000000000a33fddcfbc654c485999e40062381ff0a510a9f8359f87f0e1ce73
2063972 0000000000ae6bfc5c42b4a82fad5215f9df51f19360b283fb60fbd49ba27851
2064087 000000000144a1e2a1c29645cbafb60990b7c04f9957ca916d61d8dac6965407
2064212 00000000015dbbfced4b168e2ea2ba6f7d6a8406d32f098d95b24bf9bd451d8a
2064333 0000000001b295913877cdeee5fee31e3840a51346c1d81752f1d3556dd88f2b
2064457 00000000000237a5a7621e5a5be971cb4760ca703f8d20230eba0d1142fd730c
2064583 0000000000642da0b1c4a98e84c8b58979372789adbd9e6ac142b1fb1e2f7db5
2064710 000000000208771fa71d05ea147431ebac0ceb9c7f498205e63a927a22254e74
2064825 00000000001057605d4fa38cb516b86e96591a12612cecf42ad687582cbad0b3
2064970 00000000016c6c7cd920f5849ec3de076d148a0fc10dd5b041a2d0e71822a428
2065080 00000000012e3ff5533b2b51a36d020d837ece8260dca11f934ff4c41463debe
2065275 00000000005189ebdd1b44fcb4fd564b5ddd0dd7f9078c51609e45bef8ba8789
2065527 0000000000df9fae7530382bd02abb9586e56c883d6fd25233b988a5b4bee636
2065766 000000000181d5427966dc6d4a6dcd7934a681b329cbd0b5054c9a5d3efec9a0
2066079 0000000000a1d83f40a6ca1927f8276ba8ed06759b914369c39c0d1ccd61f4b9
2066402 000000000127c02c3a2f1547c8d83d42103131bd4607d0f18ae418384cf827bc

The testnet checkpoints from CI are:

2023-04-27T08:58:58.101873Z  INFO spawned zebra-checkpoints connected to zebrad, checkpoints should appear here... network=Testnet zebra_rpc_address=127.0.0.1:46323 zebra_tip_height=Height(2321586) last_checkpoint="2310400"



2310800 006b01ff741053041bda472af078224f3e9a07c477feda26db080c6a1f1366a1
2311200 004e43f27a39ac05bcaab5b0686bf10f7dd5af1c193539465f3e0b1eebbfd3da
2311600 0106d2e90b29d32bb4eb327cea6a336fc4884c33397511a568e326bc3657836e
2312000 00641e2f406b38f9d306fd83a8507611a253e7d09e42364e202972605e45fb4b
2312400 00575ea8a9e0688a27fcba9ed89cde7dd88e6cf2546cf908f4e340f9c8fc6a52
2312800 002a95e805c3a1630b8262c67aedb257743ef87b2ff5f2e87f4dea3b2c5829a1
2313200 004fa09d52f3cc1e5b829c4dec72dd0db6ece8a79cd4178f4a7819bf48fd456a
2313600 004ad0407803c2fd87058479d6eec3918acb8c463d1a9630bc11fb1794bdfbc0
2314000 000e92ddc18df503b738bd0c23a46c6862ad30f99452edb3f1fcd93af405325d
2314400 0060385f4943f0e409d0d020ca2686ee1961dfa82f72255b9c296aba45e3546e
2314800 000b3e0cdbba7dbf21f246a092f665136fc8966c5546a7b88e1a67a19c9b7ae9
2315200 003584a3be348692c69f786451a2d856ca4ab2565f64649cccdeb810f107fee3
2315600 00e20ed39fda7da892425ab3f31f6ab9945a29021cbe0a167840b8c7c01c3459
2316000 0013b21a75d41c2982e9d3bbacc4fa613bab66f33180c08cdb0767f5aad95ce8
2316400 00937b033ea9dabbe0467f787319b3e2e7769d9b0142828698783066b8d63b8c
2316800 0042afb0d43eb5cd4d73eeaa69d69b24c97cf8627a4a64741572ca1cb31bde66
2317200 002cb3d799234fd040267d6bb66986aefab76aaec1001112a68b3d350ab84d1c
2317600 00702b6d85769a72329cbac8fa6bb42e3ca04fc3b6f462d8e01215030d4cfeca
2318000 0007ee10ac7bc129ad80aad7e773fe761b141ccec1dabaa506d6c0db82f1e1a6
2318400 003245c3a6132228e5801689513bdd9f7760637d9b00cafe081cf44d48c0bf97
2318800 006d7d609ee491f02f71f5519aa9b3353120cb1c90339f3e675015d1620af76e
2319200 0024258df95e9c78cc40bce9431789e1d294f1ae5980f037a9dcfb65224f2384
2319600 0043eb6699f17b2fa8250ba5ee11931a9056ec0f8119ab85a40aa427cdf973d6
2320000 001e49cf9fbb81c26a1c5287839d40759be3c0ab0563717c2a5a945f85f04238
2320400 006c0fb6b6f9142dfb759360d04a84a0cc8492d15af5d4236f6cf933bed605cb
2320800 0036bacd8d916352350b7d9f3ac11a1c280681671ec1d70655b1a96c4cfdebf6
2321200 0030d073979d6a42b0c421bd9e1959ac127a14b68e94bbf7a5a11f43ce14cabd



2023-04-27T08:59:58.092926Z  INFO finished generating Zebra checkpoints network=Testnet zebra_tip_height=Height(2321586) last_checkpoint="2310400"

https://github.com/ZcashFoundation/zebra/actions/runs/4816455207/jobs/8579149326#step:8:235

I manually generated testnet checkpoints using my local zcashd instance:

$ zebra-checkpoints-release --cli zcash-cli-testnet --last-checkpoint $(tail -1 zebra-consensus/src/checkpoint/test-checkpoints.txt | cut -d" " -f1)
zebra-checkpoints launched
Command-line arguments: Args { backend: Zebrad, transport: Cli, cli: "zcash-cli-testnet", addr: None, last_checkpoint: Some(Height(2310400)), zcli_args: [] }
Fetching block info and calculating checkpoints...


2310800 006b01ff741053041bda472af078224f3e9a07c477feda26db080c6a1f1366a1
2311200 004e43f27a39ac05bcaab5b0686bf10f7dd5af1c193539465f3e0b1eebbfd3da
2311600 0106d2e90b29d32bb4eb327cea6a336fc4884c33397511a568e326bc3657836e
2312000 00641e2f406b38f9d306fd83a8507611a253e7d09e42364e202972605e45fb4b
2312400 00575ea8a9e0688a27fcba9ed89cde7dd88e6cf2546cf908f4e340f9c8fc6a52
2312800 002a95e805c3a1630b8262c67aedb257743ef87b2ff5f2e87f4dea3b2c5829a1
2313200 004fa09d52f3cc1e5b829c4dec72dd0db6ece8a79cd4178f4a7819bf48fd456a
2313600 004ad0407803c2fd87058479d6eec3918acb8c463d1a9630bc11fb1794bdfbc0
2314000 000e92ddc18df503b738bd0c23a46c6862ad30f99452edb3f1fcd93af405325d
2314400 0060385f4943f0e409d0d020ca2686ee1961dfa82f72255b9c296aba45e3546e
2314800 000b3e0cdbba7dbf21f246a092f665136fc8966c5546a7b88e1a67a19c9b7ae9
2315200 003584a3be348692c69f786451a2d856ca4ab2565f64649cccdeb810f107fee3
2315600 00e20ed39fda7da892425ab3f31f6ab9945a29021cbe0a167840b8c7c01c3459
2316000 0013b21a75d41c2982e9d3bbacc4fa613bab66f33180c08cdb0767f5aad95ce8
2316400 00937b033ea9dabbe0467f787319b3e2e7769d9b0142828698783066b8d63b8c
2316800 0042afb0d43eb5cd4d73eeaa69d69b24c97cf8627a4a64741572ca1cb31bde66
2317200 002cb3d799234fd040267d6bb66986aefab76aaec1001112a68b3d350ab84d1c
2317600 00702b6d85769a72329cbac8fa6bb42e3ca04fc3b6f462d8e01215030d4cfeca
2318000 0007ee10ac7bc129ad80aad7e773fe761b141ccec1dabaa506d6c0db82f1e1a6
2318400 003245c3a6132228e5801689513bdd9f7760637d9b00cafe081cf44d48c0bf97
2318800 006d7d609ee491f02f71f5519aa9b3353120cb1c90339f3e675015d1620af76e
2319200 0024258df95e9c78cc40bce9431789e1d294f1ae5980f037a9dcfb65224f2384
2319600 0043eb6699f17b2fa8250ba5ee11931a9056ec0f8119ab85a40aa427cdf973d6
2320000 001e49cf9fbb81c26a1c5287839d40759be3c0ab0563717c2a5a945f85f04238
2320400 006c0fb6b6f9142dfb759360d04a84a0cc8492d15af5d4236f6cf933bed605cb
2320800 0036bacd8d916352350b7d9f3ac11a1c280681671ec1d70655b1a96c4cfdebf6
2321200 0030d073979d6a42b0c421bd9e1959ac127a14b68e94bbf7a5a11f43ce14cabd
2321600 00769577ce51b202cdd291e102756be90672606ab9e84b38c76a020c5ea48cc8
2322000 009a164a4954e1078f755e39c9a7688c439e79fa23dad38ed884195381d14bea

These are the same except for the last 2 checkpoints, because I did the manual generation today, but the CI job was from yesterday.

[dconnolly]

This looks great; future iteration, it automatically opens a PR with the updated checkpoints, as long as that's not on every change? That would be if we have a specific release flow, or something

[gustavovalverde]

LGTM

[teor2345]

This is ready for re-approval, just needed to resolve some minor comment merge conflicts from the previous PR.

[dconnolly]

looks good! <3