progress-fd issue tracker

[cgwalters]

Followup to #921 which we merged, and I was hoping to do some post-merge followup on. This issue will track some of that followup.

• docs Various updates for --progress-fd #961
• integration tests tests: Add some sanity checking of --json-fd #1017
• size concerns

size concerns

In the PR we kept going back and forth on whether the progress data should require the client to keep track of state or not. @antheas was calling this "normalization". There's clear simplicity in having the protocol be stateless.

However...looking at the layer progress (one of the most important phases) in a local run here during a download, we keep accumulating and emitting more data (completed subtasks) for each layer. In this run the final ProgressBytes I got was 12 KiB which...is a bit more than I was expecting.

Click for (prettified) example:

{ "type": "ProgressBytes", "apiVersion": "org.containers.bootc.progress/v1", "task": "pulling", "description": "Pulling Image: sha256:127c32e1418aebdd9a6cbeb6517605e11c1aa4004689e4a396307babe801f309", "id": "sha256:127c32e1418aebdd9a6cbeb6517605e11c1aa4004689e4a396307babe801f309", "bytesCached": 0, "bytes": 1212857687, "bytesTotal": 1212857687, "stepsCached": 0, "steps": 74, "stepsTotal": 74, "subtasks": [ { "subtask": "ostree chunk", "description": "ostree chunk: bca923230b7dde927d4f2", "id": "bca923230b7dde927d4f2", "bytesCached": 0, "bytes": 434743772, "bytesTotal": 434743772 }, { "subtask": "ostree chunk", "description": "ostree chunk: 7787da09b798ff4c245e5", "id": "7787da09b798ff4c245e5", "bytesCached": 0, "bytes": 23673835, "bytesTotal": 23673835 }, { "subtask": "ostree chunk", "description": "ostree chunk: 46b215c7346346e749145", "id": "46b215c7346346e749145", "bytesCached": 0, "bytes": 35716920, "bytesTotal": 35716920 }, { "subtask": "ostree chunk", "description": "ostree chunk: 5ce7d0f46f7ce0107eaab", "id": "5ce7d0f46f7ce0107eaab", "bytesCached": 0, "bytes": 14600588, "bytesTotal": 14600588 }, { "subtask": "ostree chunk", "description": "ostree chunk: 1ef5317b04cbde5294f5f", "id": "1ef5317b04cbde5294f5f", "bytesCached": 0, "bytes": 14419888, "bytesTotal": 14419888 }, { "subtask": "ostree chunk", "description": "ostree chunk: bfb6ab72f6e960dc7c91f", "id": "bfb6ab72f6e960dc7c91f", "bytesCached": 0, "bytes": 29870153, "bytesTotal": 29870153 }, { "subtask": "ostree chunk", "description": "ostree chunk: 41b301e3bb75ca998bf86", "id": "41b301e3bb75ca998bf86", "bytesCached": 0, "bytes": 12703409, "bytesTotal": 12703409 }, { "subtask": "ostree chunk", "description": "ostree chunk: 43c7299a13755a59c3570", "id": "43c7299a13755a59c3570", "bytesCached": 0, "bytes": 12076932, "bytesTotal": 12076932 }, { "subtask": "ostree chunk", "description": "ostree chunk: 82fe717d3cd099e4576e7", "id": "82fe717d3cd099e4576e7", "bytesCached": 0, "bytes": 16657939, "bytesTotal": 16657939 }, { "subtask": "ostree chunk", "description": "ostree chunk: 46c7b3da4902f903056a0", "id": "46c7b3da4902f903056a0", "bytesCached": 0, "bytes": 8510610, "bytesTotal": 8510610 }, { "subtask": "ostree chunk", "description": "ostree chunk: 4e60e3792abde5311f750", "id": "4e60e3792abde5311f750", "bytesCached": 0, "bytes": 8003859, "bytesTotal": 8003859 }, { "subtask": "ostree chunk", "description": "ostree chunk: 0b796653effe66b6b6f4a", "id": "0b796653effe66b6b6f4a", "bytesCached": 0, "bytes": 6904178, "bytesTotal": 6904178 }, { "subtask": "ostree chunk", "description": "ostree chunk: decebbcc09fcb2d10260b", "id": "decebbcc09fcb2d10260b", "bytesCached": 0, "bytes": 6778508, "bytesTotal": 6778508 }, { "subtask": "ostree chunk", "description": "ostree chunk: 743d8170de20edd185b8d", "id": "743d8170de20edd185b8d", "bytesCached": 0, "bytes": 6637766, "bytesTotal": 6637766 }, { "subtask": "ostree chunk", "description": "ostree chunk: 08daf3e7216c98939f930", "id": "08daf3e7216c98939f930", "bytesCached": 0, "bytes": 5206174, "bytesTotal": 5206174 }, { "subtask": "ostree chunk", "description": "ostree chunk: 760cadc63f561de092591", "id": "760cadc63f561de092591", "bytesCached": 0, "bytes": 5662725, "bytesTotal": 5662725 }, { "subtask": "ostree chunk", "description": "ostree chunk: 820a6b1995bf16e7c42e4", "id": "820a6b1995bf16e7c42e4", "bytesCached": 0, "bytes": 11252209, "bytesTotal": 11252209 }, { "subtask": "ostree chunk", "description": "ostree chunk: 15f10449a2c985e5d097e", "id": "15f10449a2c985e5d097e", "bytesCached": 0, "bytes": 4403265, "bytesTotal": 4403265 }, { "subtask": "ostree chunk", "description": "ostree chunk: 0473f5256cda332e113a3", "id": "0473f5256cda332e113a3", "bytesCached": 0, "bytes": 2792039, "bytesTotal": 2792039 }, { "subtask": "ostree chunk", "description": "ostree chunk: 854f47b2e8b312a25526c", "id": "854f47b2e8b312a25526c", "bytesCached": 0, "bytes": 4506580, "bytesTotal": 4506580 }, { "subtask": "ostree chunk", "description": "ostree chunk: 0f0753883ec53774d8cbf", "id": "0f0753883ec53774d8cbf", "bytesCached": 0, "bytes": 4409259, "bytesTotal": 4409259 }, { "subtask": "ostree chunk", "description": "ostree chunk: b66bae9ab05760c2cae7c", "id": "b66bae9ab05760c2cae7c", "bytesCached": 0, "bytes": 3817407, "bytesTotal": 3817407 }, { "subtask": "ostree chunk", "description": "ostree chunk: 9168a98efc0f20fe7222c", "id": "9168a98efc0f20fe7222c", "bytesCached": 0, "bytes": 5073270, "bytesTotal": 5073270 }, { "subtask": "ostree chunk", "description": "ostree chunk: 635fa553888bf7fb02804", "id": "635fa553888bf7fb02804", "bytesCached": 0, "bytes": 4202725, "bytesTotal": 4202725 }, { "subtask": "ostree chunk", "description": "ostree chunk: 17bb2ef0dfb2941a561e6", "id": "17bb2ef0dfb2941a561e6", "bytesCached": 0, "bytes": 3628297, "bytesTotal": 3628297 }, { "subtask": "ostree chunk", "description": "ostree chunk: dc34b802800e0af2bf1b4", "id": "dc34b802800e0af2bf1b4", "bytesCached": 0, "bytes": 3487384, "bytesTotal": 3487384 }, { "subtask": "ostree chunk", "description": "ostree chunk: 263e158a4a0ca3e3e0312", "id": "263e158a4a0ca3e3e0312", "bytesCached": 0, "bytes": 3492706, "bytesTotal": 3492706 }, { "subtask": "ostree chunk", "description": "ostree chunk: a82c571fabd4910a1d1bb", "id": "a82c571fabd4910a1d1bb", "bytesCached": 0, "bytes": 906047, "bytesTotal": 906047 }, { "subtask": "ostree chunk", "description": "ostree chunk: 492c5bcbae66c24743149", "id": "492c5bcbae66c24743149", "bytesCached": 0, "bytes": 3402775, "bytesTotal": 3402775 }, { "subtask": "ostree chunk", "description": "ostree chunk: a0230ba465f3959c97432", "id": "a0230ba465f3959c97432", "bytesCached": 0, "bytes": 3148819, "bytesTotal": 3148819 }, { "subtask": "ostree chunk", "description": "ostree chunk: dbd1e7e16e61ac1780670", "id": "dbd1e7e16e61ac1780670", "bytesCached": 0, "bytes": 3176990, "bytesTotal": 3176990 }, { "subtask": "ostree chunk", "description": "ostree chunk: 8cd5a12c083744402472d", "id": "8cd5a12c083744402472d", "bytesCached": 0, "bytes": 2998918, "bytesTotal": 2998918 }, { "subtask": "ostree chunk", "description": "ostree chunk: dfc1365a2d046d0ecf8f9", "id": "dfc1365a2d046d0ecf8f9", "bytesCached": 0, "bytes": 1894124, "bytesTotal": 1894124 }, { "subtask": "ostree chunk", "description": "ostree chunk: b8732c5b9580f3710b2f4", "id": "b8732c5b9580f3710b2f4", "bytesCached": 0, "bytes": 2799520, "bytesTotal": 2799520 }, { "subtask": "ostree chunk", "description": "ostree chunk: 64eea8d70bf3b3b31cdb5", "id": "64eea8d70bf3b3b31cdb5", "bytesCached": 0, "bytes": 2553266, "bytesTotal": 2553266 }, { "subtask": "ostree chunk", "description": "ostree chunk: 5ccf9e04fb9dcaaa00514", "id": "5ccf9e04fb9dcaaa00514", "bytesCached": 0, "bytes": 2080498, "bytesTotal": 2080498 }, { "subtask": "ostree chunk", "description": "ostree chunk: 27f93ce479a2a4f2ed9cf", "id": "27f93ce479a2a4f2ed9cf", "bytesCached": 0, "bytes": 3732082, "bytesTotal": 3732082 }, { "subtask": "ostree chunk", "description": "ostree chunk: b0ed1c30c1810ec35555c", "id": "b0ed1c30c1810ec35555c", "bytesCached": 0, "bytes": 2654549, "bytesTotal": 2654549 }, { "subtask": "ostree chunk", "description": "ostree chunk: bf39286872071851063ce", "id": "bf39286872071851063ce", "bytesCached": 0, "bytes": 410836, "bytesTotal": 410836 }, { "subtask": "ostree chunk", "description": "ostree chunk: 20220a8b5841d3190b23d", "id": "20220a8b5841d3190b23d", "bytesCached": 0, "bytes": 3383311, "bytesTotal": 3383311 }, { "subtask": "ostree chunk", "description": "ostree chunk: 0c322000c68c88a8a6e19", "id": "0c322000c68c88a8a6e19", "bytesCached": 0, "bytes": 8447384, "bytesTotal": 8447384 }, { "subtask": "ostree chunk", "description": "ostree chunk: 9b374af64e2b534ba7997", "id": "9b374af64e2b534ba7997", "bytesCached": 0, "bytes": 1242438, "bytesTotal": 1242438 }, { "subtask": "ostree chunk", "description": "ostree chunk: 17880fd3670dc1d3ad3d7", "id": "17880fd3670dc1d3ad3d7", "bytesCached": 0, "bytes": 1205548, "bytesTotal": 1205548 }, { "subtask": "ostree chunk", "description": "ostree chunk: d514efcf47822aff479d9", "id": "d514efcf47822aff479d9", "bytesCached": 0, "bytes": 5192130, "bytesTotal": 5192130 }, { "subtask": "ostree chunk", "description": "ostree chunk: 678dee538c6f43289e3fe", "id": "678dee538c6f43289e3fe", "bytesCached": 0, "bytes": 8546705, "bytesTotal": 8546705 }, { "subtask": "ostree chunk", "description": "ostree chunk: 36a6412eee2f43f80129e", "id": "36a6412eee2f43f80129e", "bytesCached": 0, "bytes": 5906596, "bytesTotal": 5906596 }, { "subtask": "ostree chunk", "description": "ostree chunk: 569b5c232ced65ff58fec", "id": "569b5c232ced65ff58fec", "bytesCached": 0, "bytes": 16706185, "bytesTotal": 16706185 }, { "subtask": "ostree chunk", "description": "ostree chunk: f04aba303f2d539dfe8ff", "id": "f04aba303f2d539dfe8ff", "bytesCached": 0, "bytes": 1426478, "bytesTotal": 1426478 }, { "subtask": "ostree chunk", "description": "ostree chunk: c94d1855b699eaa99335c", "id": "c94d1855b699eaa99335c", "bytesCached": 0, "bytes": 1853709, "bytesTotal": 1853709 }, { "subtask": "ostree chunk", "description": "ostree chunk: f72316d3eff3bea7dfe73", "id": "f72316d3eff3bea7dfe73", "bytesCached": 0, "bytes": 1363165, "bytesTotal": 1363165 }, { "subtask": "ostree chunk", "description": "ostree chunk: aeabcb9a5095245f0a88b", "id": "aeabcb9a5095245f0a88b", "bytesCached": 0, "bytes": 990031, "bytesTotal": 990031 }, { "subtask": "ostree chunk", "description": "ostree chunk: c85e204532a1b23fb9be4", "id": "c85e204532a1b23fb9be4", "bytesCached": 0, "bytes": 753533, "bytesTotal": 753533 }, { "subtask": "ostree chunk", "description": "ostree chunk: f47ae488c3046092ace03", "id": "f47ae488c3046092ace03", "bytesCached": 0, "bytes": 1354135, "bytesTotal": 1354135 }, { "subtask": "ostree chunk", "description": "ostree chunk: 7c6dd2bb4648b78a02b9d", "id": "7c6dd2bb4648b78a02b9d", "bytesCached": 0, "bytes": 1006770, "bytesTotal": 1006770 }, { "subtask": "ostree chunk", "description": "ostree chunk: 5a5652b1d7ce060341943", "id": "5a5652b1d7ce060341943", "bytesCached": 0, "bytes": 1431015, "bytesTotal": 1431015 }, { "subtask": "ostree chunk", "description": "ostree chunk: 3371a52ab2ae279c9c097", "id": "3371a52ab2ae279c9c097", "bytesCached": 0, "bytes": 1214839, "bytesTotal": 1214839 }, { "subtask": "ostree chunk", "description": "ostree chunk: 49e518db5f56651bb7f5c", "id": "49e518db5f56651bb7f5c", "bytesCached": 0, "bytes": 1145057, "bytesTotal": 1145057 }, { "subtask": "ostree chunk", "description": "ostree chunk: 007142b4c9013f6ced073", "id": "007142b4c9013f6ced073", "bytesCached": 0, "bytes": 8810138, "bytesTotal": 8810138 }, { "subtask": "ostree chunk", "description": "ostree chunk: 60511678ad429b0f823db", "id": "60511678ad429b0f823db", "bytesCached": 0, "bytes": 8981116, "bytesTotal": 8981116 }, { "subtask": "ostree chunk", "description": "ostree chunk: 61291d26047700443893e", "id": "61291d26047700443893e", "bytesCached": 0, "bytes": 12053670, "bytesTotal": 12053670 }, { "subtask": "ostree chunk", "description": "ostree chunk: 951bf96f06ffdb2f01dc4", "id": "951bf96f06ffdb2f01dc4", "bytesCached": 0, "bytes": 9501139, "bytesTotal": 9501139 }, { "subtask": "ostree chunk", "description": "ostree chunk: 6ef4a9ab822ece2f9001e", "id": "6ef4a9ab822ece2f9001e", "bytesCached": 0, "bytes": 13734035, "bytesTotal": 13734035 }, { "subtask": "ostree chunk", "description": "ostree chunk: a61f018731661d738ea3a", "id": "a61f018731661d738ea3a", "bytesCached": 0, "bytes": 113384098, "bytesTotal": 113384098 }, { "subtask": "ostree chunk", "description": "ostree chunk: 8997f32f8b56ede6bfdee", "id": "8997f32f8b56ede6bfdee", "bytesCached": 0, "bytes": 3190, "bytesTotal": 3190 }, { "subtask": "ostree chunk", "description": "ostree chunk: 0be1d11f3c17d9670acc9", "id": "0be1d11f3c17d9670acc9", "bytesCached": 0, "bytes": 1948304, "bytesTotal": 1948304 }, { "subtask": "layer", "description": "layer: 90cc6a627d6af3292093d", "id": "90cc6a627d6af3292093d", "bytesCached": 0, "bytes": 10263, "bytesTotal": 10263 }, { "subtask": "layer", "description": "layer: b4c762a4bb09d379184a5", "id": "b4c762a4bb09d379184a5", "bytesCached": 0, "bytes": 10263, "bytesTotal": 10263 }, { "subtask": "layer", "description": "layer: cd0caee131ccee6fce0aa", "id": "cd0caee131ccee6fce0aa", "bytesCached": 0, "bytes": 89152818, "bytesTotal": 89152818 }, { "subtask": "layer", "description": "layer: e853d0b60a65fdcdb01f5", "id": "e853d0b60a65fdcdb01f5", "bytesCached": 0, "bytes": 10263, "bytesTotal": 10263 }, { "subtask": "layer", "description": "layer: 2b38f58e8f4ab5cb12812", "id": "2b38f58e8f4ab5cb12812", "bytesCached": 0, "bytes": 10263, "bytesTotal": 10263 }, { "subtask": "layer", "description": "layer: 08faeb75bbf7617c9a6fd", "id": "08faeb75bbf7617c9a6fd", "bytesCached": 0, "bytes": 10263, "bytesTotal": 10263 }, { "subtask": "layer", "description": "layer: dc91651243b6a48f8a96d", "id": "dc91651243b6a48f8a96d", "bytesCached": 0, "bytes": 169057488, "bytesTotal": 169057488 }, { "subtask": "layer", "description": "layer: 90cc6a627d6af3292093d", "id": "90cc6a627d6af3292093d", "bytesCached": 0, "bytes": 10263, "bytesTotal": 10263 }, { "subtask": "layer", "description": "layer: d2ebaa84aa5f16a1351d1", "id": "d2ebaa84aa5f16a1351d1", "bytesCached": 0, "bytes": 10263, "bytesTotal": 10263 } ] }

Does that size actually matter? Well...obviously we're talking about local IPC so 12k isn't large exactly, but we are also trying to emit this data at a relatively high frequency. Note that a default Linux pipe buffer is 64k, so if the reader slows down a bit that's only a maximum of 5 messages queued before we fill the pipe.

The size of this progress data will grow with the number of layers...let's say in the future we start using 100 layers (not at all an unreasonable thing!) that'd double to 25k of JSON etc.

This is data we're serializing and having the caller to deserialize ~5 times a second, competing with actual real work (like decompression, sha256 etc.).

OK well, benchmarking this obviously CPUs are super fast and python parses it in 44.4 usec here...but...still...it wouldn't be a huge burden at all to have a stateful protocol either.

[antheas]

Hmmm fundamentally if we make sure it does not block and is stateless, isn't it enough if the pipe fits a single message? ie size is less than 60k? Then the client can read it at its own pace, up to 5hz?

Then, even if its more than 65k, to make sure it blocks correctly you could spin up a thread that writes the current progress when the pipe has space if it has changed and e.g., at a max speed of 5Hz?

This way the client could read the state every 2 seconds without affecting the update.

The progress bar I did works really well. I will add three issues with the protocol for tracking though.

Blind Updates

Right now, you're supposed to do bootc update --check then bootc status --json to get if there is an update and the version. However, there can be days between the update check and the user pressing the update button, so the new version can change. This is a race condition: we do not know what version we are updating to.

Blind Rebases

Same as above, bootc switch is also blind and there is no bootc update --check equivalent. In order for bootc switch to apply it has to be committed.

Would be nice to replace the hash with the version or somehow include it in there.

Can block updates

As mentioned above, currently if the client does not read the output at 5hz, it can stall the update. Would be nice to fix that with e.g., a thread. I had to implement a thread to read the output so that this does not happen since I end up updating the UI at 1hz (updating the UI is very expensive).

It would be ideal if callers can read the state on demand on their own pace.

[antheas]

This is data we're serializing and having the caller to deserialize ~5 times a second, competing with actual real work (like decompression, sha256 etc.).

Compared to a thread wakeup or even calculating a single hash i think the overhead is minimal. If the update takes e.g., 10 minutes, even if the reader does all the updates thats 10 * 60 * 5 * 12k=36MB, where you currently dual hash 3GB-10GB of data to deploy an image (as both the layer and the files if derived need to be hashed).

Focusing on downloading multiple layers at a time is more important IMO.

OK well, benchmarking this obviously CPUs are super fast and python parses it in 44.4 usec here...but...still...it wouldn't be a huge burden at all to have a stateful protocol either.

And at the point where you download multiple layers, it would become a bigger problem both for you as a sender as you have to deserialize that if its not normalized.

I really like my 30 line code that works perfectly because its normalized.

Let me correct what I said above, I update at 3Hz, which is plenty.

[cgwalters]

Blind Updates

I think we just need bootc upgrade --from-cached or so.

[cgwalters]

And at the point where you download multiple layers, it would become a bigger problem both for you as a sender as you have to deserialize that if its not normalized.

I don't understand what you mean here.

I really like my 30 line code that works perfectly because its normalized.

Yeah, I would say it's trading off code simplicity for consumers versus CPU usage for every client. Dunno. Since you kicked this off and have been driving it I am OK with the status quo...worst case we have to introduce something like --json-fd1 or so (hmm should we actually add a protocol version now?).

[antheas]

And at the point where you download multiple layers, it would become a bigger problem both for you as a sender as you have to deserialize that if its not normalized.

I don't understand what you mean here.

Is there a stateful protocol that would work while there are 4-5 layers downloading that would work for both a client that wants to update 5x per second and a client that wants to update every 2 seconds? We did attempt a stateful variant in the PR.

I really like my 30 line code that works perfectly because its normalized.

Yeah, I would say it's trading off code simplicity for consumers versus CPU usage for every client. Dunno. Since you kicked this off and have been driving it I am OK with the status quo...worst case we have to introduce something like --json-fd1 or so (hmm should we actually add a protocol version now?).

Pretty much. I would contest how much that is though. Perhaps if we are talking about bootc being used in microprocessors. Although at that point, 100 layers?

A modern UI with something like react when binding to the final JSON and rerendering would use 100-1000x more processing power than reading the json string. Both python and javascript would also use more processing power reconstructing the state if the protocol is stateful, as the json parser is compiled and would only require a single wakeup to read the whole thing.

You still have --json unused so when you fix the logging issue, you can deprecate --json-fd and switch to something stateful over fd 0 if you feel its needed.

[cgwalters]

It looks to me like the current steps is (or really should be, barring bugs) always equal to the length of completed subtasks, right? It's also confusing that we have both "steps" and "subtasks". Maybe clearest to rename "subtasks" to "steps"?

[cgwalters]

Is there a stateful protocol that would work while there are 4-5 layers downloading that would work for both a client that wants to update 5x per second and a client that wants to update every 2 seconds?

The relationship or concern you have between state and update frequency is not obvious to me. Remember in the stateful version I did originally, we still always emitted an event for each byte-progress that was started or ended, so a client can't miss it if that's what you're thinking about.

It's basically just: do we emit the prior history of steps each time or not. If we don't, then a client which wants to render that state needs to remember it. Note a terminal renderer wouldn't need to care because the terminal itself stores rendering history! But a GUI that wanted to show progress-bar per layer would need to hold that state. Which honestly I just don't think is very onerous...

[antheas]

It looks to me like the current steps is (or really should be, barring bugs) always equal to the length of completed subtasks, right? It's also confusing that we have both "steps" and "subtasks". Maybe clearest to rename "subtasks" to "steps"?

Yeah... hm thats unfortunate as you have both a subtasks and a steps variable so all of them would need to be renamed.

Subtask is used within the tasks themselves so renaming subtask to step is not that clearcut. Would probably need to rename steps to subtasks instead.

If this happens, then you need to find a name for holding the currently executing subtasks.

Is there a stateful protocol that would work while there are 4-5 layers downloading that would work for both a client that wants to update 5x per second and a client that wants to update every 2 seconds?

The relationship or concern you have between state and update frequency is not obvious to me. Remember in the stateful version I did originally, we still always emitted an event for each byte-progress that was started or ended, so a client can't miss it if that's what you're thinking about.

It's basically just: do we emit the prior history of steps each time or not. If we don't, then a client which wants to render that state needs to remember it. Note a terminal renderer wouldn't need to care because the terminal itself stores rendering history! But a GUI that wanted to show progress-bar per layer would need to hold that state. Which honestly I just don't think is very onerous...

In your example you express concern for downloading 100 layers. If the process was stateful, you would have 205 blocking events over the course of the download (2 per layer + extras). You would also need to make sure that each of the 5 layers that is downloading gets 5 events per second ie you'd need to do 25 updates per second and make sure you are fair to all layers if there is a block

With the stateless approach you have 0-3 depending on how its implemented. If the client uses my code you actually never have to block, as the last event can be used to show full progress state. This is because each stage has a fixed starting offset.

My 2 cents is it would be too complicated for implementing both on the bootc side and client side

[cgwalters]

OK #961 does two important things:

• Marks this as an experimental feature for now
• Also renames it to --progress-fd

I'd like to get a new release out with just the few fixes that have landed, and then we can aim to focus on progress-fd and stabilize it in the next release.

[imbev]

Blind Updates

I think we just need bootc upgrade --from-cached or so.

What should the behavior be if the cached digest is no longer available? Fail or fallback to non-cache upgrade?

[imbev]

It's basically just: do we emit the prior history of steps each time or not. If we don't, then a client which wants to render that state needs to remember it. Note a terminal renderer wouldn't need to care because the terminal itself stores rendering history! But a GUI that wanted to show progress-bar per layer would need to hold that state. Which honestly I just don't think is very onerous...

Taking inspiration from the Matrix protocol, what if we emit the prior history with an option of either the starting id(starting to latest), latest (single), or all? Each step could provide current steps and total steps.

[cgwalters]

Taking inspiration from the Matrix protocol, what if we emit the prior history with an option of either the starting id(starting to latest), latest (single), or all? Each step could provide current steps and total steps.

The protocol is unidirectional - we operate over a pipe. I don't see how we add any options today other than either something like --progress-fd=<number>,<options> like --progress-fd=3,mode=stateful or so or we go to a full bidirectional system, but that runs straight into all the bigger picture stuff in #522

[cgwalters]

I don't see how we add any options today other than either something like --progress-fd=, like --progress-fd=3,mode=stateful

(Or probably better --progress-fd=3 --progress-options=stateful which we could clearly do later)