containerd-2.1: add configurable concurrent-download-chunk-size setting

[KCSesh]

Description of changes:
Add concurrent-download-chunk-size setting in the containerd-2.1 config.toml, allowing for transfer service default to be updated.

Related:

• feat: add concurrent_download_chunk_size to container-runtime bottlerocket-settings-sdk#99

Testing done:

Set several chunk_sizes and verfied they updated the config properly.

[root@admin]#  apiclient set settings.container-runtime.concurrent-download-chunk-size="32mib"
[root@admin]# apiclient get settings.container-runtime.concurrent-download-chunk-size
{
  "settings": {
    "container-runtime": {
      "concurrent-download-chunk-size": 33554432
    }
  }
}

Also verified via debug, containerd logs during image fetch:

Details

Aug 28 01:11:38 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:38.657403163Z" level=debug msg="fetching layer" chunk_size=33554432 digest="sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5" initial_parallelism=3 mediatype=application/vnd.oci.image.index.v1+json offset=0 parallelism=3 size=8933
    Aug 28 01:11:38 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:38.657479360Z" level=debug msg="do request" digest="sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5" mediatype=application/vnd.oci.image.index.v1+json request.header.accept="application/vnd.oci.image.index.v1+json, */*" request.header.accept-encoding="zstd;q=1.0, gzip;q=0.8, deflate;q=0.5" request.header.range="bytes=0-" request.header.user-agent=containerd/2.1.4+bottlerocket request.method=GET size=8933 url="https://registry-1.docker.io/v2/library/debian/manifests/sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5"
    Aug 28 01:11:38 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:38.853513039Z" level=debug msg="fetch response received" digest="sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5" mediatype=application/vnd.oci.image.index.v1+json response.header.connection=keep-alive response.header.content-length=157 response.header.content-type=application/json response.header.date="Thu, 28 Aug 2025 01:11:38 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=44.234.62.213 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/debian:pull\"" response.status="401 Unauthorized" size=8933 url="https://registry-1.docker.io/v2/library/debian/manifests/sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5"
    Aug 28 01:11:38 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:38.853577633Z" level=debug msg=Unauthorized digest="sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/debian:pull\"" mediatype=application/vnd.oci.image.index.v1+json size=8933
    Aug 28 01:11:38 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:38.853685865Z" level=debug msg="do request" digest="sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5" mediatype=application/vnd.oci.image.index.v1+json request.header.accept="application/vnd.oci.image.index.v1+json, */*" request.header.accept-encoding="zstd;q=1.0, gzip;q=0.8, deflate;q=0.5" request.header.range="bytes=0-" request.header.user-agent=containerd/2.1.4+bottlerocket request.method=GET size=8933 url="https://registry-1.docker.io/v2/library/debian/manifests/sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5"
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.293481878Z" level=debug msg="fetch response received" digest="sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5" mediatype=application/vnd.oci.image.index.v1+json response.header.connection=keep-alive response.header.content-length=8933 response.header.content-type=application/vnd.oci.image.index.v1+json response.header.date="Thu, 28 Aug 2025 01:11:39 GMT" response.header.docker-content-digest="sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=44.234.62.213 response.header.etag="\"sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="97;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=8933 url="https://registry-1.docker.io/v2/library/debian/manifests/sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5"
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.302582477Z" level=debug msg=fetch digest="sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e" mediatype=application/vnd.oci.image.manifest.v1+json size=1021
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.308717245Z" level=debug msg="fetching layer" chunk_size=33554432 digest="sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e" initial_parallelism=3 mediatype=application/vnd.oci.image.manifest.v1+json offset=0 parallelism=3 size=1021
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.308763513Z" level=debug msg="do request" digest="sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e" mediatype=application/vnd.oci.image.manifest.v1+json request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.accept-encoding="zstd;q=1.0, gzip;q=0.8, deflate;q=0.5" request.header.range="bytes=0-" request.header.user-agent=containerd/2.1.4+bottlerocket request.method=GET size=1021 url="https://registry-1.docker.io/v2/library/debian/manifests/sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e"
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.417055008Z" level=debug msg="fetch response received" digest="sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e" mediatype=application/vnd.oci.image.manifest.v1+json response.header.connection=keep-alive response.header.content-length=1021 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Thu, 28 Aug 2025 01:11:39 GMT" response.header.docker-content-digest="sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=44.234.62.213 response.header.etag="\"sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="96;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=1021 url="https://registry-1.docker.io/v2/library/debian/manifests/sha256:a9114963cf3cd27fdae799d3c987758a26709515db3673ce85e2c4befd88199e"
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.425644190Z" level=debug msg=fetch digest="sha256:047bd8d819400f5dab52d40aa2c424109b6627b9e4ec27c113a308e72aac0e7b" mediatype=application/vnd.oci.image.config.v1+json size=451
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.444000024Z" level=debug msg=fetch digest="sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=49278280
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.450092500Z" level=debug msg="fetching layer" chunk_size=33554432 digest="sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb" initial_parallelism=3 mediatype=application/vnd.oci.image.layer.v1.tar+gzip offset=0 parallelism=3 size=49278280
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.450142467Z" level=debug msg="do request" digest="sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.accept-encoding="zstd;q=1.0, gzip;q=0.8, deflate;q=0.5" request.header.range="bytes=0-" request.header.user-agent=containerd/2.1.4+bottlerocket request.method=GET size=49278280 url="https://registry-1.docker.io/v2/library/debian/blobs/sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb"
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.745760228Z" level=debug msg="fetch response received" digest="sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cf-ray=975fef982c9cff09-PDX response.header.connection=keep-alive response.header.content-length=49278280 response.header.content-range="bytes 0-49278279/49278280" response.header.content-type=application/octet-stream response.header.date="Thu, 28 Aug 2025 01:11:39 GMT" response.header.etag="\"5f656b1c0a883408edb64c4f18262db8\"" response.header.last-modified="Tue, 12 Aug 2025 20:45:14 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.status="206 Partial Content" size=49278280 url="https://registry-1.docker.io/v2/library/debian/blobs/sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb"
    Aug 28 01:11:39 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:39.747370761Z" level=debug msg="do request" digest="sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.accept-encoding="zstd;q=1.0, gzip;q=0.8, deflate;q=0.5" request.header.range="bytes=33554432-" request.header.user-agent=containerd/2.1.4+bottlerocket request.method=GET size=49278280 url="https://registry-1.docker.io/v2/library/debian/blobs/sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb"
    Aug 28 01:11:40 ip-192-168-47-119.us-west-2.compute.internal containerd[16099]: time="2025-08-28T01:11:40.005249637Z" level=debug msg="fetch response received" digest="sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cf-ray=975fef99fce22d7c-PDX response.header.connection=keep-alive response.header.content-length=15723848 response.header.content-range="bytes 33554432-49278279/49278280" response.header.content-type=application/octet-stream response.header.date="Thu, 28 Aug 2025 01:11:40 GMT" response.header.etag="\"5f656b1c0a883408edb64c4f18262db8\"" response.header.last-modified="Tue, 12 Aug 2025 20:45:14 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.status="206 Partial Content" size=49278280 url="https://registry-1.docker.io/v2/library/debian/blobs/sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb"

• 64mb using the alias concurrent-layer-fetch-buffer

[root@admin]# apiclient set settings.container-runtime.concurrent-layer-fetch-buffer="64mb"
[root@admin]# apiclient get settings.container-runtime
{
  "settings": {
    "container-runtime": {
      "concurrent-download-chunk-size": 64000000
    }
  }
}

config.toml:

[plugins."io.containerd.cri.v1.images"]
concurrent_layer_fetch_buffer = 64000000
use_local_image_pull = false

[plugins."io.containerd.transfer.v1.local"]
concurrent_layer_fetch_buffer = 64000000

[[plugins."io.containerd.transfer.v1.local".unpack_config]]
snapshotter = "overlayfs"
differ = "walking"
platform = "linux/amd64"

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[vrdn-23]

Thanks for the PR @KCSesh !

Just a quick question: From the original PR in containerd, it seems there are 2 other settings that can be configured to help improve image pull performance.

c_para (max number of chunks being pulled per layer at once)
chunk_size_mb ( size of chunks in mb )
ctd_max_con ( max # of layer pulled at once )

Are these already configurable via BottleRocket or can we add support for these as well in this PR?

[KCSesh]

Hey @vrdn-23, thanks for taking a look and the interest! 😄

You are correct that at one point, the PR introduced 2 additional knobs (max_concurrent_fetch_per_download and concurrent_fetch_chunk_sizes_mb, we can see all 3 values in this comment:

[plugins."io.containerd.transfer.v1.local"] # for ctr pull cfg
max_concurrent_downloads = 4
max_concurrent_fetch_per_download = 110
concurrent_fetch_chunk_sizes_mb = 32

However, later on in the PR through discussion, the 2 additional values was reduced to 1 value here with the specific commit here.

The merged result only added concurrent_layer_fetch_buffer.

The 3rd variable: max_concurrent_downloads is already supported in the bottlerocket settings for all containerd versions!

[vrdn-23]

That makes sense! Thanks for the explanation @KCSesh. Looking forward to testing this out!

[KCSesh]

^ Making use of the default instead of if/else