A test tag created just to check if go vendoring works with ADO.

Support fetch first then unpack

Currently, containerd starts unpacking layers in parallel while other layers are being
downloaded. However, there are cases where we want to force the layer download even if the
extracted layer snapshot or the content blob already exists. If containerd starts
unpacking before fetch it looks at the snapshots and doesn't attempt to fetch the layers.
To avoid this, we now always fetch all layers first before starting unpack.

Signed-off-by: Amit Barve <ambarve@microsoft.com>

CRI: improve image pulling performance

Background:

With current design, the content backend uses key-lock for long-lived
write transaction. If the content reference has been marked for write
transaction, the other requestes on the same reference will fail fast with
unavailable error. Since the metadata plugin is based on boltbd which
only supports single-writer, the content backend can't block or handle
the request too long. It requires the client to handle retry by itself,
like OpenWriter - backoff retry helper. But the maximum retry interval
can be up to 2 seconds. If there are several concurrent requestes fo the
same image, the waiters maybe wakeup at the same time and there is only
one waiter can continue. A lot of waiters will get into sleep and we will
take long time to finish all the pulling jobs and be worse if the image
has many more layers, which mentioned in issue containerd#4937.

After fetching, containerd.Pull API allows several hanlers to commit
same ChainID snapshotter but only one can be done successfully. Since
unpack tar.gz is time-consuming job, it can impact the performance on
unpacking for same ChainID snapshotter in parallel.

For instance, the Request 2 doesn't need to prepare and commit, it
should just wait for Request 1 finish, which mentioned in pull
request containerd#6318.

```text
	Request 1	Request 2

	Prepare
	   |
	   |
	   |
	   |		Prepare
	Commit		   |
			   |
			   |
			   |
			Commit(failed on exist)
```

Both content backoff retry and unnecessary unpack impacts the performance.

Solution:

Introduced the duplicate suppression in fetch and unpack context. The
deplicate suppression uses key-mutex and single-waiter-notify to support
singleflight. The caller can use the duplicate suppression in different
PullImage handlers so that we can avoid unnecessary unpack and spin-lock
in OpenWriter.

Test Result:

Before enhancement:

```bash
➜  /tmp sudo bash testing.sh "localhost:5000/redis:latest" 20
crictl pull localhost:5000/redis:latest (x20) takes ...

real	1m6.172s
user	0m0.268s
sys	0m0.193s

docker pull localhost:5000/redis:latest (x20) takes ...

real	0m1.324s
user	0m0.441s
sys	0m0.316s

➜  /tmp sudo bash testing.sh "localhost:5000/golang:latest" 20
crictl pull localhost:5000/golang:latest (x20) takes ...

real	1m47.657s
user	0m0.284s
sys	0m0.224s

docker pull localhost:5000/golang:latest (x20) takes ...

real	0m6.381s
user	0m0.488s
sys	0m0.358s
```

With this enhancement:

```bash
➜  /tmp sudo bash testing.sh "localhost:5000/redis:latest" 20
crictl pull localhost:5000/redis:latest (x20) takes ...

real	0m1.140s
user	0m0.243s
sys	0m0.178s

docker pull localhost:5000/redis:latest (x20) takes ...

real	0m1.239s
user	0m0.463s
sys	0m0.275s

➜  /tmp sudo bash testing.sh "localhost:5000/golang:latest" 20
crictl pull localhost:5000/golang:latest (x20) takes ...

real	0m5.546s
user	0m0.217s
sys	0m0.219s

docker pull localhost:5000/golang:latest (x20) takes ...

real	0m6.090s
user	0m0.501s
sys	0m0.331s
```

Test Script:

localhost:5000/{redis|golang}:latest is equal to
docker.io/library/{redis|golang}:latest. The image is hold in local registry
service by `docker run -d -p 5000:5000 --name registry registry:2`.

```bash

image_name="${1}"
pull_times="${2:-10}"

cleanup() {
  ctr image rmi "${image_name}"
  ctr -n k8s.io image rmi "${image_name}"
  crictl rmi "${image_name}"
  docker rmi "${image_name}"
  sleep 2
}

crictl_testing() {
  for idx in $(seq 1 ${pull_times}); do
    crictl pull "${image_name}" > /dev/null 2>&1 &
  done
  wait
}

docker_testing() {
  for idx in $(seq 1 ${pull_times}); do
    docker pull "${image_name}" > /dev/null 2>&1 &
  done
  wait
}

cleanup > /dev/null 2>&1

echo 3 > /proc/sys/vm/drop_caches
sleep 3
echo "crictl pull $image_name (x${pull_times}) takes ..."
time crictl_testing
echo

echo 3 > /proc/sys/vm/drop_caches
sleep 3
echo "docker pull $image_name (x${pull_times}) takes ..."
time docker_testing
```

Fixes: containerd#4937
Close: containerd#4985
Close: containerd#6318

Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 8113758)
Signed-off-by: Amit Barve <ambarve@microsoft.com>

containerd 1.6.4

Welcome to the v1.6.4 release of containerd!

The fourth patch release for containerd 1.6 includes two fixes for CNI and SELinux.

### Notable Updates

* **Update go-cni to fix teardown regression** ([containerd#6877](containerd#6877))
* **Fix broken SELinux relabeling for Kubernetes volume mounts** ([containerd#6878](containerd#6878))

See the changelog for complete list of changes

Please try out the release binaries and report any issues at
https://github.com/containerd/containerd/issues.

### Contributors

* Mike Brown
* Kazuyoshi Kato
* Michael Zappa
* Wang Bing
* Wei Fu
* Antonio Ojea
* Derek McGowan
* Henry Wang
* Nabeel Rana
* Phil Estes

### Changes
<details><summary>8 commits</summary>
<p>

* Prepare release notes for 1.6.4 ([containerd#6887](containerd#6887))
  * [`0d1d2953b`](containerd@0d1d295) Prepare release notes for 1.6.4
* [release/1.6 backport] Bump opencontainers/selinux from 1.10.0 to 1.10.1 ([containerd#6878](containerd#6878))
  * [`42d691fe6`](containerd@42d691f) Bump opencontainers/selinux from 1.10.0 to 1.10.1
* [release/1.6] Update go-cni to v1.1.5 backported due to conflict in go.mod ([containerd#6877](containerd#6877))
  * [`e9f22e008`](containerd@e9f22e0) Update go-cni to v1.1.5
* [release/1.6] cri: close fifos when container is deleted ([containerd#6859](containerd#6859))
  * [`be4909e95`](containerd@be4909e) cri: close fifos when container is deleted
</p>
</details>

### Changes from containerd/go-cni
<details><summary>8 commits</summary>
<p>

* bump github.com/containernetworking/cni v1.1.0 ([containerd#98](containerd/go-cni#98))
  * [`e24193e`](containerd/go-cni@e24193e) bump github.com/containernetworking/cni v1.1.0
* Revert "Update loopback version to support check" ([containerd#96](containerd/go-cni#96))
  * [`186662c`](containerd/go-cni@186662c) Revert "Update loopback version to support check"
* Use revive instead of golint ([containerd#92](containerd/go-cni#92))
  * [`322e8bf`](containerd/go-cni@322e8bf) Use revive instead of golint
* Bump go verion to 1.17 ([containerd#91](containerd/go-cni#91))
  * [`9b78de4`](containerd/go-cni@9b78de4) Bump go verion to 1.17
</p>
</details>

### Dependency Changes

* **github.com/containerd/go-cni**        v1.1.4 -> v1.1.5
* **github.com/containernetworking/cni**  v1.0.1 -> v1.1.0
* **github.com/opencontainers/selinux**   v1.10.0 -> v1.10.1

Previous release can be found at [v1.6.3](https://github.com/containerd/containerd/releases/tag/v1.6.3)

Cleanup leaked shim process

If containerd crashes while creating a container the shim process stays alive and is never
cleaned up. Details are discussed in issue containerd#6860. This fixes the code
to cleanup such shim processes on containerd restart.

Signed-off-by: Amit Barve <ambarve@microsoft.com>

containerd 1.6.3

Welcome to the v1.6.3 release of containerd!

The third patch release for containerd 1.6 includes various fixes and updates.

### Notable Updates

* **Fix panic when configuring tracing plugin** ([containerd#6853](containerd#6853))
* **Improve image pull performance in CRI plugin** ([containerd#6816](containerd#6816))
* **Check for duplicate nspath** ([containerd#6813](containerd#6813))
* **Fix deadlock in cgroup metrics collector** ([containerd#6801](containerd#6801))
* **Mount devmapper xfs file system with "nouuid" option** ([containerd#6731](containerd#6731))
* **Make the temp mount as ready only in container WithVolumes** ([containerd#6730](containerd#6730))
* **Fix deadlock from leaving transaction open in native snapshotter** ([containerd#6727](containerd#6727))
* **Monitor OOMKill events to prevent missing container events** ([containerd#6734](containerd#6734))

See the changelog for complete list of changes

Please try out the release binaries and report any issues at
https://github.com/containerd/containerd/issues.

### Contributors

* Stefan Berger
* Wei Fu
* Akihiro Suda
* Derek McGowan
* Phil Estes
* Kazuyoshi Kato
* Mike Brown
* Sebastiaan van Stijn
* Maksym Pavlenko
* dependabot[bot]
* Cory Snider
* Henry Wang
* Jeremi Piotrowski
* Michael Zappa
* Qiutong Song
* Tõnis Tiigi
* Ye Sijun

### Changes
<details><summary>35 commits</summary>
<p>

* Prepare release notes for v1.6.3 ([containerd#6844](containerd#6844))
  * [`baa386dc0`](containerd@baa386d) Prepare release notes for v1.6.3
* [release/1.6] tracing: fix panic on startup when configured ([containerd#6853](containerd#6853))
  * [`e8da82adc`](containerd@e8da82a) tracing: fix panic on startup when configured
* [release/1.6] CRI: improve image pulling performance ([containerd#6816](containerd#6816))
  * [`1764ea9a2`](containerd@1764ea9) CRI: improve image pulling performance
* [release/1.6] update golang to 1.17.9 ([containerd#6823](containerd#6823))
  * [`9cd76d465`](containerd@9cd76d4) [release/1.6] update golang to 1.17.9
* [release/1.6] check for duplicate nspath possibilities ([containerd#6813](containerd#6813))
  * [`c09cc1242`](containerd@c09cc12) check for duplicate nspath possibilities
* [release/1.6] metrics/cgroups: fix deadlock issue in Add during Collect ([containerd#6801](containerd#6801))
  * [`fe6ba62ce`](containerd@fe6ba62) metrics/cgroups: fix deadlock issue in Add during Collect
* [release/1.6] go.mod: update image-spec to merge-commit of v1 into main ([containerd#6766](containerd#6766))
  * [`8b81a7843`](containerd@8b81a78) [release/1.6] go.mod: update image-spec to merge-commit of v1 into main
* [release/1.6 backport] update runc to 1.1.1 ([containerd#6759](containerd#6759))
  * [`f2ba2041b`](containerd@f2ba204) update runc binary to v1.1.1
  * [`b736b4dab`](containerd@b736b4d) go.mod: github.com/opencontainers/runc v1.1.1
* [release/1.6] CI: add Rocky Linux 8 ([containerd#6752](containerd#6752))
  * [`72f1e58c7`](containerd@72f1e58) CI: add Rocky Linux 8
* [release/1.6] vendor: github.com/containerd/imgcrypt v1.1.4 ([containerd#6739](containerd#6739))
  * [`7ede40c5c`](containerd@7ede40c) [release/1.6] vendor: github.com/containerd/imgcrypt v1.1.4
* [release/1.6 backport] moving up to go-cni v1.1.4 ([containerd#6728](containerd#6728))
  * [`82a12edf2`](containerd@82a12ed) moving up to go-cni v1.1.4
* [release/1.6 backport] Update prometheus client vendor ([containerd#6732](containerd#6732))
  * [`da35c19da`](containerd@da35c19) Test turning off golang CI lint cache
  * [`a0213573b`](containerd@a021357) Add nolint:staticcheck to platform-specific calls
  * [`ad0036ed6`](containerd@ad0036e) Update prometheus client vendor
* [release/1.6 backport] Mount devmapper xfs file system with "nouuid" option. ([containerd#6731](containerd#6731))
  * [`c7bbf316f`](containerd@c7bbf31) Mount devmapper xfs file system with "nouuid" option.
* [release/1.6 backport] Make the temp mount as ready only in container WithVolumes ([containerd#6730](containerd#6730))
  * [`a1de89c3e`](containerd@a1de89c) Make the temp mount as ready only in container WithVolumes
* [release/1.6 backport] native: fix deadlock from leaving transactions open ([containerd#6727](containerd#6727))
  * [`28b44826b`](containerd@28b4482) native: fix deadlock from leaving transactions open
* [release/1.6 backport] cgroup2: monitor OOMKill instead of OOM to prevent missing container events ([containerd#6734](containerd#6734))
  * [`5538be6cf`](containerd@5538be6) cgroup2: monitor OOMKill instead of OOM to prevent missing container OOM events
</p>
</details>

### Changes from containerd/go-cni
<details><summary>8 commits</summary>
<p>

* moving up to latest CNI plugin release ([containerd#90](containerd/go-cni#90))
  * [`689fcd9`](containerd/go-cni@689fcd9) moving up to latest CNI plugin release
* Fix Loopback Version ([containerd#88](containerd/go-cni#88))
  * [`9ebcec1`](containerd/go-cni@9ebcec1) Update loopback version to support check
* Update comment for capabilities ([containerd#89](containerd/go-cni#89))
  * [`a4d8d38`](containerd/go-cni@a4d8d38) update comment for capabilities
* Add integration test for linux and update go version from 1.16 to 1.17 ([containerd#84](containerd/go-cni#84))
  * [`49aa5ab`](containerd/go-cni@49aa5ab) Add integration test and update go version
</p>
</details>

### Changes from containerd/imgcrypt
<details><summary>13 commits</summary>
<p>

* CHANGES: Updated CHANGES document for 1.1.4 release ([containerd#74](containerd/imgcrypt#74))
  * [`f576654`](containerd/imgcrypt@f576654) CHANGES: Updated CHANGES document for 1.1.4 release
* Bump github.com/containerd/containerd from 1.5.10 to 1.6.1 ([containerd#73](containerd/imgcrypt#73))
  * [`2efa871`](containerd/imgcrypt@2efa871) Bump github.com/containerd/containerd from 1.5.10 to 1.6.1
* images: prepare for typeurl.Any ([containerd#72](containerd/imgcrypt#72))
  * [`f842da4`](containerd/imgcrypt@f842da4) images: prepare for typeurl.Any
  * [`6fdd981`](containerd/imgcrypt@6fdd981) images: Add list of Platforms to CheckAuthorization()
  * [`f440058`](containerd/imgcrypt@f440058) test: Test running of encrypted image only pulled for local platform
* Bump ocicrypt to 1.1.3 ([containerd#71](containerd/imgcrypt#71))
  * [`d4d4684`](containerd/imgcrypt@d4d4684) Bump ocicrypt to 1.1.3
  * [`727850f`](containerd/imgcrypt@727850f) Bump github.com/containerd/containerd from 1.5.9 to 1.5.10
* Bump github.com/containerd/containerd from 1.5.8 to 1.5.9 ([containerd#67](containerd/imgcrypt#67))
  * [`3c7db10`](containerd/imgcrypt@3c7db10) Bump github.com/containerd/containerd from 1.5.8 to 1.5.9
</p>
</details>

### Dependency Changes

* **github.com/containerd/go-cni**            v1.1.3 -> v1.1.4
* **github.com/containerd/imgcrypt**          v1.1.3 -> v1.1.4
* **github.com/containernetworking/plugins**  v1.0.1 -> v1.1.1
* **github.com/containers/ocicrypt**          v1.1.2 -> v1.1.3
* **github.com/miekg/pkcs11**                 v1.0.3 -> v1.1.1
* **github.com/opencontainers/image-spec**    693428a734f5 -> c5a74bcca799
* **github.com/opencontainers/runc**          v1.1.0 -> v1.1.1
* **github.com/prometheus/client_golang**     v1.11.0 -> v1.11.1

Previous release can be found at [v1.6.2](https://github.com/containerd/containerd/releases/tag/v1.6.2)

Backup tag after rearranging commits for PRs

I had already rebased this on latest master but the commits & fixes after rebase still had
to be rearranged according to PRs. This tag is created after making that change and
verifying that tests pass.

Support for CimFS snapshotter

In order to use a CIM layer it needs to be mounted first. We want to share the mounted
layer CIM among different containers started from the same image. This means we can't
mount the CIM from the shim (since that is separate for each container). Hence, we need
this CimFS snapshotter. The CimFS snapshotter will take care of mounting/unmounting the
CIM when new containers are started.

Signed-off-by: Amit Barve <ambarve@microsoft.com>

Support for CIM layer writer

This commit adds a new cimfs differ that will in turn call the cim layer writer from
hcsshim to write container image layers in the CIM format.

Signed-off-by: Amit Barve <ambarve@microsoft.com>

Backup tag before rebasing this branch

This branch has been updated to use containerd 1.6 and it also has some bug fixes after
rebase. At this point the build works fine hence creating a backup tag.