containerd のイメージ取得はどのように動作するか？

次のコマンドがどのように動作するかを調査する。

# crictl pull alpine
Image is up to date for sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec

# crictl images
IMAGE                      TAG                 IMAGE ID            SIZE
docker.io/library/alpine   latest              6dbb9cc540741       2.82MB

まとめ

crictl <- CRI(v1alpha2) -> containerd -> docker registry

crictl が CRI インタフェースを経由して ImageService を実行する
containerd が docker registry にアクセスしてイメージを取得する
1. WWW-Authenticate ヘッダーに基づいてアクセストークンを取得する
2. docker registry v2 の manifest を取得する(GET /v2/<name>/manifests/<reference>)
3. docker registry v2 の config を取得する(GET /v2/<name>/blobs/<digest>)
4. docker registry v2 の layer を取得する(GET /v2/<name>/blobs/<digest>)
ローカルのメタデータを更新する

検証環境

containerd は v1.5.0 を利用する。なお Delve を利用したデバッグができるように自分でビルドした。

containerd をデバッグする手順のメモ - SIerだけど技術やりたいブログ www.kimullaa.com

検証内容のメモ

crictl のログレベルを上げる

crictl のログレベルを debug にする。

# crictl config --set runtime-endpoint="unix:///run/containerd/containerd.sock" --set debug=true

containerd のログレベルを上げる

containerd のログレベルを debug にする。

# containerd config default > /etc/containerd/config.toml

   address = ""
   format = ""
   gid = 0
-  level = "debug"
+  level = ""
   uid = 0

ログとソースコードを読む

crictl は Unix ドメインソケット(unix:///run/containerd/containerd.sock) 経由でリクエスト(PullImageRequest)を投げる。これは gRPC 通信であり、 CRI の v1alpha2 で API が定義されているっぽい。

# crictl pull alpine
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '2s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:alpine,},Auth:nil,SandboxConfig:nil,}
DEBU[0005] PullImageResponse: &PullImageResponse{ImageRef:sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec,}
Image is up to date for sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec

CRI のサーバ側は、v1alpha2 の ImageService を実装しているっぽい。

# journalctl -f -u containerd.service
-- Logs begin at Mon 2021-05-03 01:04:29 JST. --
...
 5月 05 12:33:30 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:30.375792600+09:00" level=info msg="PullImage \"alpine\""
 5月 05 12:33:30 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:30.376316100+09:00" level=debug msg="PullImage using normalized image ref: \"docker.io/library/alpine:latest\""

いちおう Delve でも呼び出し関係を出してみたけど、やっぱり v1alpha2 の ImageService を実装してる。

 1  0x000055d66003a375 in github.com/containerd/containerd/pkg/cri/server.(*criService).PullImage
    at ./pkg/cri/server/image_pull.go:139
 2  0x000055d660046396 in github.com/containerd/containerd/pkg/cri/server.(*instrumentedService).PullImage
    at ./pkg/cri/server/instrumented_service.go:324
 3  0x000055d65fd8607a in k8s.io/cri-api/pkg/apis/runtime/v1alpha2._ImageService_PullImage_Handler.func1
    at ./vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:8520
 4  0x000055d6601cbb18 in github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1
    at ./vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107
 5  0x000055d65fc991ef in k8s.io/cri-api/pkg/apis/runtime/v1alpha2._ImageService_PullImage_Handler
    at ./vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:8522
 6  0x000055d65f2c5e12 in google.golang.org/grpc.(*Server).processUnaryRPC
    at ./vendor/google.golang.org/grpc/server.go:1024
 7  0x000055d65f2c9fb2 in google.golang.org/grpc.(*Server).handleStream
    at ./vendor/google.golang.org/grpc/server.go:1313
 8  0x000055d65f2ddd66 in google.golang.org/grpc.(*Server).serveStreams.func1.1
    at ./vendor/google.golang.org/grpc/server.go:722
 9  0x000055d65ea4ed81 in runtime.goexit
    at /usr/lib/golang/src/runtime/asm_amd64.s:1373

containerd/pkg/cri/server/image_pull.go#PullImage からメインの処理が開始する。

まず、イメージを取得するサーバのドメイン名を解決する。デフォルトでは docker registry にアクセスする。 https://registry-1.docker.io/v2/library/alpine/manifests/latest にアクセスしたが 401 が返ってきたので、次のアクセスでは docker registry のアクセストークンを取得してからアクセスしている。 (remotes/docker/authorizer.goらへんの処理。ログには出てないけど)

~ 続き ~
 5月 05 12:33:30 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:30.382300000+09:00" level=debug msg=resolving host=registry-1.docker.io
 5月 05 12:33:30 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:30.382402400+09:00" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.0 request.method=HEAD url="https://registry-1.docker.io/v2/library/alpine/manifests/latest"
 5月 05 12:33:31 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:31.172415200+09:00" level=debug msg="fetch response received" host=registry-1.docker.io response.header.content-length=157 response.header.content-type=application/json response.header.date="Wed, 05 May 2021 03:33:34 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/alpine:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/library/alpine/manifests/latest"
 5月 05 12:33:31 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:31.172617000+09:00" level=debug msg=Unauthorized header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/alpine:pull\"" host=registry-1.docker.io
 5月 05 12:33:31 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:31.172716900+09:00" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.0 request.method=HEAD url="https://registry-1.docker.io/v2/library/alpine/manifests/latest"
 5月 05 12:33:32 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:32.162113000+09:00" level=debug msg="fetch response received" host=registry-1.docker.io response.header.content-length=1638 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Wed, 05 May 2021 03:33:35 GMT" response.header.docker-content-digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="100;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" url="https://registry-1.docker.io/v2/library/alpine/manifests/latest"
 5月 05 12:33:32 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:32.162329600+09:00" level=debug msg=resolved desc.digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" host=registry-1.docker.io
 5月 05 12:33:32 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:32.162584900+09:00" level=debug msg=fetch digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1638

ということで curl でいうと、以下のようなリクエストを投げている。

# export TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/alpine:pull" | jq -r '.token')
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4363    0  4363    0     0   4442      0 --:--:-- --:--:-- --:--:--  4438

#  curl -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*' -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/library/alpine/manifests/latest
HTTP/1.1 200 OK
Content-Length: 1638
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Docker-Content-Digest: sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
Date: Wed, 05 May 2021 04:44:03 GMT
Strict-Transport-Security: max-age=31536000
RateLimit-Limit: 100;w=21600
RateLimit-Remaining: 98;w=21600

続いて HEAD で取得したイメージの sha256(Docker-Content-Digest) の値をもとに、https://registry-1.docker.io/v2/library/alpine/manifests/sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" にアクセスしている。 (remotes/docker/fetcher.go#Fetch らへん)

~ 続き ~
 5月 05 12:33:32 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:32.176385100+09:00" level=debug msg="do request" digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/v1.5.0 request.method=GET size=1638 url="https://registry-1.docker.io/v2/library/alpine/manifests/sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
 5月 05 12:33:32 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:32.911839200+09:00" level=debug msg="fetch response received" digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=157 response.header.content-type=application/json response.header.date="Wed, 05 May 2021 03:33:36 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/alpine:pull\"" response.status="401 Unauthorized" size=1638 url="https://registry-1.docker.io/v2/library/alpine/manifests/sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
 5月 05 12:33:32 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:32.912110000+09:00" level=debug msg=Unauthorized digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/alpine:pull\"" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1638
 5月 05 12:33:32 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:32.912381500+09:00" level=debug msg="do request" digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/v1.5.0 request.method=GET size=1638 url="https://registry-1.docker.io/v2/library/alpine/manifests/sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
 5月 05 12:33:34 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:34.390447400+09:00" level=debug msg="fetch response received" digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=1638 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Wed, 05 May 2021 03:33:37 GMT" response.header.docker-content-digest="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="100;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=1638 url="https://registry-1.docker.io/v2/library/alpine/manifests/sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"

ということで、curl でいうと以下のリクエストを投げている。

# curl -X GET -H 'Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*' -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/library/alpine/manifests/sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f  | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1638  100  1638    0     0   1931      0 --:--:-- --:--:-- --:--:--  1931
{
  "manifests": [
    {
      "digest": "sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:ea73ecf48cd45e250f65eb731dd35808175ae37d70cca5d41f9ef57210737f04",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      },
      "size": 528
    },
    ...

レスポンス見たらわかるが manifests が複数ある。docker は CPU アーキテクチャごとにイメージが異なるため、自身のアーキテクチャに一致するものをフィルタする。今回は amd64。 (pull.go#Pull とか pull.go#Pull らへん)

フィルタした sha256 を基に、https://registry-1.docker.io/v2/library/alpine/manifests/sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748 にアクセスする。

~ 続き ~
 5月 05 12:33:34 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:34.408566200+09:00" level=debug msg=fetch digest="sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748" mediatype=application/vnd.docker.distribution.manifest.v2+json size=528
 5月 05 12:33:34 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:34.422049500+09:00" level=debug msg="do request" digest="sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.5.0 request.method=GET size=528 url="https://registry-1.docker.io/v2/library/alpine/manifests/sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748"
 5月 05 12:33:34 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:34.673509700+09:00" level=debug msg="fetch response received" digest="sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=528 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 05 May 2021 03:33:37 GMT" response.header.docker-content-digest="sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="99;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=528 url="https://registry-1.docker.io/v2/library/alpine/manifests/sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748"
 5月 05 12:33:34 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:34.693458700+09:00" level=debug msg=fetch digest="sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec" mediatype=application/vnd.docker.container.image.v1+json size=1472

ということで、curl でいうと以下のリクエストを投げている。

# curl -X GET -H 'Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*' -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/library/alpine/manifests/sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748  | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   528  100   528    0     0    590      0 --:--:-- --:--:-- --:--:--   590
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1472,
    "digest": "sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2811969,
      "digest": "sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
    }
  ]
}

次に config を取得する。先ほどのレスポンスをもとに、https://registry-1.docker.io/v2/library/alpine/blobs/sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec にリクエストする。

~ 続き ~
 5月 05 12:33:34 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:34.708595200+09:00" level=debug msg="do request" digest="sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/v1.5.0 request.method=GET size=1472 url="https://registry-1.docker.io/v2/library/alpine/blobs/sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec"
 5月 05 12:33:35 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:35.164635100+09:00" level=debug msg="fetch response received" digest="sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=1756453 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=64a6e8930b3d5f40-NRT response.header.cf-request-id=09dc2fafe700005f4084aed000000001 response.header.connection=keep-alive response.header.content-length=1472 response.header.content-type=application/octet-stream response.header.date="Wed, 05 May 2021 03:33:38 GMT" response.header.etag="\"2e65497b85dabd677d01d4df0a89678f\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 05 May 2021 07:33:38 GMT" response.header.last-modified="Wed, 14 Apr 2021 19:20:36 GMT" response.header.server=cloudflare response.header.set-cookie="__cfduid=dcf891ca02badb382e6ed884341655f131620185618; expires=Fri, 04-Jun-21 03:33:38 GMT; path=/; domain=.production.cloudflare.docker.com; HttpOnly; SameSite=Lax; Secure" response.header.vary=Accept-Encoding response.header.x-amz-id-2="OAm2/xpMWc8SEjsDPkfd64VrLWk5GTInVj/ioJKHQHUak3vI8kWdew3mLRc35FuToDRQtFZsVaA=" response.header.x-amz-request-id=T9VEHXYSG2D7G0WG response.header.x-amz-version-id=.MqwttkqRpn1QFvubwW_SLNQTVFDgFlJ response.status="200 OK" size=1472 url="https://registry-1.docker.io/v2/library/alpine/blobs/sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec"

curl でいうと以下のリクエストを投げている(以降、-L オプション付けないとリダイレクトされないので注意)。このファイルが以降どう使われるのかはわからなかった(Engine API v1.19に似た属性があったので、コンテナイメージのメタ情報？)、今度調べる。

# curl -X GET -L -H 'Accept: application/vnd.docker.container.image.v1+json, */*' -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/library/alpine/blobs/sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec
{"architecture":"amd64","config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh"],"Image":"sha256:d3d4554f8b07cf59894bfb3551e10f89a559b24ee0992c4900c54175596b1389","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"container":"60a3cdd128a8b373b313ed3e1083ff45e6badaad5dca5187282b005c38d04712","container_config":{"Hostname":"60a3cdd128a8","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","#(nop) ","CMD [\"/bin/sh\"]"],"Image":"sha256:d3d4554f8b07cf59894bfb3551e10f89a559b24ee0992c4900c54175596b1389","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":{}},"created":"2021-04-14T19:19:39.643236135Z","docker_version":"19.03.12","history":[{"created":"2021-04-14T19:19:39.267885491Z","created_by":"/bin/sh -c #(nop) ADD file:8ec69d882e7f29f0652d537557160e638168550f738d0d49f90a7ef96bf31787 in / "},{"created":"2021-04-14T19:19:39.643236135Z","created_by":"/bin/sh -c #(nop)  CMD [\"/bin/sh\"]","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116"]}

ちなみにここらへんからリクエストの結果は、/var/lib/containerd/io.containerd.content.v1.content 配下にキャッシュされているっぽい。docker registry v2 にキャッシュとか考えろと書いてあったのでそういうことかな？

# ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/
540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba  6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec
69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f  def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748

# cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1472,
      "digest": "sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2811969,
         "digest": "sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
      }
   ]

最後にレイヤを取得する。先ほどのレスポンスをもとに、https://registry-1.docker.io/v2/library/alpine/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba にリクエストする。このとき、圧縮ファイルも展開する。 (unpacker.go#unpack() らへん。圧縮ファイルの展開は archive/compression/compression.go#gzipDecompress() らへん)

~ 続き ~
 5月 05 12:33:35 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:35.190646600+09:00" level=debug msg="event published" ns=k8s.io topic=/snapshot/prepare type=containerd.events.SnapshotPrepare
 5月 05 12:33:35 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:35.190789300+09:00" level=debug msg=fetch digest="sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2811969
 5月 05 12:33:35 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:35.200473000+09:00" level=debug msg="do request" digest="sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/v1.5.0 request.method=GET size=2811969 url="https://registry-1.docker.io/v2/library/alpine/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
 5月 05 12:33:35 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:35.445804300+09:00" level=debug msg="fetch response received" digest="sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=1756453 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=64a6e894e8e65f40-NRT response.header.cf-request-id=09dc2fb10d00005f4067929000000001 response.header.connection=keep-alive response.header.content-length=2811969 response.header.content-type=application/octet-stream response.header.date="Wed, 05 May 2021 03:33:38 GMT" response.header.etag="\"cbec077710e46c3f3d65532a605c9373\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 05 May 2021 07:33:38 GMT" response.header.last-modified="Wed, 14 Apr 2021 17:59:29 GMT" response.header.server=cloudflare response.header.set-cookie="__cfduid=d8ea90e0d7358016cb1345fba811ad8f41620185618; expires=Fri, 04-Jun-21 03:33:38 GMT; path=/; domain=.production.cloudflare.docker.com; HttpOnly; SameSite=Lax; Secure" response.header.vary=Accept-Encoding response.header.x-amz-id-2="hpmILcd2Wa9bm73NxiN7umNSO1Fzj4rDY0XcjU1yOimtPnw1VKOt//1HjZmfH7jWvh0rIDE+Q3c=" response.header.x-amz-request-id=T9V0BDDF5G4BZFS6 response.header.x-amz-version-id=.YPE0QWWMApuYCA.Ws168iQ.l3P5h_JE response.status="200 OK" size=2811969 url="https://registry-1.docker.io/v2/library/alpine/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba"
 5月 05 12:33:35 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:35.941223400+09:00" level=debug msg="using pigz for decompression"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.027032200+09:00" level=debug msg="diff applied" d=86.7594ms digest="sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba" media=application/vnd.docker.image.rootfs.diff.tar.gzip size=2811969
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.033586300+09:00" level=debug msg="event published" ns=k8s.io topic=/snapshot/commit type=containerd.events.SnapshotCommit
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.038052200+09:00" level=debug msg="image 6dbb9cc54074106d46d4ccb330f2a40a682d49dda5unpacked" chainID="sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116" config="sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec"

curl でいうと以下のリクエストを投げている。なお、containerd は圧縮ファイルをバッファで処理してるっぽいので実体はなさそう(archive/compression/compression.go#gzipDecompress())。展開後にどういう処理をしてるのかよくわかってないので、また今度調べる。

# curl -X GET -L -o tmp.tar.gz  -H "Authorization: Bearer $TOKEN"  -H 'Accept: application/vnd.docker.image.rootfs.diff.tar.gzip, */*' https://registry-1.docker.io/v2/library/alpine/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba

# tar -xf tmp.tar.gz
# ls
bin  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  tmp.tar.gz  usr  var

イメージをメタデータ用の DB に登録する。

~ 続き ~
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.038098900+09:00" level=debug msg="create image" name="docker.io/library/alpine:latest" target="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.040274900+09:00" level=debug msg="event published" ns=k8s.io topic=/images/create type=containerd.services.images.v1.ImageCreate
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.040429700+09:00" level=debug msg="Received containerd event timestamp - 2021-05-05 03:33:36.0402698 +0000 UTC, namespace - \"k8s.io\", topic - \"/images/create\""
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.040501900+09:00" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/library/alpine:latest,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.042677100+09:00" level=debug msg="create image" name="sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec" target="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.044763800+09:00" level=debug msg="event published" ns=k8s.io topic=/images/create type=containerd.services.images.v1.ImageCreate
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.044805900+09:00" level=debug msg="Received containerd event timestamp - 2021-05-05 03:33:36.0447599 +0000 UTC, namespace - \"k8s.io\", topic - \"/images/create\""
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.044834300+09:00" level=info msg="ImageCreate event &ImageCreate{Name:sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.045354200+09:00" level=debug msg="create image" name="docker.io/library/alpine:latest" target="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.047002100+09:00" level=debug msg="event published" ns=k8s.io topic=/images/update type=containerd.services.images.v1.ImageUpdate
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.047131900+09:00" level=debug msg="Received containerd event timestamp - 2021-05-05 03:33:36.0469981 +0000 UTC, namespace - \"k8s.io\", topic - \"/images/update\""
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.047166300+09:00" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/library/alpine:latest,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.047434000+09:00" level=debug msg="create image" name="docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f" target="sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f"
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.049158500+09:00" level=debug msg="event published" ns=k8s.io topic=/images/create type=containerd.services.images.v1.ImageCreate
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.049189700+09:00" level=debug msg="Received containerd event timestamp - 2021-05-05 03:33:36.0491548 +0000 UTC, namespace - \"k8s.io\", topic - \"/images/create\""
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.049215700+09:00" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"

DB には Bolt という組み込みの DB を利用しているっぽい。

# lsof -p 17550
COMMAND     PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
container 17550 root  cwd       DIR              253,0      283      128 /
container 17550 root  rtd       DIR              253,0      283      128 /
container 17550 root  txt       REG              253,0 64295712 45962218 /usr/local/bin/containerd
container 17550 root  mem-W     REG              253,0   262144 57152696 /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db

以降は完了メッセージを出してるだけっぽいので省略する。

~ 続き ~
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.049675400+09:00" level=debug msg="Pulled image \"alpine\" with image id \"sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec\", repo tag \"docker.io/library/alpine:latest\", repo digest \"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f\""
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.049714100+09:00" level=info msg="PullImage \"alpine\" returns image reference \"sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec\""
 5月 05 12:33:36 localhost.localdomain containerd[13947]: time="2021-05-05T12:33:36.265149700+09:00" level=debug msg="garbage collected" d=5.0735ms

まとめ

CRI と docker registry v2 を理解したほうが良さそう。

docker image の構造と docker registry v2 については後ほど調べた。

Docker コンテナのレイヤー構造について - SIerだけど技術やりたいブログ www.kimullaa.com