[Dummgelaufen007]

Let BuildKit fetch the sources with ADD (The Docker Way)

Each recipe names its source in one ADD line above its RUN, next to its ARGs and bump lines. Docker fetches and checks all sources at the start of the build, so a broken source stops the build in its first seconds instead of after the steps before it.

How BuildKit fetches a source

This PR adds # syntax=docker/dockerfile:1 as first line, so BuildKit fetches the current Dockerfile frontend instead of using the one built into the local Docker; ADD --unpack below needs syntax 1.17 (ADD).

ADD --checksum=<hash> <url> <dest> downloads a file and checks it; --unpack=true extracts it. A git URL (git:// or ending in .git) with #<commit> is fetched as a checkout of that single commit and checked against it. No ADD source depends on an earlier step, so BuildKit fetches them all at once when the build starts.

What changes in the Dockerfile

Archives. wget, sha256sum and tar become one line; libaribb24 now enters aribb24-*, the directory inside its archive.

ADD --checksum=sha256:$VMAF_SHA256 --unpack=true $VMAF_URL /
RUN \
  cd vmaf-*/libvmaf && \

Git sources. git clone and git checkout become an ADD into the directory the recipe already enters.

ADD --checksum=$LIBGME_COMMIT $LIBGME_URL#$LIBGME_COMMIT /game-music-emu

x264 and uavs3d count their git history for their version string (x264, uavs3d), so they keep .git and fetch the history before building; the version string stays the same.

ADD --keep-git-dir=true --checksum=$X264_VERSION $X264_URL#$X264_VERSION /x264
RUN \
  cd x264 && \
  git fetch --unshallow --filter=tree:0 origin +refs/heads/master:refs/remotes/origin/master && \

libmodplug and opencore-amr now use the same sourceforge link form as lame (.../download), which ADD fetches reliably.

Other lines. wget, WGET_OPTS and TAR_OPTS go; curl, tar and git stay for libjxl deps.sh, x264 and uavs3d.

What is given up

• BuildKit does not retry a failed download, where wget did.
• fdk-aac is always downloaded, even without ENABLE_FDKAAC.

Docker version

This PR leans on newer Docker features:

• the # syntax line, for ADD --unpack; every build looks the frontend up on Docker Hub.
• ADD with a git source at a commit, and --keep-git-dir for x264 and uavs3d.

Tested

A full build with --no-cache completed, including the sanity tests. The x264 and uavs3d version strings match a full clone and versions.json lists the commits of all git sources.

Research and wording by Claude (Anthropic), directed and reviewed by me.

[wader]

What is the behavior add ADD in these cases?

[Dummgelaufen007]

BuildKit's HTTP source does not retry a failed download; the PR text says so under limits. The retry helper in BuildKit (retryhandler/retry.go) is used only by the registry resolver and auth provider, not by source/http/source.go. A transient error fails the build step; a rerun picks up the already cached downloads.

[wader]

What is the behavior add ADD in this case?

[Dummgelaufen007]

ADD --unpack extracts with BuildKit's own untar. It sets owner UID/GID only when --chown is given; otherwise no chown options are passed (unpack.go, ChownOpts only when a user is set).

[wader]

Does syntax version include what protocols ADD support etc? when was this released as stable?

[Dummgelaufen007]

docker/dockerfile:1 pulls the latest stable 1.x frontend, independent of the local Docker version (reference, syntax). The minimum versions per flag are listed in the ADD reference: --keep-git-dir 1.1, --checksum 1.6, --unpack 1.17. 1.17.0 was released as stable on 2025-06-17 (release).

[wader]

Not sure i followed why this fetch was needed, ADD does not checkout properly?

[Dummgelaufen007]

The checkout is fine, but ADD fetches only the commit. x264's version.sh counts the commits of HEAD and of origin/master..HEAD (version.sh#L10-L12), so without the full history and origin/master the version string changes. The fetch restores both; --filter=tree:0 keeps it to commits only. The resulting version string matches a full clone.

[wader]

Overall like this better than #840