This adds the code used by buildx and compose into the default CLI
program to help normalize the usage of these APIs and allow code reuse
between projects. It also allows these projects to benefit from
improvements or changes that may be made by another team.
At the moment, these APIs are a pretty thin layer on the OTEL SDK. It
configures an additional exporter to a docker endpoint that's used for
usage collection and is only active if the option is configured in
docker desktop.
This also upgrades the OTEL version to v1.19 which is the one being used
by buildkit, buildx, compose, etc.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Go's `net` package [will unlink][1] for us, as long as we used Listen &
friends to create the Unix socket.
Go will even skip the unlink when the socket appears to be abstract
(starts with a NUL, represented by an @), though we must be cautious to
only create sockets with an abstract address on platforms that actually
support it -- this caused [several][2] [bugs][3] before.
[1]: https://pkg.go.dev/net#UnixListener.SetUnlinkOnClose
[2]: https://github.com/docker/cli/pull/4783
[3]: https://github.com/docker/cli/pull/4863
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
This changes things to rely on a plugin server that manages all
connections made to the server.
An optional handler can be passed into the server when the caller wants
to do extra things with the connection.
It is the caller's responsibility to close the server.
When the server is closed, first all existing connections are closed
(and new connections are prevented).
Now the signal loop only needs to close the server and not deal with
`net.Conn`'s directly (or double-indirects as the case was before this
change).
The socket, when present in the filesystem, is no longer unlinked
eagerly, as reconnections require it to be present for the lifecycle of
the plugin server.
Co-authored-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
CodeQL autobuild uses the makefile, but outside of a container, so let's
set this variable to prevent it having to wait 10 seconds;
Use "make dev" to start an interactive development container,
use "make -f docker.Makefile " to execute this target
in a container, or set DISABLE_WARN_OUTSIDE_CONTAINER=1 to
disable this warning.
Press Ctrl+C now to abort, or wait for the script to continue..
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
CodeQL 2.16.4's auto-build added support for multi-module repositories,
and is trying to be smart by searching for modules in every directory,
including vendor directories. If no module is found, it's creating one
which is ... not what we want, so let's give it a "go.mod".
Here's from a run in CI;
/opt/hostedtoolcache/CodeQL/2.16.4/x64/codeql/codeql version --format=json
{
"productName" : "CodeQL",
"vendor" : "GitHub",
"version" : "2.16.4",
"sha" : "9727ba3cd3d5a26f8b9347bf3c3eb4f565ac077b",
"branches" : [
"codeql-cli-2.16.4"
],
"copyright" : "Copyright (C) 2019-2024 GitHub, Inc.",
"unpackedLocation" : "/opt/hostedtoolcache/CodeQL/2.16.4/x64/codeql",
"configFileLocation" : "/home/runner/.config/codeql/config",
"configFileFound" : false,
"features" : {
"analysisSummaryV2Option" : true,
"buildModeOption" : true,
"bundleSupportsIncludeDiagnostics" : true,
"featuresInVersionResult" : true,
"indirectTracingSupportsStaticBinaries" : false,
"informsAboutUnsupportedPathFilters" : true,
"supportsPython312" : true,
"mrvaPackCreate" : true,
"threatModelOption" : true,
"traceCommandUseBuildMode" : true,
"v2ramSizing" : true,
"mrvaPackCreateMultipleQueries" : true,
"setsCodeqlRunnerEnvVar" : true
}
}
With 2.16.4, first it is unable to correlate files with the project, considering
them "stray" files;
Attempting to automatically build go code
/opt/hostedtoolcache/CodeQL/2.16.4/x64/codeql/go/tools/autobuild.sh
2024/03/16 15:54:34 Autobuilder was built with go1.22.0, environment has go1.21.8
2024/03/16 15:54:34 LGTM_SRC is /home/runner/work/cli/cli
2024/03/16 15:54:34 Found no go.work files in the workspace; looking for go.mod files...
2024/03/16 15:54:34 Found stray Go source file in cli/cobra.go.
2024/03/16 15:54:34 Found stray Go source file in cli/cobra_test.go.
2024/03/16 15:54:34 Found stray Go source file in cli/command/builder/client_test.go.
2024/03/16 15:54:34 Found stray Go source file in cli/command/builder/cmd.go.
...
It then tries to build the binary, but in go modules mode, which fails (it also
seems to be doing this for each and every directory);
Use "make dev" to start an interactive development container,
use "make -f docker.Makefile " to execute this target
in a container, or set DISABLE_WARN_OUTSIDE_CONTAINER=1 to
disable this warning.
Press Ctrl+C now to abort, or wait for the script to continue..
./scripts/build/binary
Building static docker-linux-amd64
+ go build -o build/docker-linux-amd64 -tags osusergo pkcs11 -ldflags -X "github.com/docker/cli/cli/version.GitCommit=38c3ff6" -X "github.com/docker/cli/cli/version.BuildTime=2024-03-16T17:20:38Z" -X "github.com/docker/cli/cli/version.Version=38c3ff6.m" -extldflags -static -buildmode=pie github.com/docker/cli/cmd/docker
cannot find package "github.com/docker/cli/cmd/docker" in any of:
/opt/hostedtoolcache/go/1.21.8/x64/src/github.com/docker/cli/cmd/docker (from $GOROOT)
/home/runner/go/src/github.com/docker/cli/cmd/docker (from $GOPATH)
make: *** [Makefile:62: binary] Error 1
2024/03/16 17:20:38 Running /usr/bin/make [make] failed, continuing anyway: exit status 2
2024/03/16 17:20:38 Build failed, continuing to install dependencies.
2024/03/16 17:20:38 The code in vendor/gotest.tools/v3/skip seems to be missing a go.mod file. Attempting to initialize one...
2024/03/16 17:20:38 Import path is 'github.com/docker/cli'
If also seems to be doing this for ... every package?
cat 0_codeql.log | grep 'you are not in a container' | wc -l
497
After which it starts to create modules out of every directory;
The code in internal/test/network seems to be missing a go.mod file. Attempting to initialize one...
The code in internal/test/notary seems to be missing a go.mod file. Attempting to initialize one...
The code in internal/test/output seems to be missing a go.mod file. Attempting to initialize one...
The code in opts seems to be missing a go.mod file. Attempting to initialize one...
The code in service seems to be missing a go.mod file. Attempting to initialize one...
The code in service/logs seems to be missing a go.mod file. Attempting to initialize one...
The code in templates seems to be missing a go.mod file. Attempting to initialize one...
The code in vendor seems to be missing a go.mod file. Attempting to initialize one...
The code in vendor/dario.cat seems to be missing a go.mod file. Attempting to initialize one...
The code in vendor/dario.cat/mergo seems to be missing a go.mod file. Attempting to initialize one...
...
Skipping dependency package regexp.
Skipping dependency package github.com/opencontainers/go-digest.
Skipping dependency package github.com/distribution/reference.
Extracting /home/runner/work/cli/cli/cli/command/go.mod
Done extracting /home/runner/work/cli/cli/cli/command/go.mod (1ms)
Extracting /home/runner/work/cli/cli/cli/command/go.mod
Done extracting /home/runner/work/cli/cli/cli/command/go.mod (0ms)
Extracting /home/runner/work/cli/cli/cli/command/go.mod
Done extracting /home/runner/work/cli/cli/cli/command/go.mod (0ms)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
We originally added this -compat to keep a consistent format of
the vendor.mod files for cases where there were differences
between go versions.
I don't think we really need this anymore, so let's remove.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diffs:
- https://github.com/protocolbuffers/protobuf-go/compare/v1.31.0...v1.33.0
- https://github.com/golang/protobuf/compare/v1.5.3...v1.5.4
From the Go security announcement list;
> Version v1.33.0 of the google.golang.org/protobuf module fixes a bug in
> the google.golang.org/protobuf/encoding/protojson package which could cause
> the Unmarshal function to enter an infinite loop when handling some invalid
> inputs.
>
> This condition could only occur when unmarshaling into a message which contains
> a google.protobuf.Any value, or when the UnmarshalOptions.UnmarshalUnknown
> option is set. Unmarshal now correctly returns an error when handling these
> inputs.
>
> This is CVE-2024-24786.
In a follow-up post;
> A small correction: This vulnerability applies when the UnmarshalOptions.DiscardUnknown
> option is set (as well as when unmarshaling into any message which contains a
> google.protobuf.Any). There is no UnmarshalUnknown option.
>
> In addition, version 1.33.0 of google.golang.org/protobuf inadvertently
> introduced an incompatibility with the older github.com/golang/protobuf
> module. (https://github.com/golang/protobuf/issues/1596) Users of the older
> module should update to github.com/golang/protobuf@v1.5.4.
govulncheck results in our code shows that this does not affect the CLI:
govulncheck ./...
Scanning your code and 448 packages across 72 dependent modules for known vulnerabilities...
=== Symbol Results ===
No vulnerabilities found.
Your code is affected by 0 vulnerabilities.
This scan also found 1 vulnerability in packages you import and 0
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The command path sent for `docker build` should be `docker` rather than
`docker build` to be consistent with the other command paths.
* `docker buildx build` has a command path of `docker buildx`
* `docker builder build` has a command path of `docker builder`
* `docker image build` has a command path of `docker image`
The reason this gets set to `docker buildx` rather than `docker buildx
build` is because the `build` portion of the command path is processed
by the plugin. So the command path only contains the portions of the
command path that were processed by this tool.
Since the `build` of `docker build` gets forwarded to `buildx`, it is
not included in the command path.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Mark pulling legacy image formats as deprecated, and describe the
DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE environment variable.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This code was only used as part of container.RunStats, so moving the code
there instead as a non-exported type. The actual use also did not have to
handle concurrency, so the mutex is removed in the new location.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Bjorn Neergaard
Jonathan A. Sternberg
Sebastiaan van Stijn
Laura Brehm
Paweł Gronowski
Brian Goff
Alano Terblanche
Mathias Duedahl
Albin Kerouanton
Sebastiaan van Stijn
David Karlsson
Zhuo Zhi
CrazyMax