This changes things to rely on a plugin server that manages all
connections made to the server.
An optional handler can be passed into the server when the caller wants
to do extra things with the connection.
It is the caller's responsibility to close the server.
When the server is closed, first all existing connections are closed
(and new connections are prevented).
Now the signal loop only needs to close the server and not deal with
`net.Conn`'s directly (or double-indirects as the case was before this
change).
The socket, when present in the filesystem, is no longer unlinked
eagerly, as reconnections require it to be present for the lifecycle of
the plugin server.
Co-authored-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
When a plugin is invoked, the docker cli will now set
`OTEL_RESOURCE_ATTRIBUTES` to pass OTEL resource attribute names to the
plugin as additional resource attributes. At the moment, the only
resource attribute passed is `cobra.command_path`.
All resource attributes passed by the CLI are prepended with the
namespace `docker.cli` to avoid clashing with existing ones the plugin
uses or ones defined by the user.
For aliased commands like the various builder commands, the command path
is overwritten to match with the original name (such as `docker
builder`) instead of the forwarded name (such as `docker buildx build`).
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
The "github.com/docker/distribution" module moved to the distribution
org ("github.com/docker/distribution/v3"), and the new module deprecated
and removed the uuid package in favor of Google's UUID package.
While we still depend on the old module through packages and as an indirect
dependency, we may want to try avoid using it.
This patch replaces the use for the socket package, and replaces it for a
local utility, taking the same approach as `stringid.GenerateRandomID()`,
which should be random enough for this purpose.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Seems that OpenBSD behaves like darwin and requires to unlink all
socket, after it was used.
Tested on OpenBSD 7.4
Signed-off-by: Kirill A. Korinsky <kirill@korins.ky>
Adds a new plugin to the e2e plugins that simulates an older
plugin binary and a test suite to ensure older plugin binaries
keep behaving the same with newer CLI versions.
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
This reverts commit ef5e5fa03f.
Running new plugins under a new pgid isn't a viable solution due to
it causing issues with plugin processes attempting to read from the
TTY (see: https://github.com/moby/moby/issues/47073).
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
Changes were made in 1554ac3b5f to provide
a mechanism for the CLI to notify running plugin processes that they
should exit, in order to improve the general CLI/plugin UX. The current
implementation boils down to:
1. The CLI creates a socket
2. The CLI executes the plugin
3. The plugin connects to the socket
4. (When) the CLI receives a termination signal, it uses the socket to
notify the plugin that it should exit
5. The plugin's gets notified via the socket, and cancels it's `cmd.Context`,
which then gets handled appropriately
This change works in most cases and fixes the issue it sets out to solve
(see: https://github.com/docker/compose/pull/11292) however, in the case
where the user has a TTY attached and the plugin is not already handling
received signals, steps 4+ changes:
4. (When) the CLI receives a termination signal, before it can use the
socket to notify the plugin that it should exit, the plugin process
also receives a signal due to sharing the pgid with the CLI
Since we now have a proper "job control" mechanism, we can simplify the
scenarios by executing the plugins with their own process group id,
thereby removing the "double notification" issue and making it so that
plugins can handle the same whether attached to a TTY or not.
In order to make this change "plugin-binary" backwards-compatible, in
the case that a plugin does not connect to the socket, the CLI passes
the signal to the plugin process.
Co-authored-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
As macOS does not support the abstract socket namespace, use a temporary
socket in $TMPDIR to connect with the plugin. Ensure this socket is
cleaned up even in the case of crash/ungraceful termination by removing
it after the first connection is accepted.
Co-authored-by: Laura Brehm <laurabrehm@hey.com>
Signed-off-by: Bjorn Neergaard <bjorn.neergaard@docker.com>
This is a follow-up to 0e73168b7e
This repository is not yet a module (i.e., does not have a `go.mod`). This
is not problematic when building the code in GOPATH or "vendor" mode, but
when using the code as a module-dependency (in module-mode), different semantics
are applied since Go1.21, which switches Go _language versions_ on a per-module,
per-package, or even per-file base.
A condensed summary of that logic [is as follows][1]:
- For modules that have a go.mod containing a go version directive; that
version is considered a minimum _required_ version (starting with the
go1.19.13 and go1.20.8 patch releases: before those, it was only a
recommendation).
- For dependencies that don't have a go.mod (not a module), go language
version go1.16 is assumed.
- Likewise, for modules that have a go.mod, but the file does not have a
go version directive, go language version go1.16 is assumed.
- If a go.work file is present, but does not have a go version directive,
language version go1.17 is assumed.
When switching language versions, Go _downgrades_ the language version,
which means that language features (such as generics, and `any`) are not
available, and compilation fails. For example:
# github.com/docker/cli/cli/context/store
/go/pkg/mod/github.com/docker/cli@v25.0.0-beta.2+incompatible/cli/context/store/storeconfig.go:6:24: predeclared any requires go1.18 or later (-lang was set to go1.16; check go.mod)
/go/pkg/mod/github.com/docker/cli@v25.0.0-beta.2+incompatible/cli/context/store/store.go:74:12: predeclared any requires go1.18 or later (-lang was set to go1.16; check go.mod)
Note that these fallbacks are per-module, per-package, and can even be
per-file, so _(indirect) dependencies_ can still use modern language
features, as long as their respective go.mod has a version specified.
Unfortunately, these failures do not occur when building locally (using
vendor / GOPATH mode), but will affect consumers of the module.
Obviously, this situation is not ideal, and the ultimate solution is to
move to go modules (add a go.mod), but this comes with a non-insignificant
risk in other areas (due to our complex dependency tree).
We can revert to using go1.16 language features only, but this may be
limiting, and may still be problematic when (e.g.) matching signatures
of dependencies.
There is an escape hatch: adding a `//go:build` directive to files that
make use of go language features. From the [go toolchain docs][2]:
> The go line for each module sets the language version the compiler enforces
> when compiling packages in that module. The language version can be changed
> on a per-file basis by using a build constraint.
>
> For example, a module containing code that uses the Go 1.21 language version
> should have a `go.mod` file with a go line such as `go 1.21` or `go 1.21.3`.
> If a specific source file should be compiled only when using a newer Go
> toolchain, adding `//go:build go1.22` to that source file both ensures that
> only Go 1.22 and newer toolchains will compile the file and also changes
> the language version in that file to Go 1.22.
This patch adds `//go:build` directives to those files using recent additions
to the language. It's currently using go1.19 as version to match the version
in our "vendor.mod", but we can consider being more permissive ("any" requires
go1.18 or up), or more "optimistic" (force go1.21, which is the version we
currently use to build).
For completeness sake, note that any file _without_ a `//go:build` directive
will continue to use go1.16 language version when used as a module.
[1]: 58c28ba286/src/cmd/go/internal/gover/version.go (L9-L56)
[2]; https://go.dev/doc/toolchain#:~:text=The%20go%20line%20for,file%20to%20Go%201.22
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The cli/command package defined two option-types with the same signature.
This patch creates a new type instead (CLIOption), and makes the existing
types an alias for this (deprecating their old names).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously, long lived CLI plugin processes weren't
properly handled
(see: https://github.com/docker/cli/issues/4402)
resulting in plugin processes being left behind
running, after the CLI process exits.
This commit changes the plugin handling code to open
an abstract unix socket before running the plugin and
passing it to the plugin process, and changes the
signal handling on the CLI side to close this socket
which tells the plugin that it should exit.
This implementation makes use of sockets instead of
simply setting PDEATHSIG on the plugin process
so that it will work on both BSDs, assorted UNIXes
and Windows.
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
The flag-set that was returned is a pointer to the command's Flags(), which
is in itself passed by reference (as it is modified / set up).
This patch removes the flags return, to prevent assuming it's different than
the command's flags.
While SetupRootCommand is exported, a search showed that it's only used internally,
so changing the signature should not be a problem.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This field was marked deprecated in 977d3ae046,
which is part of Docker 20.10 and up.
This patch removes the field.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
the "golang.org/x/sys/execabs" package was introduced to address a security
issue on Windows, and changing the default behavior of os/exec was considered
a breaking change. go1.19 applied the behavior that was previously implemented
in the execabs package;
from the release notes: https://go.dev/doc/go1.19#os-exec-path
> Command and LookPath no longer allow results from a PATH search to be found
> relative to the current directory. This removes a common source of security
> problems but may also break existing programs that depend on using, say,
> exec.Command("prog") to run a binary named prog (or, on Windows, prog.exe)
> in the current directory. See the os/exec package documentation for information
> about how best to update such programs.
>
> On Windows, Command and LookPath now respect the NoDefaultCurrentDirectoryInExePath
> environment variable, making it possible to disable the default implicit search
> of “.” in PATH lookups on Windows systems.
With those changes, we no longer need to use the execabs package, and we can
switch back to os/exec.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This field was marked deprecated in 977d3ae046,
which is part of v20.10 and up, but the comment was missing a newline before
the deprecation message, which may be picked up by IDEs, but is not matching
the correct format, so may not be picked up by linters.
This patch fixes the format, to make sure linters pick up that the field is
deprecated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This reverts commit 62f2358b99.
Spawning a goroutine for each iteration in the loop when listing
plugins is racy unfortunately. `plugins` slice is protected with
a mutex so not sure why it fails.
I tried using a channel to collect the plugins instead of a slice
to guarantee that they will be appended to the list in the order
they are processed but no dice.
I also tried without errgroup package and simply use sync.WaitGroup
but same. I have also created an extra channel to receive errors
from the goroutines but racy too.
I think the change in this function is not related to the race
condition but newPlugin is. So revert in the meantime :(
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
We are currently loading plugin command stubs for every
invocation which still has a significant performance hit.
With this change we are doing this operation only if cobra
completion arg request is found.
- 20.10.23: `docker --version` takes ~15ms
- 23.0.1: `docker --version` takes ~93ms
With this change `docker --version` takes ~9ms
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
We are currently loading plugin commands stubs for every
command invocation to add support for Cobra v2 completion.
This cause a significant performance hit if there is a
lot of plugins in the user space (7 atm in Docker Desktop):
`docker --version` takes in current 23.0.1 ~93ms
Instead of removing completion for plugins to fix the
regression, we can slightly improve plugins discovery by
spawning a goroutine for each iteration in the loop when
listing plugins:
`docker --version` now takes ~38ms
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Both the DockerCLI and Cobra Commands provide accessors for Input, Output,
and Error streams (usually STDIN, STDOUT, STDERR). While we were already
passing DockerCLI's Output to Cobra, we were not doing so for the other
streams (and were passing none for plugin commands), potentially resulting
in DockerCLI output/input to mean something else than a Cobra Command's
intput/output/error.
This patch sets them to the same streams when constructing the Cobra
command.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The test used `gopkg.in/yaml.v2` to verify the TextMarshaller implementation,
which was implemented to allow printing the errors in JSON formatted output;
> This exists primarily to implement encoding.TextMarshaller such that
> rendering a plugin as JSON (e.g. for `docker info -f '{{json .CLIPlugins}}'`)
> renders the Err field as a useful string and not just `{}`.
Given that both yaml.Marshal and json.Marshal use this, we may as well use
Go's stdlib.
While updating, also changed some of the assertions to checks, so that we don't
fail the test early.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Looks like the linter uses an explicit -lang, which (for go1.19)
results in some additional formatting for octal values.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions of Go do not format these comments, so we can already
reformat them ahead of time to prevent gofmt linting failing once
we update to Go 1.19 or up.
Result of:
gofmt -s -w $(find . -type f -name '*.go' | grep -v "/vendor/")
With some manual adjusting.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
On Windows, the os/exec.{Command,CommandContext,LookPath} functions
resolve command names that have neither path separators nor file extension
(e.g., "git") by first looking in the current working directory before
looking in the PATH environment variable.
Go maintainers intended to match cmd.exe's historical behavior.
However, this is pretty much never the intended behavior and as an abundance of precaution
this patch prevents that when executing commands.
Example of commands that docker.exe may execute: `git`, `docker-buildx` (or other cli plugin), `docker-credential-wincred`, `docker`.
Note that this was prompted by the [Go 1.15.7 security fixes](https://blog.golang.org/path-security), but unlike in `go.exe`,
the windows path lookups in docker are not in a code path allowing remote code execution, thus there is no security impact on docker.
Signed-off-by: Tibor Vass <tibor@docker.com>
The CLI disabled experimental features by default, requiring users
to set a configuration option to enable them.
Disabling experimental features was a request from Enterprise users
that did not want experimental features to be accessible.
We are changing this policy, and now enable experimental features
by default. Experimental features may still change and/or removed,
and will be highlighted in the documentation and "usage" output.
For example, the `docker manifest inspect --help` output now shows:
EXPERIMENTAL:
docker manifest inspect is an experimental feature.
Experimental features provide early access to product functionality. These features
may change between releases without warning or can be removed entirely from a future
release. Learn more about experimental features: https://docs.docker.com/go/experimental/
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The vanity domain is down, and the project has moved
to a new location.
vendor check started failing because of this:
Collecting initial packages
Download dependencies
unrecognized import path "vbom.ml/util" (https fetch: Get https://vbom.ml/util?go-get=1: dial tcp: lookup vbom.ml on 169.254.169.254:53: no such host)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Brian Goff
Tonis Tiigi
Jonathan A. Sternberg
Sebastiaan van Stijn
Kirill A. Korinsky
Laura Brehm
Sebastiaan van Stijn
Bjorn Neergaard
Kevin Alvarez
Nicolas De Loof
Tibor Vass
Silvin Lubecki