go1.20.4 (released 2023-05-02) includes three security fixes to the html/template
package, as well as bug fixes to the compiler, the runtime, and the crypto/subtle,
crypto/tls, net/http, and syscall packages. See the Go 1.20.4 milestone on our
issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved
release notes: https://go.dev/doc/devel/release#go1.20.4
full diff: https://github.com/golang/go/compare/go1.20.3...go1.20.4
from the announcement:
> These minor releases include 3 security fixes following the security policy:
>
> - html/template: improper sanitization of CSS values
>
> Angle brackets (`<>`) were not considered dangerous characters when inserted
> into CSS contexts. Templates containing multiple actions separated by a '/'
> character could result in unexpectedly closing the CSS context and allowing
> for injection of unexpected HMTL, if executed with untrusted input.
>
> Thanks to Juho Nurminen of Mattermost for reporting this issue.
>
> This is CVE-2023-24539 and Go issue https://go.dev/issue/59720.
>
> - html/template: improper handling of JavaScript whitespace
>
> Not all valid JavaScript whitespace characters were considered to be
> whitespace. Templates containing whitespace characters outside of the character
> set "\t\n\f\r\u0020\u2028\u2029" in JavaScript contexts that also contain
> actions may not be properly sanitized during execution.
>
> Thanks to Juho Nurminen of Mattermost for reporting this issue.
>
> This is CVE-2023-24540 and Go issue https://go.dev/issue/59721.
>
> - html/template: improper handling of empty HTML attributes
>
> Templates containing actions in unquoted HTML attributes (e.g. "attr={{.}}")
> executed with empty input could result in output that would have unexpected
> results when parsed due to HTML normalization rules. This may allow injection
> of arbitrary attributes into tags.
>
> Thanks to Juho Nurminen of Mattermost for reporting this issue.
>
> This is CVE-2023-29400 and Go issue https://go.dev/issue/59722.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
notary server version 0.5.0 is linux/amd64 only.
Also, e2e stage from top level Dockerfile uses 0.6.1 notary version -
change the Dockerfiles in e2e/testdata to have the same version.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
This makes it possible to update the image loaded for e2e tests without
modifying all tests that use them.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
go1.20.3 (released 2023-04-04) includes security fixes to the go/parser,
html/template, mime/multipart, net/http, and net/textproto packages, as well
as bug fixes to the compiler, the linker, the runtime, and the time package.
See the Go 1.20.3 milestone on our issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved
full diff: https://github.com/golang/go/compare/go1.20.2...go1.20.3
Further details from the announcement on the mailing list:
We have just released Go versions 1.20.3 and 1.19.8, minor point releases.
These minor releases include 4 security fixes following the security policy:
- go/parser: infinite loop in parsing
Calling any of the Parse functions on Go source code which contains `//line`
directives with very large line numbers can cause an infinite loop due to
integer overflow.
Thanks to Philippe Antoine (Catena cyber) for reporting this issue.
This is CVE-2023-24537 and Go issue https://go.dev/issue/59180.
- html/template: backticks not treated as string delimiters
Templates did not properly consider backticks (`) as Javascript string
delimiters, and as such did not escape them as expected. Backticks are
used, since ES6, for JS template literals. If a template contained a Go
template action within a Javascript template literal, the contents of the
action could be used to terminate the literal, injecting arbitrary Javascript
code into the Go template.
As ES6 template literals are rather complex, and themselves can do string
interpolation, we've decided to simply disallow Go template actions from being
used inside of them (e.g. "var a = {{.}}"), since there is no obviously safe
way to allow this behavior. This takes the same approach as
github.com/google/safehtml. Template.Parse will now return an Error when it
encounters templates like this, with a currently unexported ErrorCode with a
value of 12. This ErrorCode will be exported in the next major release.
Users who rely on this behavior can re-enable it using the GODEBUG flag
jstmpllitinterp=1, with the caveat that backticks will now be escaped. This
should be used with caution.
Thanks to Sohom Datta, Manipal Institute of Technology, for reporting this issue.
This is CVE-2023-24538 and Go issue https://go.dev/issue/59234.
- net/http, net/textproto: denial of service from excessive memory allocation
HTTP and MIME header parsing could allocate large amounts of memory, even when
parsing small inputs.
Certain unusual patterns of input data could cause the common function used to
parse HTTP and MIME headers to allocate substantially more memory than
required to hold the parsed headers. An attacker can exploit this behavior to
cause an HTTP server to allocate large amounts of memory from a small request,
potentially leading to memory exhaustion and a denial of service.
Header parsing now correctly allocates only the memory required to hold parsed
headers.
Thanks to Jakob Ackermann (@das7pad) for discovering this issue.
This is CVE-2023-24534 and Go issue https://go.dev/issue/58975.
- net/http, net/textproto, mime/multipart: denial of service from excessive resource consumption
Multipart form parsing can consume large amounts of CPU and memory when
processing form inputs containing very large numbers of parts. This stems from
several causes:
mime/multipart.Reader.ReadForm limits the total memory a parsed multipart form
can consume. ReadForm could undercount the amount of memory consumed, leading
it to accept larger inputs than intended. Limiting total memory does not
account for increased pressure on the garbage collector from large numbers of
small allocations in forms with many parts. ReadForm could allocate a large
number of short-lived buffers, further increasing pressure on the garbage
collector. The combination of these factors can permit an attacker to cause an
program that parses multipart forms to consume large amounts of CPU and
memory, potentially resulting in a denial of service. This affects programs
that use mime/multipart.Reader.ReadForm, as well as form parsing in the
net/http package with the Request methods FormFile, FormValue,
ParseMultipartForm, and PostFormValue.
ReadForm now does a better job of estimating the memory consumption of parsed
forms, and performs many fewer short-lived allocations.
In addition, mime/multipart.Reader now imposes the following limits on the
size of parsed forms:
Forms parsed with ReadForm may contain no more than 1000 parts. This limit may
be adjusted with the environment variable GODEBUG=multipartmaxparts=. Form
parts parsed with NextPart and NextRawPart may contain no more than 10,000
header fields. In addition, forms parsed with ReadForm may contain no more
than 10,000 header fields across all parts. This limit may be adjusted with
the environment variable GODEBUG=multipartmaxheaders=.
Thanks to Jakob Ackermann for discovering this issue.
This is CVE-2023-24536 and Go issue https://go.dev/issue/59153.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
We are currently loading plugin command stubs for every
invocation which still has a significant performance hit.
With this change we are doing this operation only if cobra
completion arg request is found.
- 20.10.23: `docker --version` takes ~15ms
- 23.0.1: `docker --version` takes ~93ms
With this change `docker --version` takes ~9ms
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Before this change, the error would suggest installing buildx:
echo "FROM scratch" | DOCKER_BUILDKIT=0 docker build -
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
Install the buildx component to build images with BuildKit:
https://docs.docker.com/go/buildx/
...
However, this error would also be shown if buildx is actually installed,
but disabled through "DOCKER_BUILDKIT=0";
docker buildx version
github.com/docker/buildx v0.9.1 ed00243
With this patch, it reports that it's disabled, and how to fix:
echo "FROM scratch" | DOCKER_BUILDKIT=0 docker build -
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enabled it by removing the DOCKER_BUILDKIT=0
environment-variable.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This updates `docker context ls` to:
- not abort listing contexts when failing one (or more) contexts
- instead, adding an ERROR column to inform the user there was
an issue loading the context.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Looks like the linter uses an explicit -lang, which (for go1.19)
results in some additional formatting for octal values.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Also removing redundant defer for env.PatchAll(), which is now automatically
handled in t.Cleanup()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Picking 2 seconds, although that's just a randomly picked timeout;
given that this is only for testing, it's not too important.
e2e/plugin/basic/basic.go:25:12: G112: Potential Slowloris Attack because ReadHeaderTimeout is not configured in the http.Server (gosec)
server := http.Server{
Addr: l.Addr().String(),
Handler: http.NewServeMux(),
}
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions of Go do not format these comments, so we can already
reformat them ahead of time to prevent gofmt linting failing once
we update to Go 1.19 or up.
Result of:
gofmt -s -w $(find . -type f -name '*.go' | grep -v "/vendor/")
With some manual adjusting.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The default output for Cobra aliases only shows the subcommand as alias, which
is not very intuitive. This patch changes the output to print the full command
as it would be called by the user.
Note that there's still some improvements to be made; due to how aliases must be
set-up in Cobra, aliases at different "levels" are still not shown. So for example,
`docker ps --help` will not show `docker container ps` as alias, and vice-versa.
This will require additional changes, and can possibly be resolved using custom
metadata/annotations.
Before this patch:
docker container ls --help
Usage: docker container ls [OPTIONS]
List containers
Aliases:
ls, ps, list
After this patch:
docker container ls --help
Usage: docker container ls [OPTIONS]
List containers
Aliases:
docker container ls, docker container ps, docker container list
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Make use of existing modules and functions in order to output the merged configs.
Added skip interpolation flag of variables, so that you can pipe the output back to stack deploy without much hassle.
Signed-off-by: Stoica-Marcu Floris-Andrei <floris.sm@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- remove check for "A self-sufficient runtime for containers"; really
not important to check for.
- don't make the checks positional (just match that we find them, and
that we don't find them multiple times)
- account for leading whitespace to change instead of hard-coding the
number of spaces before output.
- change the badopt check; I think it should be sufficient to check
that the bad option was printed and that "run --help" output is
printed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Remove various tests and utilities related to testing kubernetes support
Also removing the Kubernetes and DefaultStackOrchestrator from CreateOptions
and UpdateOptions, instead updating the flags to not be bound to a variable.
This might break some consumers of those options, but given that they've become
non-functional, that's probably ok (otherwise they may ignore the deprecation
warning and end up with non-functional code).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
With this change:
echo 'FROM busybox' | DOCKER_BUILDKIT=1 docker build -
ERROR: BuildKit is enabled but the buildx component is missing or broken.
Install the buildx component to build images with BuildKit:
https://docs.docker.com/go/buildx/
echo 'FROM busybox' | docker build -
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
Install the buildx component to build images with BuildKit:
https://docs.docker.com/go/buildx/
Sending build context to Docker daemon 2.048kB
...
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Golang uses a `sync.Once` when determining the proxy to use. This means
that it's not possible to test the proxy configuration in unit tests,
because the proxy configuration will be "fixated" the first time Golang
detects the proxy configuration.
This patch changes TestNewAPIClientFromFlagsWithHttpProxyEnv to an e2e
test so that we can verify the CLI picks up the proxy configuration.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
We updated some of these functions to make sure os.environ was
preserved, but some where not.
This adds a utility to help with this, which also prevents the
os.environ to be added multiple times.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
All output of the usage / --help output uses spaces, and having a tab
in the output can be somewhat cumbersome (e.g. our YAML docs generator
doesn't like them, and copy/pasing the output in iTerm produces a warning).
This patch changes the output to use two spaces instead.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `ENV key value` form can be ambiguous, for example, the following defines
a single env-variable (`ONE`) with value `"TWO= THREE=world"`:
ENV ONE TWO= THREE=world
While we cannot deprecate/remove that syntax (as it would break existing
Dockerfiles), we should reduce exposure of the format in our examples.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The tabwriter was configured to have a min-width for columns of 20 positions.
This seemed quite wide, and caused smaller columns to be printed with a large
gap between.
Before:
docker container stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
29184b3ae391 amazing_shirley 0.00% 800KiB / 1.944GiB 0.04% 1.44kB / 0B 0B / 0B 1
403c101bad56 agitated_swartz 0.15% 34.31MiB / 1.944GiB 1.72% 10.2MB / 206kB 0B / 0B 51
0dc4b7f6c6be container2 0.00% 1.012MiB / 1.944GiB 0.05% 12.9kB / 0B 0B / 0B 5
2d99abcc6f62 container99 0.00% 972KiB / 1.944GiB 0.05% 13kB / 0B 0B / 0B 5
9f9aa90173ac foo 0.00% 820KiB / 1.944GiB 0.04% 13kB / 0B 0B / 0B 5
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29184b3ae391 docker-cli-dev "ash" 4 hours ago Up 4 hours amazing_shirley
403c101bad56 docker-dev:master "hack/dind bash" 3 days ago Up 3 days agitated_swartz
0dc4b7f6c6be nginx:alpine "/docker-entrypoint.…" 4 days ago Up 4 days 80/tcp container2
2d99abcc6f62 nginx:alpine "/docker-entrypoint.…" 4 days ago Up 4 days 80/tcp container99
9f9aa90173ac nginx:alpine "/docker-entrypoint.…" 4 days ago Up 4 days 80/tcp foo
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker-cli-dev latest 5f603caa04aa 4 hours ago 610MB
docker-cli-native latest 9dd29f8d387b 4 hours ago 519MB
docker-dev master 8132bf7a199e 3 days ago 2.02GB
docker-dev improve-build-errors 69e208994b3f 11 days ago 2.01GB
docker-dev refactor-idtools 69e208994b3f 11 days ago 2.01GB
After:
docker container stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
29184b3ae391 amazing_shirley 0.14% 5.703MiB / 1.944GiB 0.29% 1.44kB / 0B 0B / 0B 10
403c101bad56 agitated_swartz 0.15% 56.97MiB / 1.944GiB 2.86% 10.2MB / 206kB 0B / 0B 51
0dc4b7f6c6be container2 0.00% 1016KiB / 1.944GiB 0.05% 12.9kB / 0B 0B / 0B 5
2d99abcc6f62 container99 0.00% 956KiB / 1.944GiB 0.05% 13kB / 0B 0B / 0B 5
9f9aa90173ac foo 0.00% 980KiB / 1.944GiB 0.05% 13kB / 0B 0B / 0B 5
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29184b3ae391 docker-cli-dev "ash" 12 minutes ago Up 12 minutes amazing_shirley
403c101bad56 docker-dev:master "hack/dind bash" 3 days ago Up 3 days agitated_swartz
0dc4b7f6c6be nginx:alpine "/docker-entrypoint.…" 4 days ago Up 4 days 80/tcp container2
2d99abcc6f62 nginx:alpine "/docker-entrypoint.…" 4 days ago Up 4 days 80/tcp container99
9f9aa90173ac nginx:alpine "/docker-entrypoint.…" 4 days ago Up 4 days 80/tcp foo
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker-cli-dev latest 5f603caa04aa 4 hours ago 610MB
docker-cli-native latest 9dd29f8d387b 4 hours ago 519MB
docker-dev master 8132bf7a199e 3 days ago 2.02GB
docker-dev improve-build-errors 69e208994b3f 11 days ago 2.01GB
docker-dev refactor-idtools 69e208994b3f 11 days ago 2.01GB
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `docker push` command up until [v0.9.1](https://github.com/moby/moby/blob/v0.9.1/api/client.go#L998)
always pushed all tags of a given image, so `docker push foo/bar` would push (e.g.)
all of `foo/bar:latest`, `foo:/bar:v1`, `foo/bar:v1.0.0`.
Pushing all tags of an image was not desirable in many case, so docker v0.10.0
enhanced `docker push` to optionally specify a tag to push (`docker push foo/bar:v1`)
(see https://github.com/moby/moby/issues/3411 and the pull request that implemented
this: https://github.com/moby/moby/pull/4948).
This behavior exists up until today, and is confusing, because unlike other commands,
`docker push` does not default to use the `:latest` tag when omitted, but instead
makes it push "all tags of the image"
For example, in the following situation;
```
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
thajeztah/myimage latest b534869c81f0 41 hours ago 1.22MB
```
Running `docker push thajeztah/myimage` seemingly does the expected behavior (it
pushes `thajeztah/myimage:latest` to Docker Hub), however, it does not so for the
reason expected (`:latest` being the default tag), but because `:latest` happens
to be the only tag present for the `thajeztah/myimage` image.
If another tag exists for the image:
```
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
thajeztah/myimage latest b534869c81f0 41 hours ago 1.22MB
thajeztah/myimage v1.0.0 b534869c81f0 41 hours ago 1.22MB
```
Running the same command (`docker push thajeztah/myimage`) will push _both_ images
to Docker Hub.
> Note that the behavior described above is currently not (clearly) documented;
> the `docker push` reference documentation (https://docs.docker.com/engine/reference/commandline/push/)
does not mention that omitting the tag will push all tags
This patch changes the default behavior, and if no tag is specified, `:latest` is
assumed. To push _all_ tags, a new flag (`-a` / `--all-tags`) is added, similar
to the flag that's present on `docker pull`.
With this change:
- `docker push myname/myimage` will be the equivalent of `docker push myname/myimage:latest`
- to push all images, the user needs to set a flag (`--all-tags`), so `docker push --all-tags myname/myimage:latest`
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this patch:
docker push --quiet nosuchimage
docker.io/library/nosuchimage
echo $?
0
With this patch applied:
docker push --quiet nosuchimage:latest
An image does not exist locally with the tag: nosuchimage
echo $?
1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Comments should have a leading space unless the comment is
for special purposes (go:generate, nolint:)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The top-level `docker deploy` command (using the "Docker Application Bundle"
(`.dab`) file format was introduced as an experimental feature in Docker 1.13 /
17.03, but superseded by support for Docker Compose files.
With no development being done on this feature, and no active use of the file
format, support for the DAB file format and the top-level `docker deploy` command
(hidden by default in 19.03), is removed in this patch, in favour of `docker stack deploy`
using compose files.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```
e2e/cli-plugins/flags_test.go:135:27: Using the variable on range scope `args` in function literal (scopelint)
res := icmd.RunCmd(run(args...))
^
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
exec.CombinedOutput should not be used here because:
- it redirects cmd Stdout and Stderr and we want it to be the tty
- it calls cmd.Run which we already did
While at it
- use pty.Start() as it is cleaner
- make sure we don't leave a zombie running, by calling Wait() in defer
- use test.Name() for containerName
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The docker-in-docker image now enables TLS by default (added in
docker-library/docker#166), which complicates testing in our
environment, and isn't needed for the tests we're running.
This patch sets the `DOCKER_TLS_CERTDIR` to an empty value to
disable TLS.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a test to verify that killing the docker CLI forwards
the signal to the container. Test-case for moby/moby 28872
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I'm about to refactor the code which includes the Kubernetes support in a way
which relies on something vendoring `./cli/context/kubernetes/` in order to
trigger the inclusion of support for the Kubernetes endpoint in the final
binary.
In practice anything which is interested in Kubernetes must import that package
(e.g. `./cli/command/context.list` does for the `EndpointFromContext`
function). However if it was somehow possible to build without that import then
the `KUBERNETES ENDPOINT` column would be mysteriously empty.
Out of an abundance of caution add a specific check on the final binary.
Signed-off-by: Ian Campbell <ijc@docker.com>
This regressed in 3af168c7df ("Ensure plugins can use `PersistentPreRunE`
again.") but this wasn't noticed because the helloworld test plugin has it's
own `PersistentPreRunE` which has the effect of deferring the resolution of the
global variable. In the case where the hook isn't used the variable is resolved
during `newPluginCommand` which is before the global variable was set.
Initialize the plugin command with a stub function wrapping the call to the
(global) hook, this defers resolving the variable until after it has been set,
otherwise the initial value (`nil`) is used in the struct.
Signed-off-by: Ian Campbell <ijc@docker.com>
This was omitted when these tests were added.
Adding this means that the tests now see the `$DOCKER_HOST` configured (via
`$TEST_DOCKER_HOST`) where they didn't before. In some cases (specifically the
`test-e2e-connhelper-ssh` case) this results in additional output on stderr
when `-D` (debug) is used:
time="2019-04-03T11:10:27Z" level=debug msg="commandconn: starting ssh with [-l penguin 172.20.0.2 -- docker system dial-stdio]"
Address this by switching the affected test cases to use `-l info` instead of
`-D`, they all just require some option not specifically `-D`. Note that `info`
is the default log level so this is effectively a nop (which is good).
Signed-off-by: Ian Campbell <ijc@docker.com>
In the initial implementation I thought it would be good to not pass on the
deprecation to plugins (since they are new). However it turns out this causes
`docker helloworld -h` to print a spurious "pflag: help requested" line:
$ docker helloworld -h
pflag: help requested
See 'docker helloworld --help'.
Usage: docker helloworld [OPTIONS] COMMAND
A basic Hello World plugin for tests
...
Compared with:
$ docker ps -h
Flag shorthand -h has been deprecated, please use --help
Usage: docker ps [OPTIONS]
This is in essence because having the flag undefined hits a different path
within cobra, causing `c.execute()` to return early due to getting an error
(`flag.ErrHelp`) from `c.ParseFlags`, which launders the error through our
`FlagErrorFunc` which wraps it in a `StatusError` which in turn defeats an `if
err == flag.ErrHelp` check further up the call chain. If the flag is defined we
instead hit a path which returns a bare `flag.ErrHelp` without wrapping it.
I considered updating our `FlagErrorFunc` to not wrap `flag.ErrHelp` (and then
following the chain to the next thing) however while doing that I realised that
the code for `-h` (and `--help`) is deeply embedded into cobra (and its flags
library) such that actually using `-h` as a plugin argument meaning something
other than `help` is basically impossible/impractical. Therefore we may as well
have plugins behave identically to the monolithic CLI and support (deprecated)
the `-h` argument.
With this changed the help related blocks of `SetupRootCommand` and
`SetupPluginRootCommand` are now identical, so consolidate into
`setupCommonRootCommand`.
Tests are updated to check `-h` in a variety of scenarios, including the happy
case here.
Signed-off-by: Ian Campbell <ijc@docker.com>
In some cases this means switching to `icmd.Expected` rather than
`icmd.Success`, but this improves readability overall.
Signed-off-by: Ian Campbell <ijc@docker.com>
Since #1654 so far we've had problems with it not working on Windows (npipe
lacked the `CloseRead` method) and problems with using tcp with tls (the tls
connection also lacks `CloseRead`). Both of these were workedaround in #1718
which added a nop `CloseRead` method.
However I am now seeing hangs (on Windows) where the `system dial-stdio`
subprocess is not exiting (I'm unsure why so far).
I think the 3rd problem found with this is an indication that `dial-stdio` is
not quite ready for wider use outside of its initial usecase (support for
`ssh://` URLs to connect to remote daemons).
This change simply disables the `dial-stdio` path for all plugins. However
rather than completely reverting 891b3d953e ("cli-plugins: use `docker system
dial-stdio` to call the daemon") I've just disabled the functionality at the
point of use and left in a trap door environment variable so that those who
want to experiment with this mode (and perhaps fully debug it) have an easier
path do doing so.
The e2e test for this case is disabled unless the trap door envvar is set. I
also renamed the test to clarify that it is about cli plugins.
Signed-off-by: Ian Campbell <ijc@docker.com>
I regressed this in d4ced2ef77 ("allow plugins to have argument which match a
top-level flag.") by unconditionally overwriting any `PersistentRunE` that the
user may have supplied.
We need to ensure two things:
1. That the user can use `PersistentRunE` (or `PersistentRun`) for their own
purposes.
2. That our initialisation always runs, even if the user has used
`PersistentRun*`, since that will shadow the root.
To do this add a `PersistentRunE` to the helloworld plugin which logs (covers 1
above) and then use it when calling the `apiversion` subcommand (which covers 2
since that uses the client)
Signed-off-by: Ian Campbell <ijc@docker.com>
Previous commits fixed the first issue on #1661, this simply adds a test for
it. Note that this is testing the current behaviour, without regard for the
second issue in #1661 which proposes a different behaviour.
Signed-off-by: Ian Campbell <ijc@docker.com>
The issue with plugin options clashing with globals is that when cobra is
parsing the command line and it comes across an argument which doesn't start
with a `-` it (in the absence of plugins) distinguishes between "argument to
current command" and "new subcommand" based on the list of registered sub
commands.
Plugins breaks that model. When presented with `docker -D plugin -c foo` cobra
parses up to the `plugin`, sees it isn't a registered sub-command of the
top-level docker (because it isn't, it's a plugin) so it accumulates it as an
argument to the top-level `docker` command. Then it sees the `-c`, and thinks
it is the global `-c` (for AKA `--context`) option and tries to treat it as
that, which fails.
In the specific case of the top-level `docker` subcommand we know that it has
no arguments which aren't `--flags` (or `-f` short flags) and so anything which
doesn't start with a `-` must either be a (known) subcommand or an attempt to
execute a plugin.
We could simply scan for and register all installed plugins at start of day, so
that cobra can do the right thing, but we want to avoid that since it would
involve executing each plugin to fetch the metadata, even if the command wasn't
going to end up hitting a plugin.
Instead we can parse the initial set of global arguments separately before
hitting the main cobra `Execute` path, which works here exactly because we know
that the top-level has no non-flag arguments.
One slight wrinkle is that the top-level `PersistentPreRunE` is no longer
called on the plugins path (since it no longer goes via `Execute`), so we
arrange for the initialisation done there (which has to be done after global
flags are parsed to handle e.g. `--config`) to happen explictly after the
global flags are parsed. Rather than make `newDockerCommand` return the
complicated set of results needed to make this happen, instead return a closure
which achieves this.
The new functionality is introduced via a common `TopLevelCommand` abstraction
which lets us adjust the plugin entrypoint to use the same strategy for parsing
the global arguments. This isn't strictly required (in this case the stuff in
cobra's `Execute` works fine) but doing it this way avoids the possibility of
subtle differences in behaviour.
Fixes#1699, and also, as a side-effect, the first item in #1661.
Signed-off-by: Ian Campbell <ijc@docker.com>
These won't pass right now due to https://github.com/docker/cli/issues/1699
("Plugins can't re-use the same flags as cli global flags") and the change in
935d47bbe9 ("Ignore unknown arguments on the top-level command."), but the
intention is to fix them now.
Signed-off-by: Ian Campbell <ijc@docker.com>
... in preference to `chsh`, since in recent alpine 3.9.2 images that can fail
with:
Password: chsh: PAM: Authentication token manipulation error
Which seems to relate to the use of `!` as the password for `root` in `/etc/shadow`gq
Signed-off-by: Ian Campbell <ijc@docker.com>
By default, exec uses the environment of the current process, however,
if `exec.Env` is not `nil`, the environment is discarded:
e73f489494/src/os/exec/exec.go (L57-L60)
> If Env is nil, the new process uses the current process's environment.
When adding a new environment variable, prepend the current environment,
to make sure it is not discarded.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Plugins are expected to be management commands ("docker <object> <verb>").
This patch modified the usage output to shown plugins in the "Management commands"
section.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- The placement of the vendor is now in the end of the line.
- A '*' is now added as suffix of plugins' top level commands.
Signed-off-by: Ulysses Souza <ulysses.souza@docker.com>
This means that plugins can use whatever methods the monolithic CLI supports,
which is good for consistency.
This relies on `os.Args[0]` being something which can be executed again to
reach the same binary, since it is propagated (via an envvar) to the plugin for
this purpose. This essentially requires that the current working directory and
path are not modified by the monolithic CLI before it launches the plugin nor
by the plugin before it initializes the client. This should be the case.
Previously the fake apiclient used by `TestExperimentalCLI` was not being used,
since `cli.Initialize` was unconditionally overwriting it with a real one
(talking to a real daemon during unit testing, it seems). This wasn't expected
nor desirable and no longer happens with the new arrangements, exposing the
fact that no `pingFunc` is provided, leading to a panic. Add a `pingFunc` to
the fake client to avoid this.
Signed-off-by: Ian Campbell <ijc@docker.com>
They were listed twice in `docker --help` (but not `docker help`), since the
stubs were added in both `tryRunPluginHelp` and the `setHelpFunc` closure.
Calling `AddPluginStubCommands` earlier in `setHelpFunc` before the call to
`tryRunPluginHelp` is sufficient. Also it is no longer necessary to add just
valid plugins (`tryRunPluginHelp` handles invalid plugins correctly) so remove
that logic (which was in any case broken for e.g. `docker --help`).
Update the e2e test to check for duplicate entries and also to test `docker
--help` which was previously missed.
Signed-off-by: Ian Campbell <ijc@docker.com>
This allows passing argument to plugins, otherwise they are caught by the parse
loop, since cobra does not know about each plugin at this stage (to avoid
having to always scan for all plugins) this means that e.g. `docker plugin
--foo` would accumulate `plugin` as an arg to the `docker` command, then choke
on the unknown `--foo`.
This allows unknown global args only, unknown arguments on subcommands (e.g.
`docker ps --foo`) are still correctly caught.
Add an e2e test covering this case.
Signed-off-by: Ian Campbell <ijc@docker.com>
Previously a plugin which used these hooks would overwrite the top-level plugin
command's use of this hook, resulting in the dockerCli object not being fully
initialised.
Provide a function which plugins can use to chain to the required behaviour.
This required some fairly ugly arrangements to preserve state (which was
previously in-scope in `newPluginCOmmand`) to be used by the new function.
Signed-off-by: Ian Campbell <ijc@docker.com>
To achieve this we hook in at the beginning of our custom `HelpFunc` and detect
the plugin case by adding stub commands.
Signed-off-by: Ian Campbell <ijc@docker.com>
To do this we add a stub `cobra.Command` for each installed plugin (only when
invoking `help`, not for normal running).
This requires a function to list all available plugins so that is added here.
Signed-off-by: Ian Campbell <ijc@docker.com>
Sebastiaan van Stijn
Paweł Gronowski
Sebastiaan van Stijn
Kevin Alvarez
Nicolas De Loof
Stoica-Marcu Floris-Andrei
Chris Crone
Rob Gulewich
glefloch
Tibor Vass
Silvin Lubecki
Kir Kolyshkin
Ian Campbell
Silvin Lubecki
Jean-Christophe Sirot
Akihiro Suda
Corey Quon
Ulysses Souza