Commit Graph

8692 Commits

Author SHA1 Message Date
Sebastiaan van Stijn 5fd2c08502
scripts/docs/generate-md.sh: fix location of generate code
Commit 1e3622c50c moved the generator code
to a subdirectory, but forgot to update the markdown version of this script.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 37f234fbe7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-09 23:00:14 +02:00
Sebastiaan van Stijn 24d608d5f1
Merge pull request #4164 from thaJeztah/23.0_bump_go1.19.8
[23.0] update go to go1.19.8
2023-04-06 12:55:03 +02:00
Sebastiaan van Stijn 06ab5992dc
Merge pull request #4154 from thaJeztah/23.0_backport_fix_deprecation_message
[23.0 backport] cli/command: fix deprecation comments for Stream types
2023-04-05 17:01:59 +02:00
Sebastiaan van Stijn 5cd7710a04
update go to go1.19.8
go1.19.8 (released 2023-04-04) includes security fixes to the go/parser,
html/template, mime/multipart, net/http, and net/textproto packages, as well as
bug fixes to the linker, the runtime, and the time package. See the Go 1.19.8
milestone on our issue tracker for details:

https://github.com/golang/go/issues?q=milestone%3AGo1.19.8+label%3ACherryPickApproved

full diff: https://github.com/golang/go/compare/go1.19.7...go1.19.8

Further details from the announcement on the mailing list:

We have just released Go versions 1.20.3 and 1.19.8, minor point releases.
These minor releases include 4 security fixes following the security policy:

- go/parser: infinite loop in parsing

  Calling any of the Parse functions on Go source code which contains `//line`
  directives with very large line numbers can cause an infinite loop due to
  integer overflow.
  Thanks to Philippe Antoine (Catena cyber) for reporting this issue.
  This is CVE-2023-24537 and Go issue https://go.dev/issue/59180.

- html/template: backticks not treated as string delimiters

  Templates did not properly consider backticks (`) as Javascript string
  delimiters, and as such did not escape them as expected. Backticks are
  used, since ES6, for JS template literals. If a template contained a Go
  template action within a Javascript template literal, the contents of the
  action could be used to terminate the literal, injecting arbitrary Javascript
  code into the Go template.

  As ES6 template literals are rather complex, and themselves can do string
  interpolation, we've decided to simply disallow Go template actions from being
  used inside of them (e.g. "var a = {{.}}"), since there is no obviously safe
  way to allow this behavior. This takes the same approach as
  github.com/google/safehtml. Template.Parse will now return an Error when it
  encounters templates like this, with a currently unexported ErrorCode with a
  value of 12. This ErrorCode will be exported in the next major release.

  Users who rely on this behavior can re-enable it using the GODEBUG flag
  jstmpllitinterp=1, with the caveat that backticks will now be escaped. This
  should be used with caution.

  Thanks to Sohom Datta, Manipal Institute of Technology, for reporting this issue.

  This is CVE-2023-24538 and Go issue https://go.dev/issue/59234.

- net/http, net/textproto: denial of service from excessive memory allocation

  HTTP and MIME header parsing could allocate large amounts of memory, even when
  parsing small inputs.

  Certain unusual patterns of input data could cause the common function used to
  parse HTTP and MIME headers to allocate substantially more memory than
  required to hold the parsed headers. An attacker can exploit this behavior to
  cause an HTTP server to allocate large amounts of memory from a small request,
  potentially leading to memory exhaustion and a denial of service.
  Header parsing now correctly allocates only the memory required to hold parsed
  headers.

  Thanks to Jakob Ackermann (@das7pad) for discovering this issue.

  This is CVE-2023-24534 and Go issue https://go.dev/issue/58975.

- net/http, net/textproto, mime/multipart: denial of service from excessive resource consumption

  Multipart form parsing can consume large amounts of CPU and memory when
  processing form inputs containing very large numbers of parts. This stems from
  several causes:

  mime/multipart.Reader.ReadForm limits the total memory a parsed multipart form
  can consume. ReadForm could undercount the amount of memory consumed, leading
  it to accept larger inputs than intended. Limiting total memory does not
  account for increased pressure on the garbage collector from large numbers of
  small allocations in forms with many parts. ReadForm could allocate a large
  number of short-lived buffers, further increasing pressure on the garbage
  collector. The combination of these factors can permit an attacker to cause an
  program that parses multipart forms to consume large amounts of CPU and
  memory, potentially resulting in a denial of service. This affects programs
  that use mime/multipart.Reader.ReadForm, as well as form parsing in the
  net/http package with the Request methods FormFile, FormValue,
  ParseMultipartForm, and PostFormValue.

  ReadForm now does a better job of estimating the memory consumption of parsed
  forms, and performs many fewer short-lived allocations.

  In addition, mime/multipart.Reader now imposes the following limits on the
  size of parsed forms:

  Forms parsed with ReadForm may contain no more than 1000 parts. This limit may
  be adjusted with the environment variable GODEBUG=multipartmaxparts=. Form
  parts parsed with NextPart and NextRawPart may contain no more than 10,000
  header fields. In addition, forms parsed with ReadForm may contain no more
  than 10,000 header fields across all parts. This limit may be adjusted with
  the environment variable GODEBUG=multipartmaxheaders=.

  Thanks to Jakob Ackermann for discovering this issue.

  This is CVE-2023-24536 and Go issue https://go.dev/issue/59153.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-05 15:49:47 +02:00
Sebastiaan van Stijn 683b099613
Merge pull request #4157 from cpuguy83/23.0_improve_cp_progress
[23.0 backport] improve cp progress
2023-04-05 02:33:53 +02:00
Sebastiaan van Stijn 391b2f0fab
Merge pull request #4141 from crazy-max/23.0_backport_fix-perf-reg
[23.0 backport] improve and load plugin command stubs when required
2023-04-05 02:33:19 +02:00
Brian Goff 209e9c0c13 cp: Do not block transfer on writing to terminal
This moves all the terminal writing to a goroutine that updates the
terminal periodically.
In our MITM copier we just use an atomic to add to the total number of
bytes read/written, the goroutine reads the total and updates the
terminal as needed.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit eb392ff4ce)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-04-03 22:31:41 +00:00
Brian Goff ce68ce7ae8 cp: Make gocyclo happy
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit b9a1b0928a)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-04-03 22:31:34 +00:00
Brian Goff c28cb35c42 cp: Reduce number of progress updates
Only show progress updates after a time threshold has elapsed in order
to reduce the number of writes to the terminal.
This improves readability of the progress.

Also moves cursor show/hide into the progress printer to reduce chances
if messing up the user's terminal in case of cancellation.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 90b7bc36d4)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-04-03 22:31:23 +00:00
Brian Goff 948dfa91c9 cp: reduce branching in progress printer
This just makes it easier to reason about what is happening.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit efd011b793)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-04-03 22:31:13 +00:00
Brian Goff f813c9639f cp: Improve tty flashing on progress updates
- Instead of rewriting the entire line every time only clear and write
the parts that changed.
- Hide the cursor while writing progress

Both these things make the progress updates significantly easier to
read.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit ccae6e9299)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-04-03 22:31:06 +00:00
Brian Goff 97d91f13aa cp: do not emit progress if stderr is not a term
This fixes a case where a non-tty will have control characters + the log
line for every single read operation.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit f27927d934)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-04-03 22:30:56 +00:00
Sebastiaan van Stijn 6efc4eaccb
cli/command: fix deprecation comments for Stream types
These were deprecated in 6c400a9c2009bba9376ad61ab59c04c1ad675871 (docker 19.03),
but the "Deprecated:" comments were missing a newline before them.

While most IDEs will detect such comments as "deprecated", pkg.go.dev and linters
will ignore them, which may result in users not being aware of them being deprecated.

This patch;

- Fixes the "Deprecated:" comments.
- Changes the var aliases to functions, which is slightly more boilerplating,
  but makes sure the functions are documented as "function", instead of shown
  in the "variables" section on pkg.go.dev.
- Adds some punctuation and adds "doc links", which allows readers to navigate
  to related content on pkg.go.dev.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 817897f891)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-01 20:38:27 +02:00
Kevin Alvarez c7cdab58d2
improve plugins listing performance
We can slightly improve plugins listing by spawning a
goroutine for each iteration.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit 89583b92b7)

# Conflicts:
#	vendor.mod
#	vendor/modules.txt
2023-04-01 16:34:51 +02:00
Kevin Alvarez d93d78588d
load plugin command stubs when required
We are currently loading plugin command stubs for every
invocation which still has a significant performance hit.
With this change we are doing this operation only if cobra
completion arg request is found.

- 20.10.23: `docker --version` takes ~15ms
- 23.0.1: `docker --version` takes ~93ms

With this change `docker --version` takes ~9ms

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit c39c711a18)
2023-03-31 16:38:01 +02:00
Sebastiaan van Stijn 3e7cbfdee1
Merge pull request #4139 from thaJeztah/23.0_backport_fix_go_version
[23.0 backport] gha: align stray go 1.19.4 version
2023-03-31 16:06:26 +02:00
Sebastiaan van Stijn 8e38271f23
gha: align stray go 1.19.4 version
looks like this one was forgotten to be updated :)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e4436853e8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-30 16:56:58 +02:00
Sebastiaan van Stijn 569dd73db1
Merge pull request #4126 from thaJeztah/23.0_backport_align_go_ver
[23.0 backport] Dockerfile: align go version
2023-03-27 17:44:26 +02:00
CrazyMax f6643207a2
don't use null values in the bake definition
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit bec5d37e91)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-27 17:18:38 +02:00
CrazyMax f381e08425
Dockerfile: align go version
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit b854eff300)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-27 17:18:35 +02:00
Sebastiaan van Stijn 18f20a5537
Merge pull request #4124 from thaJeztah/23.0_e2e_fix_certs
[23.0 backport] e2e: update notary certificates
2023-03-27 17:18:13 +02:00
CrazyMax d3a36fc38c
e2e: update notary certificates
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit b201ce5efd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-27 15:26:23 +02:00
CrazyMax 59bb07f2e4
e2e: increase tests certificates duration (10 years)
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit c6c33380da)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-27 15:26:22 +02:00
CrazyMax 80f27987f4
bake target to generate certs for e2e tets
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
(cherry picked from commit d234a81de7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-27 15:26:20 +02:00
Sebastiaan van Stijn 6a8406e602
Merge pull request #4092 from crazy-max/23.0_backport_buildx-completion
[23.0 backport] Add bash completion for available plugins
2023-03-22 20:05:09 +01:00
Sebastiaan van Stijn c2c122fb65
Merge pull request #4107 from thaJeztah/23.0_backport_size_flag_ps
[23.0 backport] Don't automatically request size if `--size` was explicitly set to `false`
2023-03-21 17:54:09 +01:00
Sebastiaan van Stijn 40a48e4154
Merge pull request #4106 from thaJeztah/23.0_backport_fix_comments
[23.0 backport] cli/command: ElectAuthServer: fix deprecation comment
2023-03-21 17:52:39 +01:00
Laura Brehm a43c9f3440
Don't automatically request size if `--size` was explicitly set to `false`
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
(cherry picked from commit 9733334487)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-21 17:02:35 +01:00
Sebastiaan van Stijn 114e17ac4b
cli/command: fix imports formatting
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 742881fc58)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-21 16:58:47 +01:00
Sebastiaan van Stijn e2c402118c
cli/command: ElectAuthServer: fix deprecation comment
The comment was not formatted correctly, and because of that not picked up as
being deprecated.

updates b4ca1c7368

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e3fa7280ad)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-21 16:58:47 +01:00
CrazyMax d07453890c
Add bash completion for available plugins
Signed-off-by: CrazyMax <github@crazymax.dev>
(cherry picked from commit aa0aa4a6dc)
2023-03-17 15:06:45 +01:00
Sebastiaan van Stijn 288b6c79fe
Merge pull request #4083 from thaJeztah/23.0_backport_windows_drive_cwd_env
[23.0 backport] stack/loader: Ignore cmd.exe special env variables
2023-03-10 13:04:29 +01:00
Sebastiaan van Stijn fbab8cd2be
Merge pull request #4086 from thaJeztah/23.0_backport_bump_go1.19.7
[23.0 backport] update to go1.19.7
2023-03-10 13:04:03 +01:00
Sebastiaan van Stijn b898a46135
Merge pull request #4088 from thaJeztah/23.0_backport_update_buildx
[23.0 backport] Dockerfile: update buildx to v0.10.4
2023-03-10 12:54:16 +01:00
Sebastiaan van Stijn 90a72a5894
Dockerfile: update buildx to v0.10.4
release notes: https://github.com/docker/buildx/releases/tag/v0.10.4

full diff: https://github.com/docker/buildx/compare/v0.10.3...v0.10.4

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 74c4ed4171)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-10 12:33:03 +01:00
Sebastiaan van Stijn 4c63110a92
update to go1.19.7
Includes a security fix for crypto/elliptic (CVE-2023-24532).

> go1.19.7 (released 2023-03-07) includes a security fix to the crypto/elliptic
> package, as well as bug fixes to the linker, the runtime, and the crypto/x509
> and syscall packages. See the Go 1.19.7 milestone on our issue tracker for
> details.

https://go.dev/doc/devel/release#go1.19.minor

From the announcement:

> We have just released Go versions 1.20.2 and 1.19.7, minor point releases.
>
> These minor releases include 1 security fixes following the security policy:
>
> - crypto/elliptic: incorrect P-256 ScalarMult and ScalarBaseMult results
    >
    >   The ScalarMult and ScalarBaseMult methods of the P256 Curve may return an
    >   incorrect result if called with some specific unreduced scalars (a scalar larger
    >   than the order of the curve).
    >
    >   This does not impact usages of crypto/ecdsa or crypto/ecdh.
>
> This is CVE-2023-24532 and Go issue https://go.dev/issue/58647.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 23da1cec6c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-10 10:24:22 +01:00
Paweł Gronowski b61b5a9878
stack: Change unexpected environment variable error
Make the error more specific by stating that it's caused by a specific
environment variable and not an environment as a whole.
Also don't escape the variable to make it more readable.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 012b77952e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-09 22:13:20 +01:00
Paweł Gronowski 84fe451ec7
stack/loader: Ignore cmd.exe special env variables
On Windows, ignore all variables that start with "=" when building an
environment variables map for stack.
For MS-DOS compatibility cmd.exe can set some special environment
variables that start with a "=" characters, which breaks the general
assumption that the first encountered "=" separates a variable name from
variable value and causes trouble when parsing.

These variables don't seem to be documented anywhere, but they are
described by some third-party sources and confirmed empirically on my
Windows installation.

Useful sources:
https://devblogs.microsoft.com/oldnewthing/20100506-00/?p=14133
https://ss64.com/nt/syntax-variables.html

Known variables:

- `=ExitCode` stores the exit code returned by external command (in hex
  format)
- `=ExitCodeAscii` - same as above, except the value is the ASCII
  representation of the code (so exit code 65 (0x41) becomes 'A').
- `=::=::\` and friends - store drive specific working directory.
  There is one env variable for each separate drive letter that was
  accessed in the shell session and stores the working directory for that
  specific drive.
  The general format for these is:
    `=<DRIVE_LETTER>:=<CWD>`  (key=`=<DRIVE_LETTER>:`, value=`<CWD>`)
  where <CWD> is a working directory for the drive that is assigned to
  the letter <DRIVE_LETTER>

  A couple of examples:
    `=C:=C:\some\dir`  (key: `=C:`, value: `C:\some\dir`)
    `=D:=D:\some\other\dir`  (key: `=C:`, value: `C:\some\dir`)
    `=Z:=Z:\`  (key: `=Z:`, value: `Z:\`)

  `=::=::\` is the one that seems to be always set and I'm not exactly
  sure what this one is for (what's drive `::`?). Others are set as
  soon as you CD to a path on some drive. Considering that you start a
  cmd.exe also has some working directory, there are 2 of these on start.

All these variables can be safely ignored because they can't be
deliberately set by the user, their meaning is only relevant to the
cmd.exe session and they're all are related to the MS-DOS/Batch feature
that are irrelevant for us.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit a47058bbd5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-09 22:13:18 +01:00
Sebastiaan van Stijn 71615c2df1
Merge pull request #4077 from thaJeztah/23.0_update_buildx
[23.0 backport] Dockerfile: update buildx to v0.10.3
2023-03-09 12:08:43 +01:00
Paweł Gronowski a1acc9af91
Merge pull request #4076 from thaJeztah/23.0_backport_deprecate_buildinfo
[23.0 backport] docs: Deprecate buildkit's build information
2023-03-06 20:06:22 +01:00
Jacopo Rigoli 95066ff3a2
Dockerfile: update buildx to v0.10.3
release notes: https://github.com/docker/buildx/releases/tag/v0.10.3

Signed-off-by: Jacopo Rigoli <rigoli.jacopo@gmail.com>
(cherry picked from commit dac79b19a7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-06 18:28:30 +01:00
Paweł Gronowski 0dbf70fad2
docs: Deprecate buildkit's build information
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 8bc1aaceae)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-06 18:25:03 +01:00
Sebastiaan van Stijn e0b8e19687
Merge pull request #4035 from thaJeztah/23.0_backport_carry_4027
[23.0 backport] changed the container name in docker stats page
2023-03-03 16:47:47 +01:00
Sebastiaan van Stijn 98e874dac7
Merge pull request #4039 from thaJeztah/23.0_backport_bump_go_1.19.6
[23.0 backport] update to go1.19.6
2023-03-02 14:34:36 +01:00
Sebastiaan van Stijn 92164b0306
Merge pull request #4065 from vvoland/dangling-images-none-23
[23.0 backport] formatter: Consider empty RepoTags and RepoDigests as dangling
2023-03-02 14:31:31 +01:00
Paweł Gronowski 5af8077eeb
formatter: Consider empty RepoTags and RepoDigests as dangling
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 89687d5b3f)
2023-03-02 09:48:45 +01:00
Sebastiaan van Stijn d352c504a8
Merge pull request #4061 from vvoland/test-fakecli-images-mock-23
[23.0 backport] test/cli: Use empty array as empty output of images/json
2023-03-01 21:03:06 +01:00
Sebastiaan van Stijn 28c74b759b
Merge pull request #4063 from thaJeztah/23.0_backport_write_file
[23.0 backport] context: avoid corrupt file writes
2023-03-01 21:02:40 +01:00
Nick Santos 57a502772b
context: avoid corrupt file writes
Write to a tempfile then move, so that if the
process dies mid-write it doesn't corrupt the store.

Also improve error messaging so that if a file does
get corrupted, the user has some hope of figuring
out which file is broken.

For background, see:
https://github.com/docker/for-win/issues/13180
https://github.com/docker/for-win/issues/12561

For a repro case, see:
https://github.com/nicks/contextstore-sandbox

Signed-off-by: Nick Santos <nick.santos@docker.com>
(cherry picked from commit c2487c2997)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-03-01 16:14:03 +01:00
Paweł Gronowski 14ac8db968
test/cli: Use empty array as empty output of images/json
Tests mocking the output of GET images/json with fakeClient used an
array with one empty element as an empty response.
Change it to just an empty array.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit a1953e19b2)
2023-03-01 15:59:31 +01:00