From ffed6f5ff9eec40c4c3da0c5455d79ccdbab1aa8 Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Thu, 24 Aug 2017 14:37:35 -0400 Subject: [PATCH] Add TESTING document. Signed-off-by: Daniel Nephin --- TESTING.md | 87 ++++++++++++++++++++++++++++++++++++++++++++ internal/test/doc.go | 2 +- 2 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 TESTING.md diff --git a/TESTING.md b/TESTING.md new file mode 100644 index 0000000000..63bba1d3dc --- /dev/null +++ b/TESTING.md @@ -0,0 +1,87 @@ +# Testing + +The following guidelines summarize the testing policy for docker/cli. + +## Unit Test Suite + +All code changes should have unit test coverage. + +Error cases should be tested with unit tests. + +Bug fixes should be covered by new unit tests or additional assertions in +existing unit tests. + +### Details + +The unit test suite follows the standard Go testing convention. Tests are +located in the package directory in `_test.go` files. + +Unit tests should be named using the convention: + +``` +Test +``` + +[Table tests](https://github.com/golang/go/wiki/TableDrivenTests) should be used +where appropriate, but may not be appropriate in all cases. + +Assertions should be made using +[testify/assert](https://godoc.org/github.com/stretchr/testify/assert) and test +requirements should be verified using +[testify/require](https://godoc.org/github.com/stretchr/testify/require). + +Fakes, and testing utilities can be found in +[internal/test](https://godoc.org/github.com/docker/cli/internal/test) and +[gotestyourself](https://godoc.org/github.com/gotestyourself/gotestyourself). + +## End-to-End Test Suite + +The end-to-end test suite tests a cli binary against a real API backend. + +### Guidelines + +Each feature (subcommand) should have a single end-to-end test for +the success case. The test should include all (or most) flags/options supported +by that feature. + +In some rare cases a couple additional end-to-end tests may be written for a +sufficiently complex and critical feature (ex: `container run`, `service +create`, `service update`, and `docker build` may have ~3-5 cases each). + +In some rare cases a sufficiently critical error paths may have a single +end-to-end test case. + +In all other cases the behaviour should be covered by unit tests. + +If a code change adds a new flag, that flag should be added to the existing +"success case" end-to-end test. + +If a code change fixes a bug, that bug fix should be covered either by adding +assertions to the existing end-to-end test, or with one or more unit test. + +### Details + +The end-to-end test suite is located in +[./e2e](https://github.com/docker/cli/tree/master/e2e). Each directory in `e2e` +corresponds to a directory in `cli/command` and contains the tests for that +subcommand. Files in each directory should be named `_test.go` where +command is the basename of the command (ex: the test for `docker stack deploy` +is found in `e2e/stack/deploy_test.go`). + +Tests should be named using the convention: + +``` +Test[] +``` + +where the test case name is only required when there are multiple test cases for +a single command. + +End-to-end test should run the `docker` binary using +[gotestyourself/icmd](https://godoc.org/github.com/gotestyourself/gotestyourself/icmd) +and make assertions about the exit code, stdout, stderr, and local file system. + +Any Docker image or registry operations should use `registry:5000/` +to communicate with the local instance of the Docker registry. To load +additional fixture images to the registry see +[scripts/test/e2e/run](https://github.com/docker/cli/blob/master/scripts/test/e2e/run). diff --git a/internal/test/doc.go b/internal/test/doc.go index 3a8609716f..342441d533 100644 --- a/internal/test/doc.go +++ b/internal/test/doc.go @@ -1,5 +1,5 @@ // Package test is a test-only package that can be used by other cli package to write unit test. // -// It as an internal package and cannot be used outside of github.com/docker/cli/cli package. +// It as an internal package and cannot be used outside of github.com/docker/cli package. // package test