DockerCLI/cli/compose/loader/loader.go

989 lines
28 KiB
Go
Raw Normal View History

add //go:build directives to prevent downgrading to go1.16 language This is a follow-up to 0e73168b7e6d1d029d76d05b843b1aaec46739a8 This repository is not yet a module (i.e., does not have a `go.mod`). This is not problematic when building the code in GOPATH or "vendor" mode, but when using the code as a module-dependency (in module-mode), different semantics are applied since Go1.21, which switches Go _language versions_ on a per-module, per-package, or even per-file base. A condensed summary of that logic [is as follows][1]: - For modules that have a go.mod containing a go version directive; that version is considered a minimum _required_ version (starting with the go1.19.13 and go1.20.8 patch releases: before those, it was only a recommendation). - For dependencies that don't have a go.mod (not a module), go language version go1.16 is assumed. - Likewise, for modules that have a go.mod, but the file does not have a go version directive, go language version go1.16 is assumed. - If a go.work file is present, but does not have a go version directive, language version go1.17 is assumed. When switching language versions, Go _downgrades_ the language version, which means that language features (such as generics, and `any`) are not available, and compilation fails. For example: # github.com/docker/cli/cli/context/store /go/pkg/mod/github.com/docker/cli@v25.0.0-beta.2+incompatible/cli/context/store/storeconfig.go:6:24: predeclared any requires go1.18 or later (-lang was set to go1.16; check go.mod) /go/pkg/mod/github.com/docker/cli@v25.0.0-beta.2+incompatible/cli/context/store/store.go:74:12: predeclared any requires go1.18 or later (-lang was set to go1.16; check go.mod) Note that these fallbacks are per-module, per-package, and can even be per-file, so _(indirect) dependencies_ can still use modern language features, as long as their respective go.mod has a version specified. Unfortunately, these failures do not occur when building locally (using vendor / GOPATH mode), but will affect consumers of the module. Obviously, this situation is not ideal, and the ultimate solution is to move to go modules (add a go.mod), but this comes with a non-insignificant risk in other areas (due to our complex dependency tree). We can revert to using go1.16 language features only, but this may be limiting, and may still be problematic when (e.g.) matching signatures of dependencies. There is an escape hatch: adding a `//go:build` directive to files that make use of go language features. From the [go toolchain docs][2]: > The go line for each module sets the language version the compiler enforces > when compiling packages in that module. The language version can be changed > on a per-file basis by using a build constraint. > > For example, a module containing code that uses the Go 1.21 language version > should have a `go.mod` file with a go line such as `go 1.21` or `go 1.21.3`. > If a specific source file should be compiled only when using a newer Go > toolchain, adding `//go:build go1.22` to that source file both ensures that > only Go 1.22 and newer toolchains will compile the file and also changes > the language version in that file to Go 1.22. This patch adds `//go:build` directives to those files using recent additions to the language. It's currently using go1.19 as version to match the version in our "vendor.mod", but we can consider being more permissive ("any" requires go1.18 or up), or more "optimistic" (force go1.21, which is the version we currently use to build). For completeness sake, note that any file _without_ a `//go:build` directive will continue to use go1.16 language version when used as a module. [1]: https://github.com/golang/go/blob/58c28ba286dd0e98fe4cca80f5d64bbcb824a685/src/cmd/go/internal/gover/version.go#L9-L56 [2]; https://go.dev/doc/toolchain#:~:text=The%20go%20line%20for,file%20to%20Go%201.22 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-12-14 07:51:57 -05:00
// FIXME(thaJeztah): remove once we are a module; the go:build directive prevents go from downgrading language version to go1.16:
//go:build go1.22
add //go:build directives to prevent downgrading to go1.16 language This is a follow-up to 0e73168b7e6d1d029d76d05b843b1aaec46739a8 This repository is not yet a module (i.e., does not have a `go.mod`). This is not problematic when building the code in GOPATH or "vendor" mode, but when using the code as a module-dependency (in module-mode), different semantics are applied since Go1.21, which switches Go _language versions_ on a per-module, per-package, or even per-file base. A condensed summary of that logic [is as follows][1]: - For modules that have a go.mod containing a go version directive; that version is considered a minimum _required_ version (starting with the go1.19.13 and go1.20.8 patch releases: before those, it was only a recommendation). - For dependencies that don't have a go.mod (not a module), go language version go1.16 is assumed. - Likewise, for modules that have a go.mod, but the file does not have a go version directive, go language version go1.16 is assumed. - If a go.work file is present, but does not have a go version directive, language version go1.17 is assumed. When switching language versions, Go _downgrades_ the language version, which means that language features (such as generics, and `any`) are not available, and compilation fails. For example: # github.com/docker/cli/cli/context/store /go/pkg/mod/github.com/docker/cli@v25.0.0-beta.2+incompatible/cli/context/store/storeconfig.go:6:24: predeclared any requires go1.18 or later (-lang was set to go1.16; check go.mod) /go/pkg/mod/github.com/docker/cli@v25.0.0-beta.2+incompatible/cli/context/store/store.go:74:12: predeclared any requires go1.18 or later (-lang was set to go1.16; check go.mod) Note that these fallbacks are per-module, per-package, and can even be per-file, so _(indirect) dependencies_ can still use modern language features, as long as their respective go.mod has a version specified. Unfortunately, these failures do not occur when building locally (using vendor / GOPATH mode), but will affect consumers of the module. Obviously, this situation is not ideal, and the ultimate solution is to move to go modules (add a go.mod), but this comes with a non-insignificant risk in other areas (due to our complex dependency tree). We can revert to using go1.16 language features only, but this may be limiting, and may still be problematic when (e.g.) matching signatures of dependencies. There is an escape hatch: adding a `//go:build` directive to files that make use of go language features. From the [go toolchain docs][2]: > The go line for each module sets the language version the compiler enforces > when compiling packages in that module. The language version can be changed > on a per-file basis by using a build constraint. > > For example, a module containing code that uses the Go 1.21 language version > should have a `go.mod` file with a go line such as `go 1.21` or `go 1.21.3`. > If a specific source file should be compiled only when using a newer Go > toolchain, adding `//go:build go1.22` to that source file both ensures that > only Go 1.22 and newer toolchains will compile the file and also changes > the language version in that file to Go 1.22. This patch adds `//go:build` directives to those files using recent additions to the language. It's currently using go1.19 as version to match the version in our "vendor.mod", but we can consider being more permissive ("any" requires go1.18 or up), or more "optimistic" (force go1.21, which is the version we currently use to build). For completeness sake, note that any file _without_ a `//go:build` directive will continue to use go1.16 language version when used as a module. [1]: https://github.com/golang/go/blob/58c28ba286dd0e98fe4cca80f5d64bbcb824a685/src/cmd/go/internal/gover/version.go#L9-L56 [2]; https://go.dev/doc/toolchain#:~:text=The%20go%20line%20for,file%20to%20Go%201.22 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-12-14 07:51:57 -05:00
package loader
import (
"fmt"
"path"
"path/filepath"
"reflect"
"sort"
golangci-lint: enable perfsprint linter cli/compose/types/types.go:568:17: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return []byte(fmt.Sprintf("%v", e.External)), nil ^ cli/command/formatter/buildcache.go:174:9: fmt.Sprintf can be replaced with faster strconv.Itoa (perfsprint) return fmt.Sprintf("%d", c.v.UsageCount) ^ cli/command/formatter/buildcache.go:178:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%t", c.v.InUse) ^ cli/command/formatter/buildcache.go:182:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%t", c.v.Shared) ^ cli/command/formatter/image.go:259:9: fmt.Sprintf can be replaced with faster strconv.FormatInt (perfsprint) return fmt.Sprintf("%d", c.i.Containers) ^ cli/command/formatter/tabwriter/tabwriter_test.go:698:9: fmt.Sprintf can be replaced with faster strconv.Itoa (perfsprint) b.Run(fmt.Sprintf("%d", x), func(b *testing.B) { ^ cli/command/formatter/tabwriter/tabwriter_test.go:720:9: fmt.Sprintf can be replaced with faster strconv.Itoa (perfsprint) b.Run(fmt.Sprintf("%d", h), func(b *testing.B) { ^ cli/command/image/prune.go:62:31: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) pruneFilters.Add("dangling", fmt.Sprintf("%v", !options.all)) ^ cli/command/network/formatter.go:92:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%v", c.n.EnableIPv6) ^ cli/command/network/formatter.go:96:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%v", c.n.Internal) ^ cli/command/service/formatter.go:745:9: fmt.Sprintf can be replaced with faster strconv.FormatUint (perfsprint) pub = fmt.Sprintf("%d", pr.pStart) ^ cli/command/service/formatter.go:750:9: fmt.Sprintf can be replaced with faster strconv.FormatUint (perfsprint) tgt = fmt.Sprintf("%d", pr.tStart) ^ cli/command/service/opts.go:49:10: fmt.Sprintf can be replaced with faster strconv.FormatUint (perfsprint) return fmt.Sprintf("%v", *i.value) ^ cli/compose/loader/loader.go:720:36: fmt.Sprint can be replaced with faster strconv.Itoa (perfsprint) v, err := toServicePortConfigs(fmt.Sprint(value)) ^ Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-11-20 10:18:19 -05:00
"strconv"
"strings"
"time"
interp "github.com/docker/cli/cli/compose/interpolation"
"github.com/docker/cli/cli/compose/schema"
"github.com/docker/cli/cli/compose/template"
"github.com/docker/cli/cli/compose/types"
"github.com/docker/cli/opts"
"github.com/docker/docker/api/types/versions"
"github.com/docker/go-connections/nat"
units "github.com/docker/go-units"
"github.com/go-viper/mapstructure/v2"
"github.com/google/shlex"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
yaml "gopkg.in/yaml.v2"
)
// Options supported by Load
type Options struct {
// Skip schema validation
SkipValidation bool
// Skip interpolation
SkipInterpolation bool
// Interpolation options
Interpolate *interp.Options
// Discard 'env_file' entries after resolving to 'environment' section
discardEnvFiles bool
}
// WithDiscardEnvFiles sets the Options to discard the `env_file` section after resolving to
// the `environment` section
func WithDiscardEnvFiles(options *Options) {
options.discardEnvFiles = true
}
// ParseYAML reads the bytes from a file, parses the bytes into a mapping
// structure, and returns it.
func ParseYAML(source []byte) (map[string]any, error) {
var cfg any
if err := yaml.Unmarshal(source, &cfg); err != nil {
return nil, err
}
cfgMap, ok := cfg.(map[any]any)
if !ok {
return nil, errors.Errorf("top-level object must be a mapping")
}
converted, err := convertToStringKeysRecursive(cfgMap, "")
if err != nil {
return nil, err
}
return converted.(map[string]any), nil
}
// Load reads a ConfigDetails and returns a fully loaded configuration
func Load(configDetails types.ConfigDetails, opt ...func(*Options)) (*types.Config, error) {
if len(configDetails.ConfigFiles) < 1 {
return nil, errors.Errorf("No files specified")
}
options := &Options{
Interpolate: &interp.Options{
Substitute: template.Substitute,
LookupValue: configDetails.LookupEnv,
TypeCastMapping: interpolateTypeCastMapping,
},
}
for _, op := range opt {
op(options)
}
configs := []*types.Config{}
var err error
for _, file := range configDetails.ConfigFiles {
configDict := file.Config
version := schema.Version(configDict)
if configDetails.Version == "" {
configDetails.Version = version
}
if configDetails.Version != version {
return nil, errors.Errorf("version mismatched between two composefiles : %v and %v", configDetails.Version, version)
}
if err := validateForbidden(configDict); err != nil {
return nil, err
}
if !options.SkipInterpolation {
configDict, err = interpolateConfig(configDict, *options.Interpolate)
if err != nil {
return nil, err
}
}
if !options.SkipValidation {
if err := schema.Validate(configDict, configDetails.Version); err != nil {
return nil, err
}
}
cfg, err := loadSections(configDict, configDetails)
if err != nil {
return nil, err
}
cfg.Filename = file.Filename
if options.discardEnvFiles {
for i := range cfg.Services {
cfg.Services[i].EnvFile = nil
}
}
configs = append(configs, cfg)
}
return merge(configs)
}
func validateForbidden(configDict map[string]any) error {
servicesDict, ok := configDict["services"].(map[string]any)
if !ok {
return nil
}
forbidden := getProperties(servicesDict, types.ForbiddenProperties)
if len(forbidden) > 0 {
return &ForbiddenPropertiesError{Properties: forbidden}
}
return nil
}
func loadSections(config map[string]any, configDetails types.ConfigDetails) (*types.Config, error) {
var err error
cfg := types.Config{
Version: schema.Version(config),
}
loaders := []struct {
key string
fnc func(config map[string]any) error
}{
{
key: "services",
fnc: func(config map[string]any) error {
cfg.Services, err = LoadServices(config, configDetails.WorkingDir, configDetails.LookupEnv)
return err
},
},
{
key: "networks",
fnc: func(config map[string]any) error {
cfg.Networks, err = LoadNetworks(config, configDetails.Version)
return err
},
},
{
key: "volumes",
fnc: func(config map[string]any) error {
cfg.Volumes, err = LoadVolumes(config, configDetails.Version)
return err
},
},
{
key: "secrets",
fnc: func(config map[string]any) error {
cfg.Secrets, err = LoadSecrets(config, configDetails)
return err
},
},
{
key: "configs",
fnc: func(config map[string]any) error {
cfg.Configs, err = LoadConfigObjs(config, configDetails)
return err
},
},
}
for _, loader := range loaders {
if err := loader.fnc(getSection(config, loader.key)); err != nil {
return nil, err
}
}
cfg.Extras = getExtras(config)
return &cfg, nil
}
func getSection(config map[string]any, key string) map[string]any {
section, ok := config[key]
if !ok {
return make(map[string]any)
}
return section.(map[string]any)
}
// GetUnsupportedProperties returns the list of any unsupported properties that are
// used in the Compose files.
func GetUnsupportedProperties(configDicts ...map[string]any) []string {
unsupported := map[string]bool{}
for _, configDict := range configDicts {
for _, service := range getServices(configDict) {
serviceDict := service.(map[string]any)
for _, property := range types.UnsupportedProperties {
if _, isSet := serviceDict[property]; isSet {
unsupported[property] = true
}
}
}
}
return sortedKeys(unsupported)
}
func sortedKeys(set map[string]bool) []string {
linting: Consider pre-allocating sliceVar (prealloc) While updating, also addressed some redundant fmt.Sprintf() opts/throttledevice.go:86:2: Consider pre-allocating `out` (prealloc) var out []string ^ opts/ulimit.go:37:2: Consider pre-allocating `out` (prealloc) var out []string ^ opts/ulimit.go:47:2: Consider pre-allocating `ulimits` (prealloc) var ulimits []*units.Ulimit ^ opts/weightdevice.go:68:2: Consider pre-allocating `out` (prealloc) var out []string ^ cli/context/store/metadatastore.go:96:2: Consider pre-allocating `res` (prealloc) var res []Metadata ^ cli/context/store/store.go:127:2: Consider pre-allocating `names` (prealloc) var names []string ^ cli/compose/loader/loader.go:223:2: Consider pre-allocating `keys` (prealloc) var keys []string ^ cli/compose/loader/loader.go:397:2: Consider pre-allocating `services` (prealloc) var services []types.ServiceConfig ^ cli/command/stack/loader/loader.go:63:2: Consider pre-allocating `msgs` (prealloc) var msgs []string ^ cli/command/stack/loader/loader.go:118:2: Consider pre-allocating `configFiles` (prealloc) var configFiles []composetypes.ConfigFile ^ cli/command/formatter/container.go:245:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/formatter/container.go:265:2: Consider pre-allocating `mounts` (prealloc) var mounts []string ^ cli/command/formatter/container.go:316:2: Consider pre-allocating `result` (prealloc) var result []string ^ cli/command/formatter/displayutils.go:43:2: Consider pre-allocating `display` (prealloc) var ( ^ cli/command/formatter/volume.go:103:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli-plugins/manager/manager_test.go:49:2: Consider pre-allocating `dirs` (prealloc) var dirs []string ^ cli/command/swarm/init.go:69:2: Consider pre-allocating `defaultAddrPool` (prealloc) var defaultAddrPool []string ^ cli/command/manifest/push.go:195:2: Consider pre-allocating `blobReqs` (prealloc) var blobReqs []manifestBlob ^ cli/command/secret/formatter.go:111:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/network/formatter.go:104:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/context/list.go:52:2: Consider pre-allocating `contexts` (prealloc) var contexts []*formatter.ClientContext ^ cli/command/config/formatter.go:104:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/trust/common_test.go:23:2: Consider pre-allocating `targetNames` (prealloc) var targetNames []string ^ cli/command/service/generic_resource_opts.go:55:2: Consider pre-allocating `generic` (prealloc) var generic []swarm.GenericResource ^ cli/command/service/generic_resource_opts.go:98:2: Consider pre-allocating `l` (prealloc) var l []swarm.GenericResource ^ cli/command/service/opts.go:378:2: Consider pre-allocating `netAttach` (prealloc) var netAttach []swarm.NetworkAttachmentConfig ^ cli/command/service/update.go:731:2: Consider pre-allocating `limits` (prealloc) var limits []*units.Ulimit ^ cli/command/service/update.go:1315:2: Consider pre-allocating `newNetworks` (prealloc) var newNetworks []swarm.NetworkAttachmentConfig ^ cli/command/service/update.go:1514:2: Consider pre-allocating `out` (prealloc) var out []string ^ cli/compose/convert/service.go:713:2: Consider pre-allocating `ulimits` (prealloc) var ulimits []*units.Ulimit ^ cli/compose/convert/volume.go:13:2: Consider pre-allocating `mounts` (prealloc) var mounts []mount.Mount ^ cli/command/stack/swarm/list.go:39:2: Consider pre-allocating `stacks` (prealloc) var stacks []*formatter.Stack ^ Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-03 14:07:29 -04:00
keys := make([]string, 0, len(set))
for key := range set {
keys = append(keys, key)
}
sort.Strings(keys)
return keys
}
// GetDeprecatedProperties returns the list of any deprecated properties that
// are used in the compose files.
func GetDeprecatedProperties(configDicts ...map[string]any) map[string]string {
deprecated := map[string]string{}
for _, configDict := range configDicts {
deprecatedProperties := getProperties(getServices(configDict), types.DeprecatedProperties)
for key, value := range deprecatedProperties {
deprecated[key] = value
}
}
return deprecated
}
func getProperties(services map[string]any, propertyMap map[string]string) map[string]string {
output := map[string]string{}
for _, service := range services {
if serviceDict, ok := service.(map[string]any); ok {
for property, description := range propertyMap {
if _, isSet := serviceDict[property]; isSet {
output[property] = description
}
}
}
}
return output
}
// ForbiddenPropertiesError is returned when there are properties in the Compose
// file that are forbidden.
type ForbiddenPropertiesError struct {
Properties map[string]string
}
func (e *ForbiddenPropertiesError) Error() string {
return "Configuration contains forbidden properties"
}
func getServices(configDict map[string]any) map[string]any {
if services, ok := configDict["services"]; ok {
if servicesDict, ok := services.(map[string]any); ok {
return servicesDict
}
}
return map[string]any{}
}
// Transform converts the source into the target struct with compose types transformer
// and the specified transformers if any.
func Transform(source any, target any, additionalTransformers ...Transformer) error {
data := mapstructure.Metadata{}
config := &mapstructure.DecoderConfig{
DecodeHook: mapstructure.ComposeDecodeHookFunc(
createTransformHook(additionalTransformers...),
mapstructure.StringToTimeDurationHookFunc()),
Result: target,
Metadata: &data,
}
decoder, err := mapstructure.NewDecoder(config)
if err != nil {
return err
}
return decoder.Decode(source)
}
// TransformerFunc defines a function to perform the actual transformation
type TransformerFunc func(any) (any, error)
// Transformer defines a map to type transformer
type Transformer struct {
TypeOf reflect.Type
Func TransformerFunc
}
func createTransformHook(additionalTransformers ...Transformer) mapstructure.DecodeHookFuncType {
transforms := map[reflect.Type]func(any) (any, error){
reflect.TypeOf(types.External{}): transformExternal,
reflect.TypeOf(types.HealthCheckTest{}): transformHealthCheckTest,
reflect.TypeOf(types.ShellCommand{}): transformShellCommand,
reflect.TypeOf(types.StringList{}): transformStringList,
reflect.TypeOf(map[string]string{}): transformMapStringString,
reflect.TypeOf(types.UlimitsConfig{}): transformUlimits,
reflect.TypeOf(types.UnitBytes(0)): transformSize,
reflect.TypeOf([]types.ServicePortConfig{}): transformServicePort,
reflect.TypeOf(types.ServiceSecretConfig{}): transformStringSourceMap,
reflect.TypeOf(types.ServiceConfigObjConfig{}): transformStringSourceMap,
reflect.TypeOf(types.StringOrNumberList{}): transformStringOrNumberList,
reflect.TypeOf(map[string]*types.ServiceNetworkConfig{}): transformServiceNetworkMap,
reflect.TypeOf(types.Mapping{}): transformMappingOrListFunc("=", false),
reflect.TypeOf(types.MappingWithEquals{}): transformMappingOrListFunc("=", true),
reflect.TypeOf(types.Labels{}): transformMappingOrListFunc("=", false),
reflect.TypeOf(types.MappingWithColon{}): transformMappingOrListFunc(":", false),
reflect.TypeOf(types.HostsList{}): transformHostsList,
reflect.TypeOf(types.ServiceVolumeConfig{}): transformServiceVolumeConfig,
reflect.TypeOf(types.BuildConfig{}): transformBuildConfig,
reflect.TypeOf(types.Duration(0)): transformStringToDuration,
}
for _, transformer := range additionalTransformers {
transforms[transformer.TypeOf] = transformer.Func
}
return func(_ reflect.Type, target reflect.Type, data any) (any, error) {
transform, ok := transforms[target]
if !ok {
return data, nil
}
return transform(data)
}
}
// keys needs to be converted to strings for jsonschema
func convertToStringKeysRecursive(value any, keyPrefix string) (any, error) {
if mapping, ok := value.(map[any]any); ok {
dict := make(map[string]any)
for key, entry := range mapping {
str, ok := key.(string)
if !ok {
return nil, formatInvalidKeyError(keyPrefix, key)
}
var newKeyPrefix string
if keyPrefix == "" {
newKeyPrefix = str
} else {
newKeyPrefix = fmt.Sprintf("%s.%s", keyPrefix, str)
}
convertedEntry, err := convertToStringKeysRecursive(entry, newKeyPrefix)
if err != nil {
return nil, err
}
dict[str] = convertedEntry
}
return dict, nil
}
if list, ok := value.([]any); ok {
var convertedList []any
for index, entry := range list {
newKeyPrefix := fmt.Sprintf("%s[%d]", keyPrefix, index)
convertedEntry, err := convertToStringKeysRecursive(entry, newKeyPrefix)
if err != nil {
return nil, err
}
convertedList = append(convertedList, convertedEntry)
}
return convertedList, nil
}
return value, nil
}
func formatInvalidKeyError(keyPrefix string, key any) error {
var location string
if keyPrefix == "" {
location = "at top level"
} else {
location = "in " + keyPrefix
}
return errors.Errorf("non-string key %s: %#v", location, key)
}
// LoadServices produces a ServiceConfig map from a compose file Dict
// the servicesDict is not validated if directly used. Use Load() to enable validation
func LoadServices(servicesDict map[string]any, workingDir string, lookupEnv template.Mapping) ([]types.ServiceConfig, error) {
linting: Consider pre-allocating sliceVar (prealloc) While updating, also addressed some redundant fmt.Sprintf() opts/throttledevice.go:86:2: Consider pre-allocating `out` (prealloc) var out []string ^ opts/ulimit.go:37:2: Consider pre-allocating `out` (prealloc) var out []string ^ opts/ulimit.go:47:2: Consider pre-allocating `ulimits` (prealloc) var ulimits []*units.Ulimit ^ opts/weightdevice.go:68:2: Consider pre-allocating `out` (prealloc) var out []string ^ cli/context/store/metadatastore.go:96:2: Consider pre-allocating `res` (prealloc) var res []Metadata ^ cli/context/store/store.go:127:2: Consider pre-allocating `names` (prealloc) var names []string ^ cli/compose/loader/loader.go:223:2: Consider pre-allocating `keys` (prealloc) var keys []string ^ cli/compose/loader/loader.go:397:2: Consider pre-allocating `services` (prealloc) var services []types.ServiceConfig ^ cli/command/stack/loader/loader.go:63:2: Consider pre-allocating `msgs` (prealloc) var msgs []string ^ cli/command/stack/loader/loader.go:118:2: Consider pre-allocating `configFiles` (prealloc) var configFiles []composetypes.ConfigFile ^ cli/command/formatter/container.go:245:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/formatter/container.go:265:2: Consider pre-allocating `mounts` (prealloc) var mounts []string ^ cli/command/formatter/container.go:316:2: Consider pre-allocating `result` (prealloc) var result []string ^ cli/command/formatter/displayutils.go:43:2: Consider pre-allocating `display` (prealloc) var ( ^ cli/command/formatter/volume.go:103:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli-plugins/manager/manager_test.go:49:2: Consider pre-allocating `dirs` (prealloc) var dirs []string ^ cli/command/swarm/init.go:69:2: Consider pre-allocating `defaultAddrPool` (prealloc) var defaultAddrPool []string ^ cli/command/manifest/push.go:195:2: Consider pre-allocating `blobReqs` (prealloc) var blobReqs []manifestBlob ^ cli/command/secret/formatter.go:111:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/network/formatter.go:104:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/context/list.go:52:2: Consider pre-allocating `contexts` (prealloc) var contexts []*formatter.ClientContext ^ cli/command/config/formatter.go:104:2: Consider pre-allocating `joinLabels` (prealloc) var joinLabels []string ^ cli/command/trust/common_test.go:23:2: Consider pre-allocating `targetNames` (prealloc) var targetNames []string ^ cli/command/service/generic_resource_opts.go:55:2: Consider pre-allocating `generic` (prealloc) var generic []swarm.GenericResource ^ cli/command/service/generic_resource_opts.go:98:2: Consider pre-allocating `l` (prealloc) var l []swarm.GenericResource ^ cli/command/service/opts.go:378:2: Consider pre-allocating `netAttach` (prealloc) var netAttach []swarm.NetworkAttachmentConfig ^ cli/command/service/update.go:731:2: Consider pre-allocating `limits` (prealloc) var limits []*units.Ulimit ^ cli/command/service/update.go:1315:2: Consider pre-allocating `newNetworks` (prealloc) var newNetworks []swarm.NetworkAttachmentConfig ^ cli/command/service/update.go:1514:2: Consider pre-allocating `out` (prealloc) var out []string ^ cli/compose/convert/service.go:713:2: Consider pre-allocating `ulimits` (prealloc) var ulimits []*units.Ulimit ^ cli/compose/convert/volume.go:13:2: Consider pre-allocating `mounts` (prealloc) var mounts []mount.Mount ^ cli/command/stack/swarm/list.go:39:2: Consider pre-allocating `stacks` (prealloc) var stacks []*formatter.Stack ^ Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-09-03 14:07:29 -04:00
services := make([]types.ServiceConfig, 0, len(servicesDict))
for name, serviceDef := range servicesDict {
serviceConfig, err := LoadService(name, serviceDef.(map[string]any), workingDir, lookupEnv)
if err != nil {
return nil, err
}
services = append(services, *serviceConfig)
}
return services, nil
}
// LoadService produces a single ServiceConfig from a compose file Dict
// the serviceDict is not validated if directly used. Use Load() to enable validation
func LoadService(name string, serviceDict map[string]any, workingDir string, lookupEnv template.Mapping) (*types.ServiceConfig, error) {
serviceConfig := &types.ServiceConfig{}
if err := Transform(serviceDict, serviceConfig); err != nil {
return nil, err
}
serviceConfig.Name = name
if err := resolveEnvironment(serviceConfig, workingDir, lookupEnv); err != nil {
return nil, err
}
if err := resolveVolumePaths(serviceConfig.Volumes, workingDir, lookupEnv); err != nil {
return nil, err
}
serviceConfig.Extras = getExtras(serviceDict)
return serviceConfig, nil
}
func loadExtras(name string, source map[string]any) map[string]any {
if dict, ok := source[name].(map[string]any); ok {
return getExtras(dict)
}
return nil
}
func getExtras(dict map[string]any) map[string]any {
extras := map[string]any{}
for key, value := range dict {
if strings.HasPrefix(key, "x-") {
extras[key] = value
}
}
if len(extras) == 0 {
return nil
}
return extras
}
func updateEnvironment(environment map[string]*string, vars map[string]*string, lookupEnv template.Mapping) {
for k, v := range vars {
interpolatedV, ok := lookupEnv(k)
if (v == nil || *v == "") && ok {
// lookupEnv is prioritized over vars
environment[k] = &interpolatedV
} else {
environment[k] = v
}
}
}
func resolveEnvironment(serviceConfig *types.ServiceConfig, workingDir string, lookupEnv template.Mapping) error {
environment := make(map[string]*string)
if len(serviceConfig.EnvFile) > 0 {
var envVars []string
for _, file := range serviceConfig.EnvFile {
filePath := absPath(workingDir, file)
fileVars, err := opts.ParseEnvFile(filePath)
if err != nil {
return err
}
envVars = append(envVars, fileVars...)
}
updateEnvironment(environment,
opts.ConvertKVStringsToMapWithNil(envVars), lookupEnv)
}
updateEnvironment(environment, serviceConfig.Environment, lookupEnv)
serviceConfig.Environment = environment
return nil
}
func resolveVolumePaths(volumes []types.ServiceVolumeConfig, workingDir string, lookupEnv template.Mapping) error {
for i, volume := range volumes {
if volume.Type != "bind" {
continue
}
if volume.Source == "" {
return errors.New(`invalid mount config for type "bind": field Source must not be empty`)
}
filePath := expandUser(volume.Source, lookupEnv)
// Check if source is an absolute path (either Unix or Windows), to
// handle a Windows client with a Unix daemon or vice-versa.
//
// Note that this is not required for Docker for Windows when specifying
// a local Windows path, because Docker for Windows translates the Windows
// path into a valid path within the VM.
if !path.IsAbs(filePath) && !isAbs(filePath) {
filePath = absPath(workingDir, filePath)
}
volume.Source = filePath
volumes[i] = volume
}
return nil
}
// TODO: make this more robust
func expandUser(srcPath string, lookupEnv template.Mapping) string {
if strings.HasPrefix(srcPath, "~") {
home, ok := lookupEnv("HOME")
if !ok {
logrus.Warn("cannot expand '~', because the environment lacks HOME")
return srcPath
}
return strings.Replace(srcPath, "~", home, 1)
}
return srcPath
}
func transformUlimits(data any) (any, error) {
switch value := data.(type) {
case int:
return types.UlimitsConfig{Single: value}, nil
case map[string]any:
ulimit := types.UlimitsConfig{}
ulimit.Soft = value["soft"].(int)
ulimit.Hard = value["hard"].(int)
return ulimit, nil
default:
return data, errors.Errorf("invalid type %T for ulimits", value)
}
}
// LoadNetworks produces a NetworkConfig map from a compose file Dict
// the source Dict is not validated if directly used. Use Load() to enable validation
func LoadNetworks(source map[string]any, version string) (map[string]types.NetworkConfig, error) {
networks := make(map[string]types.NetworkConfig)
err := Transform(source, &networks)
if err != nil {
return networks, err
}
for name, network := range networks {
if !network.External.External {
continue
}
switch {
case network.External.Name != "":
if network.Name != "" {
return nil, errors.Errorf("network %s: network.external.name and network.name conflict; only use network.name", name)
}
if versions.GreaterThanOrEqualTo(version, "3.5") {
logrus.Warnf("network %s: network.external.name is deprecated in favor of network.name", name)
}
network.Name = network.External.Name
network.External.Name = ""
case network.Name == "":
network.Name = name
}
network.Extras = loadExtras(name, source)
networks[name] = network
}
return networks, nil
}
func externalVolumeError(volume, key string) error {
return errors.Errorf(
"conflicting parameters \"external\" and %q specified for volume %q",
key, volume)
}
// LoadVolumes produces a VolumeConfig map from a compose file Dict
// the source Dict is not validated if directly used. Use Load() to enable validation
func LoadVolumes(source map[string]any, version string) (map[string]types.VolumeConfig, error) {
volumes := make(map[string]types.VolumeConfig)
if err := Transform(source, &volumes); err != nil {
return volumes, err
}
for name, volume := range volumes {
if !volume.External.External {
continue
}
switch {
case volume.Driver != "":
return nil, externalVolumeError(name, "driver")
case len(volume.DriverOpts) > 0:
return nil, externalVolumeError(name, "driver_opts")
case len(volume.Labels) > 0:
return nil, externalVolumeError(name, "labels")
case volume.External.Name != "":
if volume.Name != "" {
return nil, errors.Errorf("volume %s: volume.external.name and volume.name conflict; only use volume.name", name)
}
if versions.GreaterThanOrEqualTo(version, "3.4") {
logrus.Warnf("volume %s: volume.external.name is deprecated in favor of volume.name", name)
}
volume.Name = volume.External.Name
volume.External.Name = ""
case volume.Name == "":
volume.Name = name
}
volume.Extras = loadExtras(name, source)
volumes[name] = volume
}
return volumes, nil
}
// LoadSecrets produces a SecretConfig map from a compose file Dict
// the source Dict is not validated if directly used. Use Load() to enable validation
func LoadSecrets(source map[string]any, details types.ConfigDetails) (map[string]types.SecretConfig, error) {
secrets := make(map[string]types.SecretConfig)
if err := Transform(source, &secrets); err != nil {
return secrets, err
}
for name, secret := range secrets {
obj, err := loadFileObjectConfig(name, "secret", types.FileObjectConfig(secret), details)
if err != nil {
return nil, err
}
secretConfig := types.SecretConfig(obj)
secretConfig.Extras = loadExtras(name, source)
secrets[name] = secretConfig
}
return secrets, nil
}
// LoadConfigObjs produces a ConfigObjConfig map from a compose file Dict
// the source Dict is not validated if directly used. Use Load() to enable validation
func LoadConfigObjs(source map[string]any, details types.ConfigDetails) (map[string]types.ConfigObjConfig, error) {
configs := make(map[string]types.ConfigObjConfig)
if err := Transform(source, &configs); err != nil {
return configs, err
}
for name, config := range configs {
obj, err := loadFileObjectConfig(name, "config", types.FileObjectConfig(config), details)
if err != nil {
return nil, err
}
configConfig := types.ConfigObjConfig(obj)
configConfig.Extras = loadExtras(name, source)
configs[name] = configConfig
}
return configs, nil
}
func loadFileObjectConfig(name string, objType string, obj types.FileObjectConfig, details types.ConfigDetails) (types.FileObjectConfig, error) {
// if "external: true"
switch {
case obj.External.External:
// handle deprecated external.name
if obj.External.Name != "" {
if obj.Name != "" {
return obj, errors.Errorf("%[1]s %[2]s: %[1]s.external.name and %[1]s.name conflict; only use %[1]s.name", objType, name)
}
if versions.GreaterThanOrEqualTo(details.Version, "3.5") {
logrus.Warnf("%[1]s %[2]s: %[1]s.external.name is deprecated in favor of %[1]s.name", objType, name)
}
obj.Name = obj.External.Name
obj.External.Name = ""
linting: address else/if/elseif statements found by gocritic cli/command/formatter/tabwriter/tabwriter.go:579:10: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ cli/connhelper/connhelper.go:43:2: singleCaseSwitch: should rewrite switch statement to if statement (gocritic) switch scheme := u.Scheme; scheme { ^ cli/compose/loader/loader.go:666:10: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ opts/hosts_test.go:173:10: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ cli-plugins/manager/candidate_test.go:78:4: ifElseChain: rewrite if-else to switch statement (gocritic) if tc.err != "" { ^ cli/command/checkpoint/formatter.go:15:2: singleCaseSwitch: should rewrite switch statement to if statement (gocritic) switch source { ^ cli/command/image/formatter_history.go:25:2: singleCaseSwitch: should rewrite switch statement to if statement (gocritic) switch source { ^ cli/command/service/scale.go:107:2: ifElseChain: rewrite if-else to switch statement (gocritic) if serviceMode.Replicated != nil { ^ cli/command/service/update.go:804:9: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ cli/command/service/update.go:222:2: ifElseChain: rewrite if-else to switch statement (gocritic) if sendAuth { ^ cli/command/container/formatter_diff.go:17:2: singleCaseSwitch: should rewrite switch statement to if statement (gocritic) switch source { ^ cli/command/container/start.go:79:2: ifElseChain: rewrite if-else to switch statement (gocritic) if opts.Attach || opts.OpenStdin { ^ cli/command/container/utils.go:84:11: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ cli/command/container/exec_test.go:200:11: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ cli/command/container/logs_test.go:52:11: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ cli/command/container/opts_test.go:1014:10: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic) } else { ^ cli/command/system/info.go:297:7: singleCaseSwitch: should rewrite switch statement to if statement (gocritic) switch o.Key { ^ cli/command/system/version.go:164:4: singleCaseSwitch: should rewrite switch statement to if statement (gocritic) switch component.Name { ^ cli/command/system/info_test.go:478:4: ifElseChain: rewrite if-else to switch statement (gocritic) if tc.expectedOut != "" { ^ Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-11-20 07:54:53 -05:00
} else if obj.Name == "" {
obj.Name = name
}
// if not "external: true"
case obj.Driver != "":
if obj.File != "" {
return obj, errors.Errorf("%[1]s %[2]s: %[1]s.driver and %[1]s.file conflict; only use %[1]s.driver", objType, name)
}
default:
obj.File = absPath(details.WorkingDir, obj.File)
}
return obj, nil
}
func absPath(workingDir string, filePath string) string {
if filepath.IsAbs(filePath) {
return filePath
}
return filepath.Join(workingDir, filePath)
}
var transformMapStringString TransformerFunc = func(data any) (any, error) {
switch value := data.(type) {
case map[string]any:
return toMapStringString(value, false), nil
case map[string]string:
return value, nil
default:
return data, errors.Errorf("invalid type %T for map[string]string", value)
}
}
var transformExternal TransformerFunc = func(data any) (any, error) {
switch value := data.(type) {
case bool:
return map[string]any{"external": value}, nil
case map[string]any:
return map[string]any{"external": true, "name": value["name"]}, nil
default:
return data, errors.Errorf("invalid type %T for external", value)
}
}
var transformServicePort TransformerFunc = func(data any) (any, error) {
switch entries := data.(type) {
case []any:
// We process the list instead of individual items here.
// The reason is that one entry might be mapped to multiple ServicePortConfig.
// Therefore we take an input of a list and return an output of a list.
ports := []any{}
for _, entry := range entries {
switch value := entry.(type) {
case int:
golangci-lint: enable perfsprint linter cli/compose/types/types.go:568:17: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return []byte(fmt.Sprintf("%v", e.External)), nil ^ cli/command/formatter/buildcache.go:174:9: fmt.Sprintf can be replaced with faster strconv.Itoa (perfsprint) return fmt.Sprintf("%d", c.v.UsageCount) ^ cli/command/formatter/buildcache.go:178:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%t", c.v.InUse) ^ cli/command/formatter/buildcache.go:182:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%t", c.v.Shared) ^ cli/command/formatter/image.go:259:9: fmt.Sprintf can be replaced with faster strconv.FormatInt (perfsprint) return fmt.Sprintf("%d", c.i.Containers) ^ cli/command/formatter/tabwriter/tabwriter_test.go:698:9: fmt.Sprintf can be replaced with faster strconv.Itoa (perfsprint) b.Run(fmt.Sprintf("%d", x), func(b *testing.B) { ^ cli/command/formatter/tabwriter/tabwriter_test.go:720:9: fmt.Sprintf can be replaced with faster strconv.Itoa (perfsprint) b.Run(fmt.Sprintf("%d", h), func(b *testing.B) { ^ cli/command/image/prune.go:62:31: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) pruneFilters.Add("dangling", fmt.Sprintf("%v", !options.all)) ^ cli/command/network/formatter.go:92:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%v", c.n.EnableIPv6) ^ cli/command/network/formatter.go:96:9: fmt.Sprintf can be replaced with faster strconv.FormatBool (perfsprint) return fmt.Sprintf("%v", c.n.Internal) ^ cli/command/service/formatter.go:745:9: fmt.Sprintf can be replaced with faster strconv.FormatUint (perfsprint) pub = fmt.Sprintf("%d", pr.pStart) ^ cli/command/service/formatter.go:750:9: fmt.Sprintf can be replaced with faster strconv.FormatUint (perfsprint) tgt = fmt.Sprintf("%d", pr.tStart) ^ cli/command/service/opts.go:49:10: fmt.Sprintf can be replaced with faster strconv.FormatUint (perfsprint) return fmt.Sprintf("%v", *i.value) ^ cli/compose/loader/loader.go:720:36: fmt.Sprint can be replaced with faster strconv.Itoa (perfsprint) v, err := toServicePortConfigs(fmt.Sprint(value)) ^ Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-11-20 10:18:19 -05:00
v, err := toServicePortConfigs(strconv.Itoa(value))
if err != nil {
return data, err
}
ports = append(ports, v...)
case string:
v, err := toServicePortConfigs(value)
if err != nil {
return data, err
}
ports = append(ports, v...)
case map[string]any:
ports = append(ports, value)
default:
return data, errors.Errorf("invalid type %T for port", value)
}
}
return ports, nil
default:
return data, errors.Errorf("invalid type %T for port", entries)
}
}
var transformStringSourceMap TransformerFunc = func(data any) (any, error) {
switch value := data.(type) {
case string:
return map[string]any{"source": value}, nil
case map[string]any:
return data, nil
default:
return data, errors.Errorf("invalid type %T for secret", value)
}
}
var transformBuildConfig TransformerFunc = func(data any) (any, error) {
switch value := data.(type) {
case string:
return map[string]any{"context": value}, nil
case map[string]any:
return data, nil
default:
return data, errors.Errorf("invalid type %T for service build", value)
}
}
var transformServiceVolumeConfig TransformerFunc = func(data any) (any, error) {
switch value := data.(type) {
case string:
return ParseVolume(value)
case map[string]any:
return data, nil
default:
return data, errors.Errorf("invalid type %T for service volume", value)
}
}
var transformServiceNetworkMap TransformerFunc = func(value any) (any, error) {
if list, ok := value.([]any); ok {
mapValue := map[any]any{}
for _, name := range list {
mapValue[name] = nil
}
return mapValue, nil
}
return value, nil
}
var transformStringOrNumberList TransformerFunc = func(value any) (any, error) {
list := value.([]any)
result := make([]string, len(list))
for i, item := range list {
result[i] = fmt.Sprint(item)
}
return result, nil
}
var transformStringList TransformerFunc = func(data any) (any, error) {
switch value := data.(type) {
case string:
return []string{value}, nil
case []any:
return value, nil
default:
return data, errors.Errorf("invalid type %T for string list", value)
}
}
var transformHostsList TransformerFunc = func(data any) (any, error) {
hl := transformListOrMapping(data, ":", false, []string{"=", ":"})
// Remove brackets from IP addresses if present (for example "[::1]" -> "::1").
result := make([]string, 0, len(hl))
for _, hip := range hl {
host, ip, _ := strings.Cut(hip, ":")
if len(ip) > 2 && ip[0] == '[' && ip[len(ip)-1] == ']' {
ip = ip[1 : len(ip)-1]
}
result = append(result, fmt.Sprintf("%s:%s", host, ip))
Preserve sort-order of extra hosts, and allow duplicate entries Extra hosts (`extra_hosts` in compose-file, or `--hosts` in services) adds custom host/ip mappings to the container's `/etc/hosts`. The current implementation used a `map[string]string{}` as intermediate storage, and sorted the results alphabetically when converting to a service-spec. As a result, duplicate hosts were removed, and order of host/ip mappings was not preserved (in case the compose-file used a list instead of a map). According to the **host.conf(5)** man page (http://man7.org/linux/man-pages/man5/host.conf.5.html) multi Valid values are on and off. If set to on, the resolver library will return all valid addresses for a host that appears in the /etc/hosts file, instead of only the first. This is off by default, as it may cause a substantial performance loss at sites with large hosts files. Multiple entries for a host are allowed, and even required for some situations, for example, to add mappings for IPv4 and IPv6 addreses for a host, as illustrated by the example hosts file in the **hosts(5)** man page (http://man7.org/linux/man-pages/man5/hosts.5.html): # The following lines are desirable for IPv4 capable hosts 127.0.0.1 localhost # 127.0.1.1 is often used for the FQDN of the machine 127.0.1.1 thishost.mydomain.org thishost 192.168.1.10 foo.mydomain.org foo 192.168.1.13 bar.mydomain.org bar 146.82.138.7 master.debian.org master 209.237.226.90 www.opensource.org # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters This patch changes the intermediate storage format to use a `[]string`, and only sorts entries if the input format in the compose file is a mapping. If the input format is a list, the original sort-order is preserved. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-10-29 20:33:23 -04:00
}
return result, nil
Preserve sort-order of extra hosts, and allow duplicate entries Extra hosts (`extra_hosts` in compose-file, or `--hosts` in services) adds custom host/ip mappings to the container's `/etc/hosts`. The current implementation used a `map[string]string{}` as intermediate storage, and sorted the results alphabetically when converting to a service-spec. As a result, duplicate hosts were removed, and order of host/ip mappings was not preserved (in case the compose-file used a list instead of a map). According to the **host.conf(5)** man page (http://man7.org/linux/man-pages/man5/host.conf.5.html) multi Valid values are on and off. If set to on, the resolver library will return all valid addresses for a host that appears in the /etc/hosts file, instead of only the first. This is off by default, as it may cause a substantial performance loss at sites with large hosts files. Multiple entries for a host are allowed, and even required for some situations, for example, to add mappings for IPv4 and IPv6 addreses for a host, as illustrated by the example hosts file in the **hosts(5)** man page (http://man7.org/linux/man-pages/man5/hosts.5.html): # The following lines are desirable for IPv4 capable hosts 127.0.0.1 localhost # 127.0.1.1 is often used for the FQDN of the machine 127.0.1.1 thishost.mydomain.org thishost 192.168.1.10 foo.mydomain.org foo 192.168.1.13 bar.mydomain.org bar 146.82.138.7 master.debian.org master 209.237.226.90 www.opensource.org # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters This patch changes the intermediate storage format to use a `[]string`, and only sorts entries if the input format in the compose file is a mapping. If the input format is a list, the original sort-order is preserved. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-10-29 20:33:23 -04:00
}
// transformListOrMapping transforms pairs of strings that may be represented as
// a map, or a list of '=' or ':' separated strings, into a list of ':' separated
// strings.
func transformListOrMapping(listOrMapping any, sep string, allowNil bool, allowSeps []string) []string {
Preserve sort-order of extra hosts, and allow duplicate entries Extra hosts (`extra_hosts` in compose-file, or `--hosts` in services) adds custom host/ip mappings to the container's `/etc/hosts`. The current implementation used a `map[string]string{}` as intermediate storage, and sorted the results alphabetically when converting to a service-spec. As a result, duplicate hosts were removed, and order of host/ip mappings was not preserved (in case the compose-file used a list instead of a map). According to the **host.conf(5)** man page (http://man7.org/linux/man-pages/man5/host.conf.5.html) multi Valid values are on and off. If set to on, the resolver library will return all valid addresses for a host that appears in the /etc/hosts file, instead of only the first. This is off by default, as it may cause a substantial performance loss at sites with large hosts files. Multiple entries for a host are allowed, and even required for some situations, for example, to add mappings for IPv4 and IPv6 addreses for a host, as illustrated by the example hosts file in the **hosts(5)** man page (http://man7.org/linux/man-pages/man5/hosts.5.html): # The following lines are desirable for IPv4 capable hosts 127.0.0.1 localhost # 127.0.1.1 is often used for the FQDN of the machine 127.0.1.1 thishost.mydomain.org thishost 192.168.1.10 foo.mydomain.org foo 192.168.1.13 bar.mydomain.org bar 146.82.138.7 master.debian.org master 209.237.226.90 www.opensource.org # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters This patch changes the intermediate storage format to use a `[]string`, and only sorts entries if the input format in the compose file is a mapping. If the input format is a list, the original sort-order is preserved. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-10-29 20:33:23 -04:00
switch value := listOrMapping.(type) {
case map[string]any:
Preserve sort-order of extra hosts, and allow duplicate entries Extra hosts (`extra_hosts` in compose-file, or `--hosts` in services) adds custom host/ip mappings to the container's `/etc/hosts`. The current implementation used a `map[string]string{}` as intermediate storage, and sorted the results alphabetically when converting to a service-spec. As a result, duplicate hosts were removed, and order of host/ip mappings was not preserved (in case the compose-file used a list instead of a map). According to the **host.conf(5)** man page (http://man7.org/linux/man-pages/man5/host.conf.5.html) multi Valid values are on and off. If set to on, the resolver library will return all valid addresses for a host that appears in the /etc/hosts file, instead of only the first. This is off by default, as it may cause a substantial performance loss at sites with large hosts files. Multiple entries for a host are allowed, and even required for some situations, for example, to add mappings for IPv4 and IPv6 addreses for a host, as illustrated by the example hosts file in the **hosts(5)** man page (http://man7.org/linux/man-pages/man5/hosts.5.html): # The following lines are desirable for IPv4 capable hosts 127.0.0.1 localhost # 127.0.1.1 is often used for the FQDN of the machine 127.0.1.1 thishost.mydomain.org thishost 192.168.1.10 foo.mydomain.org foo 192.168.1.13 bar.mydomain.org bar 146.82.138.7 master.debian.org master 209.237.226.90 www.opensource.org # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters This patch changes the intermediate storage format to use a `[]string`, and only sorts entries if the input format in the compose file is a mapping. If the input format is a list, the original sort-order is preserved. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-10-29 20:33:23 -04:00
return toStringList(value, sep, allowNil)
case []any:
result := make([]string, 0, len(value))
for _, entry := range value {
for i, allowSep := range allowSeps {
entry := fmt.Sprint(entry)
k, v, ok := strings.Cut(entry, allowSep)
if ok {
// Entry uses this allowed separator. Add it to the result, using
// sep as a separator.
result = append(result, fmt.Sprintf("%s%s%s", k, sep, v))
break
} else if i == len(allowSeps)-1 {
// No more separators to try, keep the entry if allowNil.
if allowNil {
result = append(result, k)
}
}
}
}
return result
Preserve sort-order of extra hosts, and allow duplicate entries Extra hosts (`extra_hosts` in compose-file, or `--hosts` in services) adds custom host/ip mappings to the container's `/etc/hosts`. The current implementation used a `map[string]string{}` as intermediate storage, and sorted the results alphabetically when converting to a service-spec. As a result, duplicate hosts were removed, and order of host/ip mappings was not preserved (in case the compose-file used a list instead of a map). According to the **host.conf(5)** man page (http://man7.org/linux/man-pages/man5/host.conf.5.html) multi Valid values are on and off. If set to on, the resolver library will return all valid addresses for a host that appears in the /etc/hosts file, instead of only the first. This is off by default, as it may cause a substantial performance loss at sites with large hosts files. Multiple entries for a host are allowed, and even required for some situations, for example, to add mappings for IPv4 and IPv6 addreses for a host, as illustrated by the example hosts file in the **hosts(5)** man page (http://man7.org/linux/man-pages/man5/hosts.5.html): # The following lines are desirable for IPv4 capable hosts 127.0.0.1 localhost # 127.0.1.1 is often used for the FQDN of the machine 127.0.1.1 thishost.mydomain.org thishost 192.168.1.10 foo.mydomain.org foo 192.168.1.13 bar.mydomain.org bar 146.82.138.7 master.debian.org master 209.237.226.90 www.opensource.org # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters This patch changes the intermediate storage format to use a `[]string`, and only sorts entries if the input format in the compose file is a mapping. If the input format is a list, the original sort-order is preserved. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-10-29 20:33:23 -04:00
}
panic(errors.Errorf("expected a map or a list, got %T: %#v", listOrMapping, listOrMapping))
}
func transformMappingOrListFunc(sep string, allowNil bool) TransformerFunc {
return func(data any) (any, error) {
return transformMappingOrList(data, sep, allowNil), nil
}
}
func transformMappingOrList(mappingOrList any, sep string, allowNil bool) any {
switch values := mappingOrList.(type) {
case map[string]any:
return toMapStringString(values, allowNil)
case []any:
result := make(map[string]any)
for _, v := range values {
key, val, hasValue := strings.Cut(v.(string), sep)
switch {
case !hasValue && allowNil:
result[key] = nil
case !hasValue && !allowNil:
result[key] = ""
default:
result[key] = val
}
}
return result
}
panic(errors.Errorf("expected a map or a list, got %T: %#v", mappingOrList, mappingOrList))
}
var transformShellCommand TransformerFunc = func(value any) (any, error) {
if str, ok := value.(string); ok {
return shlex.Split(str)
}
return value, nil
}
var transformHealthCheckTest TransformerFunc = func(data any) (any, error) {
switch value := data.(type) {
case string:
return append([]string{"CMD-SHELL"}, value), nil
case []any:
return value, nil
default:
return value, errors.Errorf("invalid type %T for healthcheck.test", value)
}
}
var transformSize TransformerFunc = func(value any) (any, error) {
switch value := value.(type) {
case int:
return int64(value), nil
case string:
return units.RAMInBytes(value)
}
panic(errors.Errorf("invalid type for size %T", value))
}
var transformStringToDuration TransformerFunc = func(value any) (any, error) {
switch value := value.(type) {
case string:
d, err := time.ParseDuration(value)
if err != nil {
return value, err
}
return types.Duration(d), nil
default:
return value, errors.Errorf("invalid type %T for duration", value)
}
}
func toServicePortConfigs(value string) ([]any, error) {
var portConfigs []any
ports, portBindings, err := nat.ParsePortSpecs([]string{value})
if err != nil {
return nil, err
}
// We need to sort the key of the ports to make sure it is consistent
keys := []string{}
for port := range ports {
keys = append(keys, string(port))
}
sort.Strings(keys)
for _, key := range keys {
// Reuse ConvertPortToPortConfig so that it is consistent
portConfig, err := opts.ConvertPortToPortConfig(nat.Port(key), portBindings)
if err != nil {
return nil, err
}
for _, p := range portConfig {
portConfigs = append(portConfigs, types.ServicePortConfig{
Protocol: string(p.Protocol),
Target: p.TargetPort,
Published: p.PublishedPort,
Mode: string(p.PublishMode),
})
}
}
return portConfigs, nil
}
func toMapStringString(value map[string]any, allowNil bool) map[string]any {
output := make(map[string]any)
for key, value := range value {
output[key] = toString(value, allowNil)
}
return output
}
func toString(value any, allowNil bool) any {
switch {
case value != nil:
return fmt.Sprint(value)
case allowNil:
return nil
default:
return ""
}
}
Preserve sort-order of extra hosts, and allow duplicate entries Extra hosts (`extra_hosts` in compose-file, or `--hosts` in services) adds custom host/ip mappings to the container's `/etc/hosts`. The current implementation used a `map[string]string{}` as intermediate storage, and sorted the results alphabetically when converting to a service-spec. As a result, duplicate hosts were removed, and order of host/ip mappings was not preserved (in case the compose-file used a list instead of a map). According to the **host.conf(5)** man page (http://man7.org/linux/man-pages/man5/host.conf.5.html) multi Valid values are on and off. If set to on, the resolver library will return all valid addresses for a host that appears in the /etc/hosts file, instead of only the first. This is off by default, as it may cause a substantial performance loss at sites with large hosts files. Multiple entries for a host are allowed, and even required for some situations, for example, to add mappings for IPv4 and IPv6 addreses for a host, as illustrated by the example hosts file in the **hosts(5)** man page (http://man7.org/linux/man-pages/man5/hosts.5.html): # The following lines are desirable for IPv4 capable hosts 127.0.0.1 localhost # 127.0.1.1 is often used for the FQDN of the machine 127.0.1.1 thishost.mydomain.org thishost 192.168.1.10 foo.mydomain.org foo 192.168.1.13 bar.mydomain.org bar 146.82.138.7 master.debian.org master 209.237.226.90 www.opensource.org # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters This patch changes the intermediate storage format to use a `[]string`, and only sorts entries if the input format in the compose file is a mapping. If the input format is a list, the original sort-order is preserved. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-10-29 20:33:23 -04:00
func toStringList(value map[string]any, separator string, allowNil bool) []string {
Preserve sort-order of extra hosts, and allow duplicate entries Extra hosts (`extra_hosts` in compose-file, or `--hosts` in services) adds custom host/ip mappings to the container's `/etc/hosts`. The current implementation used a `map[string]string{}` as intermediate storage, and sorted the results alphabetically when converting to a service-spec. As a result, duplicate hosts were removed, and order of host/ip mappings was not preserved (in case the compose-file used a list instead of a map). According to the **host.conf(5)** man page (http://man7.org/linux/man-pages/man5/host.conf.5.html) multi Valid values are on and off. If set to on, the resolver library will return all valid addresses for a host that appears in the /etc/hosts file, instead of only the first. This is off by default, as it may cause a substantial performance loss at sites with large hosts files. Multiple entries for a host are allowed, and even required for some situations, for example, to add mappings for IPv4 and IPv6 addreses for a host, as illustrated by the example hosts file in the **hosts(5)** man page (http://man7.org/linux/man-pages/man5/hosts.5.html): # The following lines are desirable for IPv4 capable hosts 127.0.0.1 localhost # 127.0.1.1 is often used for the FQDN of the machine 127.0.1.1 thishost.mydomain.org thishost 192.168.1.10 foo.mydomain.org foo 192.168.1.13 bar.mydomain.org bar 146.82.138.7 master.debian.org master 209.237.226.90 www.opensource.org # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters This patch changes the intermediate storage format to use a `[]string`, and only sorts entries if the input format in the compose file is a mapping. If the input format is a list, the original sort-order is preserved. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2017-10-29 20:33:23 -04:00
output := []string{}
for key, value := range value {
if value == nil && !allowNil {
continue
}
output = append(output, fmt.Sprintf("%s%s%s", key, separator, value))
}
sort.Strings(output)
return output
}