diff --git a/README.md b/README.md index acfcb36..d040037 100644 --- a/README.md +++ b/README.md @@ -1,66 +1,78 @@ # OpenCHAMI Configurator -The `configurator` (portmanteau of config + generator) is an extensible tool that fetchs data from an instance of [SMD](https://github.com/OpenCHAMI/smd) to generate commonly used config files based on Jinja 2 template files. The tool and generator plugins are written in Go and plugins can be written by following the ["Creating Generator Plugins"](#creating-generator-plugins) section of this README. +The `configurator` is an extensible tool that is capable of dynamically generating files on the fly. The tool includes a built-in generator that fetchs data from an instance of [SMD](https://github.com/OpenCHAMI/smd) to generate files based on Jinja 2 template files. The tool and generator plugins are written in Go and plugins can be written by following the ["Creating Generator Plugins"](#creating-generator-plugins) section of this README. ## Building and Usage -The `configurator` is built using standard `go` build tools. The project separates the client and server with build tags. To get started, clone the project, download the dependencies, and build the project: +The `configurator` is built using standard `go` build tools. The project separates the client, server, and generator components using build tags. To get started, clone the project, download the dependencies, and build the project: ```bash git clone https://github.com/OpenCHAMI/configurator.git go mod tidy go build --tags all # equivalent to `go build --tags client,server`` - -## ...or just run `make` in project directory ``` -This will build the main driver program, but also requires generator plugins to define how new config files are generated. The default plugins can be built using the following build command: +This will build the main driver program with the default generators that are found in the `pkg/generators` directory. + +> [!WARNING] +> Not all of the plugins have completed generation implementations and are a WIP. + +### Running Configurator with CLI + +After you build the program, run the following command to use the tool: ```bash -go build -buildmode=plugin -o lib/conman.so internal/generator/plugins/conman/conman.go -go build -buildmode=plugin -o lib/coredhcp.so internal/generator/plugins/coredhcp/coredhcp.go -go build -buildmode=plugin -o lib/dnsmasq.so internal/generator/plugins/dnsmasq/dnsmasq.go -go build -buildmode=plugin -o lib/powerman.so internal/generator/plugins/powerman/powerman.go -go build -buildmode=plugin -o lib/syslog.so internal/generator/plugins/syslog/syslog.go +export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs... +./configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert ochami.pem ``` -**NOTE: Not all of the plugins have completed generation implementations and are WIP.** +This will generate a new `coredhcp` config file based on the Jinja 2 template specified in the config file for "coredhcp". The files will be written to `coredhcp.conf` as specified with the `-o/--output` flag. The `--target` flag specifies the type of config file to generate by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). -These commands will build the default plugins and store them in the "lib" directory by default. Alternatively, the plugins can be built using `make plugins` if GNU make is installed and available. After you build the plugins, run the following to use the tool: +In other words, there should be an entry in the config file that looks like this: + +```yaml +... +targets: + coredhcp: + plugin: "lib/coredhcp.so" # optional, if we want to use an external plugin instead + templates: + - templates/coredhcp.j2 +... -```bash -./configurator generate --config config.yaml --target dnsmasq -o dnsmasq.conf ``` -This will generate a new `dnsmasq` config file based on the Jinja 2 template specified in the config file for "dnsmasq". The `--target` flag specifies the type of config file to generate by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). The `configurator` tool requires a valid access token when making requests to an instance of SMD that has protected routes. +> [!NOTE] +> The `configurator` tool requires a valid access token when making requests to an instance of SMD that has protected routes. + +### Running Configurator as a Service The tool can also run as a service to generate files for clients: ```bash +export CONFIGURATOR_JWKS_URL="http://my.openchami.cluster:8443/key" ./configurator serve --config config.yaml ``` Once the server is up and listening for HTTP requests, you can try making a request to it with `curl` or `configurator fetch`. Both commands below are essentially equivalent: ```bash -curl http://127.0.0.1:3334/generate?target=dnsmasq -H "Authorization: Bearer $ACCESS_TOKEN" +export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs... +curl http://127.0.0.1:3334/generate?target=coredhcp -X GET -H "Authorization: Bearer $ACCESS_TOKEN" --cacert ochami.pem # ...or... -./configurator fetch --target dnsmasq --host http://127.0.0.1 --port 3334 +./configurator fetch --target coredhcp --host http://127.0.0.1:3334 --cacert ochami.pem ``` -This will do the same thing as the `generate` subcommand, but remotely. The access token is only required if the `CONFIGURATOR_JWKS_URL` environment variable is set. The `ACCESS_TOKEN` environment variable passed to `curl` and it's corresponding CLI argument both expects a token as a JWT. +This will do the same thing as the `generate` subcommand, but through a GET request where the file contents is returned in the response. The access token is only required if the `CONFIGURATOR_JWKS_URL` environment variable is set when starting the server with `serve`. The `ACCESS_TOKEN` environment variable is passed to `curl` using the `Authorization` header and expects a token as a JWT. ### Docker -New images can be built and tested using the `Dockerfile` provided in the project. However, the binary executable and the generator plugins must first be built before building the image since the Docker build copies the binary over. Therefore, build all of the binaries first by following the first section of ["Building and Usage"](#building-and-usage). If you run the `make docker`, this will be done for you. Otherwise, run the `docker build` command after building the executable and libraries. +New images can be built and tested using the `Dockerfile` provided in the project. However, the binary executable and the generator plugins must first be built before building the image since the Docker build copies the binary over. Therefore, build all of the binaries first by following the first section of ["Building and Usage"](#building-and-usage). Running `make docker` from the Makefile will automate this process. Otherwise, run the `docker build` command after building the executable and libraries. ```bash docker build -t configurator:testing path/to/configurator/Dockerfile -# ...or -make docker ``` -Keep in mind that all plugins included in the project are build in the `lib/` directory and copied from there. If you want to easily include your own external generator plugins, you can build it and copy the `lib.so` file to that location. Make sure that the `Generator` interface is implemented correct as described in the ["Creating Generator Plugins"](#creating-generator-plugins) or the plugin will not load. Additionally, the name string returned from the `GetName()` method is used for looking up the plugin after all plugins have been loaded by the main driver. +If you want to easily include your own external generator plugins, you can build it and copy the `lib.so` file to `lib/`. Make sure that the `Generator` interface is implemented correctly as described in the ["Creating Generator Plugins"](#creating-generator-plugins) or the plugin will not load (you should get an error that specifically says this). Additionally, the name string returned from the `GetName()` method is used for looking up the plugin with the `--target` flag by the main driver program. Alternatively, pull the latest existing image/container from the GitHub container repository. @@ -68,23 +80,27 @@ Alternatively, pull the latest existing image/container from the GitHub containe docker pull ghcr.io/openchami/configurator:latest ``` -Then, run the container similarly to the binary. +Then, run the Docker container similarly to running the binary. -``` -docker run ghcr.io/openchami/configurator:latest configurator generate --config config.yaml --target dnsmasq +```bash +export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs... +docker run ghcr.io/openchami/configurator:latest configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert configurator.pem ``` ### Creating Generator Plugins -The `configurator` uses generator plugins to define how config files are generated using a `Generator` interface. The interface is defined like so: +The `configurator` uses built-in and user-defined generators that implement the `Generator` interface to describe how config files should be generated. The interface is defined like so: ```go -type Files = map[string][]byte +// maps the file path to its contents +type FileMap = map[string][]byte + +// interface for generator plugins type Generator interface { GetName() string GetVersion() string GetDescription() string - Generate(config *configurator.Config, opts ...util.Option) (Files, error) + Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) } ``` @@ -93,23 +109,30 @@ A new plugin can be created by implementing the methods from interface and expor ```go package main -type MyGenerator struct {} +type MyGenerator struct { + PluginInfo map[string]any +} + +var pluginInfo map[string]any + +// this function is not a part of the `Generator` interface +func (g *MyGenerator) LoadFromFile() map[string]any{ /*...*/ } func (g *MyGenerator) GetName() string { // just an example...this can be done however you want - pluginInfo := LoadFromFile("path/to/plugin/info.json") - return pluginInfo["name"] + g.PluginInfo := LoadFromFile("path/to/plugin/info.json") + return g.PluginInfo["name"] } func (g *MyGenerator) GetVersion() string { - return "v1.0.0" + return g.PluginInfo["version"] // "v1.0.0" } func (g *MyGenerator) GetDescription() string { - return "This is an example plugin." + return g.PluginInfo["description"] // "This is an example plugin." } -func (g *MyGenerator) Generate(config *configurator.Config, opts ...util.Option) (generator.Files, error) { +func (g *MyGenerator) Generate(config *configurator.Config, opts ...util.Option) (generator.FileMap, error) { // do config generation stuff here... var ( params = generator.GetParams(opts...) @@ -121,7 +144,7 @@ func (g *MyGenerator) Generate(config *configurator.Config, opts ...util.Option) // ... blah, blah, blah, check error, format output, and so on... } - // apply the substitutions to Jinja template and return output as byte array + // apply the substitutions to Jinja template and return output as FileMap (i.e. path and it's contents) return generator.ApplyTemplate(path, generator.Mappings{ "plugin_name": g.GetName(), "plugin_version": g.GetVersion(), @@ -140,7 +163,7 @@ Finally, build the plugin and put it somewhere specified by `plugins` in your co go build -buildmode=plugin -o lib/mygenerator.so path/to/mygenerator.go ``` -Now your plugin should be available to use with the `configurator` main driver. If you get an error about not loading the correct symbol type, make sure that you generator function definitions match the `Generator` interface exactly. +Now your plugin should be available to use with the `configurator` main driver program. If you get an error about not loading the correct symbol type, make sure that your generator function definitions match the `Generator` interface entirely and that you don't have a partially implemented interface. ## Configuration @@ -150,46 +173,33 @@ Here is an example config file to start using configurator: server: # Server-related parameters when using as service host: 127.0.0.1 port: 3334 - jwks: # Set the JWKS uri to protect /generate route + jwks: # Set the JWKS uri for protected routes uri: "" retries: 5 -smd: . # SMD-related parameters - host: http://127.0.0.1 - port: 27779 +smd: # SMD-related parameters + host: http://127.0.0.1:27779 plugins: # path to plugin directories - "lib/" targets: # targets to call with --target flag - dnsmasq: + coredhcp: templates: - - templates/dnsmasq.jinja - warewulf: - templates: # files using Jinja templating - - templates/warewulf/vnfs/dhcpd-template.jinja - - templates/warewulf/vnfs/dnsmasq-template.jinja + - templates/coredhcp.j2 files: # files to be copied without templating - - templates/warewulf/defaults/provision.jinja - - templates/warewulf/defaults/node.jinja - - templates/warewulf/filesystem/examples/* - - templates/warewulf/vnfs/* - - templates/warewulf/bootstrap.jinja - - templates/warewulf/database.jinja - targets: # additional targets to run + - extra/nodes.conf + targets: # additional targets to run (does not run recursively) - dnsmasq ``` -The `server` section sets the properties for running the `configurator` tool as a service and is not required if you're only using the CLI. Also note that the `jwks-uri` parameter is only needs for protecting endpoints. If it is not set, then the API is entirely public. The `smd` section tells the `configurator` tool where to find SMD to pull state management data used by the internal client. The `templates` section is where the paths are mapped to each generator plugin by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). The `plugins` is a list of paths to load generator plugins. +The `server` section sets the properties for running the `configurator` tool as a service and is not required if you're only using the CLI. Also note that the `jwks.uri` parameter is only needed for protecting endpoints. If it is not set, then all API routes are entirely public. The `smd` section tells the `configurator` tool where to find the SMD service to pull state management data used internally by the client's generator. The `templates` section is where the paths are mapped to each generator by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). The `plugins` is a list of paths to search for and load external generator plugins. ## Running the Tests -The `configurator` project includes a collection of tests focused on verifying plugin behavior and generating files. The tests do not currently test fetching information from SMD (or whatever remote source). The tests can be ran with either of the following commands: +The `configurator` project includes a collection of tests focused on verifying plugin behavior and generating files. The tests do not include fetching information from any remote sources, can be ran with the following command: ```bash go test ./tests/generate_test.go --tags=all -# ...or alternatively with GNU make... -make test ``` - ## Known Issues - Adds a new `OAuthClient` with every token request diff --git a/cmd/config.go b/cmd/config.go index 1de4015..f56c34c 100644 --- a/cmd/config.go +++ b/cmd/config.go @@ -1,11 +1,10 @@ package cmd import ( - "fmt" - + "github.com/rs/zerolog/log" "github.com/spf13/cobra" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -20,10 +19,10 @@ var configCmd = &cobra.Command{ for _, path := range args { // check and make sure something doesn't exist first if exists, err := util.PathExists(path); exists || err != nil { - fmt.Printf("file or directory exists\n") + log.Error().Err(err).Msg("file or directory exists") continue } - configurator.SaveDefaultConfig(path) + config.SaveDefault(path) } }, } diff --git a/cmd/fetch.go b/cmd/fetch.go index c2ba4e5..0b105a7 100644 --- a/cmd/fetch.go +++ b/cmd/fetch.go @@ -24,16 +24,22 @@ var fetchCmd = &cobra.Command{ return } + // check if we actually have any targets to run + if len(targets) <= 0 { + log.Error().Msg("must specify a target") + os.Exit(1) + } + // check to see if an access token is available from env - if config.AccessToken == "" { + if conf.AccessToken == "" { // check if OCHAMI_ACCESS_TOKEN env var is set if no access token is provided and use that instead accessToken := os.Getenv("ACCESS_TOKEN") if accessToken != "" { - config.AccessToken = accessToken + conf.AccessToken = accessToken } else { // TODO: try and fetch token first if it is needed if verbose { - fmt.Printf("No token found. Attempting to generate config without one...\n") + log.Warn().Msg("No token found. Attempting to generate config without one...") } } } @@ -46,25 +52,23 @@ var fetchCmd = &cobra.Command{ for _, target := range targets { // make a request for each target - url := fmt.Sprintf("%s:%d/generate?target=%s", remoteHost, remotePort, target) + url := fmt.Sprintf("%s/generate?target=%s", remoteHost, target) res, body, err := util.MakeRequest(url, http.MethodGet, nil, headers) if err != nil { - log.Error().Err(err).Msg("failed to make request") + log.Error().Err(err).Msg("failed to fetch files") return } // handle getting other error codes other than a 200 if res != nil { - if res.StatusCode == http.StatusOK { - log.Info().Msgf("%s\n", string(body)) - } + // NOTE: the server responses are already marshaled to JSON + fmt.Print(string(body)) } } }, } func init() { - fetchCmd.Flags().StringVar(&remoteHost, "host", "", "set the remote configurator host") - fetchCmd.Flags().IntVar(&remotePort, "port", 3334, "set the remote configurator port") + fetchCmd.Flags().StringVar(&remoteHost, "host", "", "set the remote configurator host and port") fetchCmd.Flags().StringSliceVar(&targets, "target", nil, "set the target configs to make") fetchCmd.Flags().StringVarP(&outputPath, "output", "o", "", "set the output path for config targets") fetchCmd.Flags().StringVar(&accessToken, "access-token", "o", "set the output path for config targets") diff --git a/cmd/generate.go b/cmd/generate.go index c373b9b..44e4e5d 100644 --- a/cmd/generate.go +++ b/cmd/generate.go @@ -9,7 +9,8 @@ import ( "os" "path/filepath" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/generator" "github.com/OpenCHAMI/configurator/pkg/util" "github.com/rs/zerolog/log" @@ -28,48 +29,74 @@ var generateCmd = &cobra.Command{ Short: "Generate a config file from state management", Run: func(cmd *cobra.Command, args []string) { // make sure that we have a token present before trying to make request - if config.AccessToken == "" { + if conf.AccessToken == "" { // check if OCHAMI_ACCESS_TOKEN env var is set if no access token is provided and use that instead accessToken := os.Getenv("ACCESS_TOKEN") if accessToken != "" { - config.AccessToken = accessToken + conf.AccessToken = accessToken } else { // TODO: try and fetch token first if it is needed if verbose { - fmt.Printf("No token found. Attempting to generate config without one...\n") + log.Warn().Msg("No token found. Attempting to generate conf without one...\n") } } } // use cert path from cobra if empty - if config.CertPath == "" { - config.CertPath = cacertPath + if conf.CertPath == "" { + conf.CertPath = cacertPath } - // show config as JSON and generators if verbose + // show conf as JSON and generators if verbose if verbose { - b, err := json.MarshalIndent(config, "", " ") + b, err := json.MarshalIndent(conf, "", " ") if err != nil { - fmt.Printf("failed to marshal config: %v\n", err) + log.Error().Err(err).Msg("failed to marshal config") } + // print the config file as JSON fmt.Printf("%v\n", string(b)) } // run all of the target recursively until completion if provided if len(targets) > 0 { - RunTargets(&config, args, targets...) + RunTargets(&conf, args, targets...) } else { if pluginPath == "" { - fmt.Printf("no plugin path specified") + log.Error().Msg("no plugin path specified") return } + // load the templates to use + templates := map[string]generator.Template{} + for _, path := range templatePaths { + template := generator.Template{} + template.LoadFromFile(path) + if !template.IsEmpty() { + templates[path] = template + } + } + + params := generator.Params{ + Templates: templates, + } + + // set the client options + if conf.AccessToken != "" { + params.ClientOpts = append(params.ClientOpts, client.WithAccessToken(conf.AccessToken)) + } + if conf.CertPath != "" { + params.ClientOpts = append(params.ClientOpts, client.WithCertPoolFile(conf.CertPath)) + } + // run generator.Generate() with just plugin path and templates provided - generator.Generate(&config, generator.Params{ - PluginPath: pluginPath, - TemplatePaths: templatePaths, - }) + outputBytes, err := generator.Generate(pluginPath, params) + if err != nil { + log.Error().Err(err).Msg("failed to generate files") + } + // if we have more than one target and output is set, create configs in directory + outputMap := generator.ConvertContentsToString(outputBytes) + writeOutput(outputBytes, len(targets), len(outputMap)) } }, } @@ -80,103 +107,98 @@ var generateCmd = &cobra.Command{ // child targets if it is the same as the parent. // // NOTE: This may be changed in the future how this is done. -func RunTargets(config *configurator.Config, args []string, targets ...string) { +func RunTargets(conf *config.Config, args []string, targets ...string) { // generate config with each supplied target for _, target := range targets { - outputBytes, err := generator.GenerateWithTarget(config, generator.Params{ - Args: args, - PluginPath: pluginPath, - Target: target, - Verbose: verbose, - }) + outputBytes, err := generator.GenerateWithTarget(conf, target) if err != nil { - log.Error().Err(err).Msg("failed to generate config") + log.Error().Err(err).Str("target", target).Msg("failed to generate config") os.Exit(1) } // if we have more than one target and output is set, create configs in directory - var ( - outputMap = generator.ConvertContentsToString(outputBytes) - targetCount = len(targets) - templateCount = len(outputMap) - ) - if outputPath == "" { - // write only to stdout by default - if len(outputMap) == 1 { - for _, contents := range outputMap { - fmt.Printf("%s\n", string(contents)) - } - } else { - for path, contents := range outputMap { - fmt.Printf("-- file: %s, size: %d B\n%s\n", path, len(contents), string(contents)) - } - } - } else if outputPath != "" && targetCount == 1 && templateCount == 1 { - // write just a single file using provided name - for _, contents := range outputBytes { - err := os.WriteFile(outputPath, contents, 0o644) - if err != nil { - log.Error().Err(err).Msg("failed to write config to file") - os.Exit(1) - } - log.Info().Msgf("wrote file to '%s'\n", outputPath) - } - } else if outputPath != "" && targetCount > 1 && useCompression { - // write multiple files to archive, compress, then save to output path - out, err := os.Create(fmt.Sprintf("%s.tar.gz", outputPath)) - if err != nil { - log.Error().Err(err).Msg("failed to write archive") - os.Exit(1) + outputMap := generator.ConvertContentsToString(outputBytes) + writeOutput(outputBytes, len(targets), len(outputMap)) + + // remove any targets that are the same as current to prevent infinite loop + nextTargets := util.CopyIf(conf.Targets[target].RunTargets, func(nextTarget string) bool { + return nextTarget != target + }) + + // ...then, run any other targets that the current target has + RunTargets(conf, args, nextTargets...) + } +} + +func writeOutput(outputBytes generator.FileMap, targetCount int, templateCount int) { + outputMap := generator.ConvertContentsToString(outputBytes) + if outputPath == "" { + // write only to stdout by default + if len(outputMap) == 1 { + for _, contents := range outputMap { + fmt.Printf("%s\n", string(contents)) } - files := make([]string, len(outputBytes)) - i := 0 - for path := range outputBytes { - files[i] = path - i++ + } else { + for path, contents := range outputMap { + fmt.Printf("-- file: %s, size: %d B\n%s\n", path, len(contents), string(contents)) } - err = util.CreateArchive(files, out) + } + } else if outputPath != "" && targetCount == 1 && templateCount == 1 { + // write just a single file using provided name + for _, contents := range outputBytes { + err := os.WriteFile(outputPath, contents, 0o644) if err != nil { - log.Error().Err(err).Msg("failed to create archive") + log.Error().Err(err).Str("path", outputPath).Msg("failed to write config file") os.Exit(1) } + log.Info().Msgf("wrote file to '%s'\n", outputPath) + } + } else if outputPath != "" && targetCount > 1 && useCompression { + // write multiple files to archive, compress, then save to output path + out, err := os.Create(fmt.Sprintf("%s.tar.gz", outputPath)) + if err != nil { + log.Error().Err(err).Str("path", outputPath).Msg("failed to write archive") + os.Exit(1) + } + files := make([]string, len(outputBytes)) + i := 0 + for path := range outputBytes { + files[i] = path + i++ + } + err = util.CreateArchive(files, out) + if err != nil { + log.Error().Err(err).Str("path", outputPath).Msg("failed to create archive") + os.Exit(1) + } - } else if outputPath != "" && targetCount > 1 || templateCount > 1 { - // write multiple files in directory using template name - err := os.MkdirAll(filepath.Clean(outputPath), 0o755) + } else if outputPath != "" && targetCount > 1 || templateCount > 1 { + // write multiple files in directory using template name + err := os.MkdirAll(filepath.Clean(outputPath), 0o755) + if err != nil { + log.Error().Err(err).Str("path", filepath.Clean(outputPath)).Msg("failed to make output directory") + os.Exit(1) + } + for path, contents := range outputBytes { + filename := filepath.Base(path) + cleanPath := fmt.Sprintf("%s/%s", filepath.Clean(outputPath), filename) + err := os.WriteFile(cleanPath, contents, 0o755) if err != nil { - log.Error().Err(err).Msg("failed to make output directory") + log.Error().Err(err).Str("path", path).Msg("failed to write config to file") os.Exit(1) } - for path, contents := range outputBytes { - filename := filepath.Base(path) - cleanPath := fmt.Sprintf("%s/%s", filepath.Clean(outputPath), filename) - err := os.WriteFile(cleanPath, contents, 0o755) - if err != nil { - log.Error().Err(err).Msg("failed to write config to file") - os.Exit(1) - } - log.Info().Msgf("wrote file to '%s'\n", cleanPath) - } + log.Info().Msgf("wrote file to '%s'\n", cleanPath) } - - // remove any targets that are the same as current to prevent infinite loop - nextTargets := util.CopyIf(config.Targets[target].RunTargets, func(nextTarget string) bool { - return nextTarget != target - }) - - // ...then, run any other targets that the current target has - RunTargets(config, args, nextTargets...) } } func init() { - generateCmd.Flags().StringSliceVar(&targets, "target", []string{}, "set the targets to run pre-defined config") + generateCmd.Flags().StringSliceVar(&targets, "target", []string{}, "set the targets to run pre-defined conf") generateCmd.Flags().StringSliceVar(&templatePaths, "template", []string{}, "set the paths for the Jinja 2 templates to use") generateCmd.Flags().StringVar(&pluginPath, "plugin", "", "set the generator plugin path") - generateCmd.Flags().StringVarP(&outputPath, "output", "o", "", "set the output path for config targets") + generateCmd.Flags().StringVarP(&outputPath, "output", "o", "", "set the output path for conf targets") generateCmd.Flags().IntVar(&tokenFetchRetries, "fetch-retries", 5, "set the number of retries to fetch an access token") generateCmd.Flags().StringVar(&remoteHost, "host", "http://localhost", "set the remote host") - generateCmd.Flags().IntVar(&remotePort, "port", 80, "set the remote port") generateCmd.Flags().BoolVar(&useCompression, "compress", false, "set whether to archive and compress multiple file outputs") // requires either 'target' by itself or 'plugin' and 'templates' together diff --git a/cmd/root.go b/cmd/root.go index ab190da..1d1997c 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -4,14 +4,14 @@ import ( "fmt" "os" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" "github.com/rs/zerolog/log" "github.com/spf13/cobra" ) var ( - config configurator.Config + conf config.Config configPath string cacertPath string verbose bool @@ -19,12 +19,11 @@ var ( outputPath string accessToken string remoteHost string - remotePort int ) var rootCmd = &cobra.Command{ Use: "configurator", - Short: "Tool for building common config files", + Short: "Dynamically generate files defined by generators", Run: func(cmd *cobra.Command, args []string) { if len(args) == 0 { cmd.Help() @@ -52,10 +51,10 @@ func initConfig() { if configPath != "" { exists, err := util.PathExists(configPath) if err != nil { - fmt.Printf("failed to load config") + log.Error().Err(err).Str("path", configPath).Msg("failed to load config") os.Exit(1) } else if exists { - config = configurator.LoadConfig(configPath) + conf = config.Load(configPath) } else { // show error and exit since a path was specified log.Error().Str("path", configPath).Msg("config file not found") @@ -64,7 +63,7 @@ func initConfig() { } else { // set to the default value and create a new one configPath = "./config.yaml" - config = configurator.NewConfig() + conf = config.New() } // @@ -74,6 +73,6 @@ func initConfig() { // set the JWKS url if we find the CONFIGURATOR_JWKS_URL environment variable jwksUrl := os.Getenv("CONFIGURATOR_JWKS_URL") if jwksUrl != "" { - config.Server.Jwks.Uri = jwksUrl + conf.Server.Jwks.Uri = jwksUrl } } diff --git a/cmd/serve.go b/cmd/serve.go index 647482b..48e3629 100644 --- a/cmd/serve.go +++ b/cmd/serve.go @@ -10,8 +10,8 @@ import ( "net/http" "os" - "github.com/OpenCHAMI/configurator/pkg/generator" "github.com/OpenCHAMI/configurator/pkg/server" + "github.com/rs/zerolog/log" "github.com/spf13/cobra" ) @@ -20,66 +20,48 @@ var serveCmd = &cobra.Command{ Short: "Start configurator as a server and listen for requests", Run: func(cmd *cobra.Command, args []string) { // make sure that we have a token present before trying to make request - if config.AccessToken == "" { - // TODO: make request to check if request will need token - - // check if OCHAMI_ACCESS_TOKEN env var is set if no access token is provided and use that instead + if conf.AccessToken == "" { + // check if ACCESS_TOKEN env var is set if no access token is provided and use that instead accessToken := os.Getenv("ACCESS_TOKEN") if accessToken != "" { - config.AccessToken = accessToken + conf.AccessToken = accessToken } else { - // TODO: try and fetch token first if it is needed if verbose { - fmt.Printf("No token found. Attempting to generate config without one...\n") + log.Warn().Msg("No token found. Continuing without one...\n") } } } // show config as JSON and generators if verbose if verbose { - b, err := json.MarshalIndent(config, "", " ") + b, err := json.MarshalIndent(conf, "", "\t") if err != nil { - fmt.Printf("failed to marshal config: %v\n", err) + log.Error().Err(err).Msg("failed to marshal config") + os.Exit(1) } fmt.Printf("%v\n", string(b)) } - // set up the routes and start the serve - server := server.Server{ - Config: &config, - Server: &http.Server{ - Addr: fmt.Sprintf("%s:%d", config.Server.Host, config.Server.Port), - }, - Jwks: server.Jwks{ - Uri: config.Server.Jwks.Uri, - Retries: config.Server.Jwks.Retries, - }, - GeneratorParams: generator.Params{ - Args: args, - // PluginPath: pluginPath, - // Target: target, // NOTE: targets are set via HTTP requests (ex: curl http://configurator:3334/generate?target=dnsmasq) - Verbose: verbose, - }, - } - // start listening with the server - err := server.Serve(cacertPath) + var ( + s *server.Server = server.New(&conf) + err error = s.Serve() + ) if errors.Is(err, http.ErrServerClosed) { if verbose { - fmt.Printf("Server closed.") + log.Info().Msg("server closed") } } else if err != nil { - fmt.Errorf("failed to start server: %v", err) + log.Error().Err(err).Msg("failed to start server") os.Exit(1) } }, } func init() { - serveCmd.Flags().StringVar(&config.Server.Host, "host", config.Server.Host, "set the server host") - serveCmd.Flags().IntVar(&config.Server.Port, "port", config.Server.Port, "set the server port") + serveCmd.Flags().StringVar(&conf.Server.Host, "host", conf.Server.Host, "set the server host and port") // serveCmd.Flags().StringVar(&pluginPath, "plugin", "", "set the generator plugins directory path") - serveCmd.Flags().StringVar(&config.Server.Jwks.Uri, "jwks-uri", config.Server.Jwks.Uri, "set the JWKS url to fetch public key") - serveCmd.Flags().IntVar(&config.Server.Jwks.Retries, "jwks-fetch-retries", config.Server.Jwks.Retries, "set the JWKS fetch retry count") + serveCmd.Flags().StringVar(&conf.Server.Jwks.Uri, "jwks-uri", conf.Server.Jwks.Uri, "set the JWKS url to fetch public key") + serveCmd.Flags().IntVar(&conf.Server.Jwks.Retries, "jwks-fetch-retries", conf.Server.Jwks.Retries, "set the JWKS fetch retry count") rootCmd.AddCommand(serveCmd) } diff --git a/pkg/client/client.go b/pkg/client/client.go new file mode 100644 index 0000000..a4b7454 --- /dev/null +++ b/pkg/client/client.go @@ -0,0 +1,66 @@ +package client + +import ( + "crypto/tls" + "crypto/x509" + "net" + "net/http" + "os" + "time" +) + +type Option func(*Params) +type Params struct { + Host string `yaml:"host"` + AccessToken string `yaml:"access-token"` + Transport *http.Transport +} + +func ToParams(opts ...Option) *Params { + params := &Params{} + for _, opt := range opts { + opt(params) + } + return params +} + +func WithHost(host string) Option { + return func(c *Params) { + c.Host = host + } +} + +func WithAccessToken(token string) Option { + return func(c *Params) { + c.AccessToken = token + } +} + +func WithCertPool(certPool *x509.CertPool) Option { + return func(c *Params) { + c.Transport = &http.Transport{ + TLSClientConfig: &tls.Config{ + RootCAs: certPool, + InsecureSkipVerify: true, + }, + DisableKeepAlives: true, + Dial: (&net.Dialer{ + Timeout: 120 * time.Second, + KeepAlive: 120 * time.Second, + }).Dial, + TLSHandshakeTimeout: 120 * time.Second, + ResponseHeaderTimeout: 120 * time.Second, + } + } +} + +// FIXME: Need to check for errors when reading from a file +func WithCertPoolFile(certPath string) Option { + if certPath == "" { + return func(sc *Params) {} + } + cacert, _ := os.ReadFile(certPath) + certPool := x509.NewCertPool() + certPool.AppendCertsFromPEM(cacert) + return WithCertPool(certPool) +} diff --git a/pkg/client.go b/pkg/client/smd.go similarity index 52% rename from pkg/client.go rename to pkg/client/smd.go index ddbfab0..eb80ce8 100644 --- a/pkg/client.go +++ b/pkg/client/smd.go @@ -1,128 +1,64 @@ -package configurator +package client import ( "bytes" - "crypto/tls" - "crypto/x509" "encoding/json" "fmt" "io" - "net" "net/http" - "os" - "time" - "github.com/OpenCHAMI/configurator/pkg/util" + configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/rs/zerolog/log" ) -type ClientOption func(*SmdClient) - // An struct that's meant to extend functionality of the base HTTP client by // adding commonly made requests to SMD. The implemented functions are can be // used in generator plugins to fetch data when it is needed to substitute // values for the Jinja templates used. type SmdClient struct { - http.Client `json:"-"` + http.Client `json:"-" yaml:"-"` Host string `yaml:"host"` Port int `yaml:"port"` AccessToken string `yaml:"access-token"` } -// Constructor function that allows supplying ClientOption arguments to set +// Constructor function that allows supplying Option arguments to set // things like the host, port, access token, etc. -func NewSmdClient(opts ...ClientOption) SmdClient { - client := SmdClient{} - for _, opt := range opts { - opt(&client) - } - return client -} - -func WithHost(host string) ClientOption { - return func(c *SmdClient) { - c.Host = host - } -} - -func WithPort(port int) ClientOption { - return func(c *SmdClient) { - c.Port = port - } -} - -func WithAccessToken(token string) ClientOption { - return func(c *SmdClient) { - c.AccessToken = token - } -} - -func WithCertPool(certPool *x509.CertPool) ClientOption { - return func(c *SmdClient) { - c.Client.Transport = &http.Transport{ - TLSClientConfig: &tls.Config{ - RootCAs: certPool, - InsecureSkipVerify: true, - }, - DisableKeepAlives: true, - Dial: (&net.Dialer{ - Timeout: 120 * time.Second, - KeepAlive: 120 * time.Second, - }).Dial, - TLSHandshakeTimeout: 120 * time.Second, - ResponseHeaderTimeout: 120 * time.Second, +func NewSmdClient(opts ...Option) SmdClient { + var ( + params = ToParams(opts...) + client = SmdClient{ + Host: params.Host, + AccessToken: params.AccessToken, } - } -} - -// FIXME: Need to check for errors when reading from a file -func WithCertPoolFile(certPath string) ClientOption { - if certPath == "" { - return func(sc *SmdClient) {} - } - cacert, _ := os.ReadFile(certPath) - certPool := x509.NewCertPool() - certPool.AppendCertsFromPEM(cacert) - return WithCertPool(certPool) -} - -func WithVerbosity() util.Option { - return func(p util.Params) { - p["verbose"] = true - } -} + ) -// Create a set of params with all default values. -func NewParams() util.Params { - return util.Params{ - "verbose": false, - } + return client } // Fetch the ethernet interfaces from SMD service using its API. An access token may be required if the SMD // service SMD_JWKS_URL envirnoment variable is set. -func (client *SmdClient) FetchEthernetInterfaces(opts ...util.Option) ([]EthernetInterface, error) { +func (client *SmdClient) FetchEthernetInterfaces(verbose bool) ([]configurator.EthernetInterface, error) { var ( - params = util.ToDict(opts...) - verbose = util.Get[bool](params, "verbose") - eths = []EthernetInterface{} + eths = []configurator.EthernetInterface{} + bytes []byte + err error ) // make request to SMD endpoint - b, err := client.makeRequest("/Inventory/EthernetInterfaces") + bytes, err = client.makeRequest("/Inventory/EthernetInterfaces") if err != nil { return nil, fmt.Errorf("failed to read HTTP response: %v", err) } // unmarshal response body JSON and extract in object - err = json.Unmarshal(b, ðs) + err = json.Unmarshal(bytes, ðs) if err != nil { return nil, fmt.Errorf("failed to unmarshal response: %v", err) } // print what we got if verbose is set - if verbose != nil { - if *verbose { - fmt.Printf("Ethernet Interfaces: %v\n", string(b)) - } + if verbose { + log.Info().Str("ethernet_interfaces", string(bytes)).Msg("found interfaces") } return eths, nil @@ -130,68 +66,68 @@ func (client *SmdClient) FetchEthernetInterfaces(opts ...util.Option) ([]Etherne // Fetch the components from SMD using its API. An access token may be required if the SMD // service SMD_JWKS_URL envirnoment variable is set. -func (client *SmdClient) FetchComponents(opts ...util.Option) ([]Component, error) { +func (client *SmdClient) FetchComponents(verbose bool) ([]configurator.Component, error) { var ( - params = util.ToDict(opts...) - verbose = util.Get[bool](params, "verbose") - comps = []Component{} + comps = []configurator.Component{} + bytes []byte + err error ) // make request to SMD endpoint - b, err := client.makeRequest("/State/Components") + bytes, err = client.makeRequest("/State/Components") if err != nil { return nil, fmt.Errorf("failed to make HTTP request: %v", err) } // make sure our response is actually JSON - if !json.Valid(b) { - return nil, fmt.Errorf("expected valid JSON response: %v", string(b)) + if !json.Valid(bytes) { + return nil, fmt.Errorf("expected valid JSON response: %v", string(bytes)) } // unmarshal response body JSON and extract in object var tmp map[string]any - err = json.Unmarshal(b, &tmp) + err = json.Unmarshal(bytes, &tmp) if err != nil { return nil, fmt.Errorf("failed to unmarshal response: %v", err) } - b, err = json.Marshal(tmp["RedfishEndpoints"].([]any)) + bytes, err = json.Marshal(tmp["RedfishEndpoints"].([]any)) if err != nil { return nil, fmt.Errorf("failed to marshal JSON: %v", err) } - err = json.Unmarshal(b, &comps) + err = json.Unmarshal(bytes, &comps) if err != nil { return nil, fmt.Errorf("failed to unmarshal response: %v", err) } // print what we got if verbose is set - if verbose != nil { - if *verbose { - fmt.Printf("Components: %v\n", string(b)) - } + if verbose { + log.Info().Str("components", string(bytes)).Msg("found components") } return comps, nil } -func (client *SmdClient) FetchRedfishEndpoints(opts ...util.Option) ([]RedfishEndpoint, error) { +// TODO: improve implementation of this function +func (client *SmdClient) FetchRedfishEndpoints(verbose bool) ([]configurator.RedfishEndpoint, error) { var ( - params = util.ToDict(opts...) - verbose = util.Get[bool](params, "verbose") - eps = []RedfishEndpoint{} + eps = []configurator.RedfishEndpoint{} + tmp map[string]any ) + // make initial request to get JSON with 'RedfishEndpoints' as property b, err := client.makeRequest("/Inventory/RedfishEndpoints") if err != nil { return nil, fmt.Errorf("failed to make HTTP resquest: %v", err) } + // make sure response is in JSON if !json.Valid(b) { return nil, fmt.Errorf("expected valid JSON response: %v", string(b)) } - var tmp map[string]any err = json.Unmarshal(b, &tmp) if err != nil { return nil, fmt.Errorf("failed to unmarshal response: %v", err) } + // marshal RedfishEndpoint JSON back to configurator.RedfishEndpoint b, err = json.Marshal(tmp["RedfishEndpoints"].([]any)) if err != nil { return nil, fmt.Errorf("failed to marshal JSON: %v", err) @@ -201,10 +137,9 @@ func (client *SmdClient) FetchRedfishEndpoints(opts ...util.Option) ([]RedfishEn return nil, fmt.Errorf("failed to unmarshal response: %v", err) } - if verbose != nil { - if *verbose { - fmt.Printf("Redfish endpoints: %v\n", string(b)) - } + // show the final result + if verbose { + log.Info().Str("redfish_endpoints", string(b)).Msg("found redfish endpoints") } return eps, nil @@ -216,7 +151,7 @@ func (client *SmdClient) makeRequest(endpoint string) ([]byte, error) { } // fetch DHCP related information from SMD's endpoint: - url := fmt.Sprintf("%s:%d/hsm/v2%s", client.Host, client.Port, endpoint) + url := fmt.Sprintf("%s/hsm/v2%s", client.Host, endpoint) req, err := http.NewRequest(http.MethodGet, url, bytes.NewBuffer([]byte{})) if err != nil { return nil, fmt.Errorf("failed to create new HTTP request: %v", err) diff --git a/pkg/config.go b/pkg/config.go deleted file mode 100644 index 89db7ab..0000000 --- a/pkg/config.go +++ /dev/null @@ -1,129 +0,0 @@ -package configurator - -import ( - "log" - "os" - "path/filepath" - - "gopkg.in/yaml.v2" -) - -type Options struct{} - -type Target struct { - PluginPath string `yaml:"plugin,omitempty"` - TemplatePaths []string `yaml:"templates,omitempty"` - FilePaths []string `yaml:"files,omitempty"` - RunTargets []string `yaml:"targets,omitempty"` -} - -type Jwks struct { - Uri string `yaml:"uri"` - Retries int `yaml:"retries,omitempty"` -} - -type Server struct { - Host string `yaml:"host"` - Port int `yaml:"port"` - Jwks Jwks `yaml:"jwks,omitempty"` -} - -type Config struct { - Version string `yaml:"version,omitempty"` - Server Server `yaml:"server,omitempty"` - SmdClient SmdClient `yaml:"smd,omitempty"` - AccessToken string `yaml:"access-token,omitempty"` - Targets map[string]Target `yaml:"targets,omitempty"` - PluginDirs []string `yaml:"plugins,omitempty"` - CertPath string `yaml:"cacert,omitempty"` - Options Options `yaml:"options,omitempty"` -} - -// Creates a new config with default parameters. -func NewConfig() Config { - return Config{ - Version: "", - SmdClient: SmdClient{ - Host: "http://127.0.0.1", - Port: 27779, - }, - Targets: map[string]Target{ - "dnsmasq": Target{ - PluginPath: "", - TemplatePaths: []string{}, - }, - "conman": Target{ - PluginPath: "", - TemplatePaths: []string{}, - }, - "warewulf": Target{ - PluginPath: "", - TemplatePaths: []string{ - "templates/warewulf/defaults/node.jinja", - "templates/warewulf/defaults/provision.jinja", - }, - }, - }, - - PluginDirs: []string{}, - Server: Server{ - Host: "127.0.0.1", - Port: 3334, - Jwks: Jwks{ - Uri: "", - Retries: 5, - }, - }, - Options: Options{}, - } -} - -func LoadConfig(path string) Config { - var c Config = NewConfig() - file, err := os.ReadFile(path) - if err != nil { - log.Printf("failed to read config file: %v\n", err) - return c - } - err = yaml.Unmarshal(file, &c) - if err != nil { - log.Fatalf("failed to unmarshal config: %v\n", err) - return c - } - return c -} - -func (config *Config) SaveConfig(path string) { - path = filepath.Clean(path) - if path == "" || path == "." { - path = "config.yaml" - } - data, err := yaml.Marshal(config) - if err != nil { - log.Printf("failed to marshal config: %v\n", err) - return - } - err = os.WriteFile(path, data, os.ModePerm) - if err != nil { - log.Printf("failed to write default config file: %v\n", err) - return - } -} - -func SaveDefaultConfig(path string) { - path = filepath.Clean(path) - if path == "" || path == "." { - path = "config.yaml" - } - var c = NewConfig() - data, err := yaml.Marshal(c) - if err != nil { - log.Printf("failed to marshal config: %v\n", err) - return - } - err = os.WriteFile(path, data, os.ModePerm) - if err != nil { - log.Printf("failed to write default config file: %v\n", err) - return - } -} diff --git a/pkg/config/config.go b/pkg/config/config.go new file mode 100644 index 0000000..c20797b --- /dev/null +++ b/pkg/config/config.go @@ -0,0 +1,100 @@ +package config + +import ( + "os" + "path/filepath" + + "github.com/rs/zerolog/log" + + configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "gopkg.in/yaml.v2" +) + +type Jwks struct { + Uri string `yaml:"uri"` + Retries int `yaml:"retries,omitempty"` +} + +type Server struct { + Host string `yaml:"host"` + Port int `yaml:"port"` + Jwks Jwks `yaml:"jwks,omitempty"` +} + +type Config struct { + Version string `yaml:"version,omitempty"` + Server Server `yaml:"server,omitempty"` + SmdClient client.SmdClient `yaml:"smd,omitempty"` + AccessToken string `yaml:"access-token,omitempty"` + Targets map[string]configurator.Target `yaml:"targets,omitempty"` + PluginDirs []string `yaml:"plugins,omitempty"` + CertPath string `yaml:"cacert,omitempty"` +} + +// Creates a new config with default parameters. +func New() Config { + return Config{ + Version: "", + SmdClient: client.SmdClient{Host: "http://127.0.0.1:27779"}, + Targets: map[string]configurator.Target{}, + PluginDirs: []string{}, + Server: Server{ + Host: "127.0.0.1:3334", + Jwks: Jwks{ + Uri: "", + Retries: 5, + }, + }, + } +} + +func Load(path string) Config { + var c Config = New() + file, err := os.ReadFile(path) + if err != nil { + log.Error().Err(err).Msg("failed to read config file") + return c + } + err = yaml.Unmarshal(file, &c) + if err != nil { + log.Error().Err(err).Msg("failed to unmarshal config") + return c + } + return c +} + +func (config *Config) Save(path string) { + path = filepath.Clean(path) + if path == "" || path == "." { + path = "config.yaml" + } + data, err := yaml.Marshal(config) + if err != nil { + log.Error().Err(err).Msg("failed to marshal config") + return + } + err = os.WriteFile(path, data, 0o644) + if err != nil { + log.Error().Err(err).Msg("failed to write default config file") + return + } +} + +func SaveDefault(path string) { + path = filepath.Clean(path) + if path == "" || path == "." { + path = "config.yaml" + } + var c = New() + data, err := yaml.Marshal(c) + if err != nil { + log.Error().Err(err).Msg("failed to marshal config") + return + } + err = os.WriteFile(path, data, 0o644) + if err != nil { + log.Error().Err(err).Msg("failed to write default config file") + return + } +} diff --git a/pkg/configurator.go b/pkg/configurator.go index 68dae99..7a19a36 100644 --- a/pkg/configurator.go +++ b/pkg/configurator.go @@ -2,6 +2,13 @@ package configurator import "encoding/json" +type Target struct { + Plugin string `yaml:"plugin,omitempty"` // Set the plugin or it's path + TemplatePaths []string `yaml:"templates,omitempty"` // Set the template paths + FilePaths []string `yaml:"files,omitempty"` // Set the file paths + RunTargets []string `yaml:"targets,omitempty"` // Set additional targets to run +} + type IPAddr struct { IpAddress string `json:"IPAddress"` Network string `json:"Network"` diff --git a/pkg/generator/conman.go b/pkg/generator/conman.go index 53358b5..ccadaa6 100644 --- a/pkg/generator/conman.go +++ b/pkg/generator/conman.go @@ -4,6 +4,8 @@ import ( "fmt" configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -21,32 +23,20 @@ func (g *Conman) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s'.", g.GetName()) } -func (g *Conman) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *Conman) Generate(config *config.Config, params Params) (FileMap, error) { var ( - params = GetParams(opts...) - client = GetClient(params) - targetKey = params["target"].(string) // required param - target = config.Targets[targetKey] - eps []configurator.RedfishEndpoint = nil - err error = nil - // serverOpts = "" - // globalOpts = "" - consoles = "" + smdClient = client.NewSmdClient(params.ClientOpts...) + eps = []configurator.RedfishEndpoint{} + err error = nil + consoles = "" ) // fetch required data from SMD to create config - if client != nil { - eps, err = client.FetchRedfishEndpoints(opts...) - if err != nil { - return nil, fmt.Errorf("failed to fetch redfish endpoints with client: %v", err) - } + eps, err = smdClient.FetchRedfishEndpoints(params.Verbose) + if err != nil { + return nil, fmt.Errorf("failed to fetch redfish endpoints with client: %v", err) } - // add any additional conman or server opts - // if extraOpts, ok := params["opts"].(map[string]any); ok { - - // } - // format output to write to config file consoles = "# ========== DYNAMICALLY GENERATED BY OPENCHAMI CONFIGURATOR ==========\n" for _, ep := range eps { @@ -55,12 +45,12 @@ func (g *Conman) Generate(config *configurator.Config, opts ...util.Option) (Fil consoles += "# =====================================================================" // apply template substitutions and return output as byte array - return ApplyTemplateFromFiles(Mappings{ + return ApplyTemplates(Mappings{ "plugin_name": g.GetName(), "plugin_version": g.GetVersion(), "plugin_description": g.GetDescription(), "server_opts": "", "global_opts": "", "consoles": consoles, - }, target.TemplatePaths...) + }, params.Templates) } diff --git a/pkg/generator/coredhcp.go b/pkg/generator/coredhcp.go index 817790c..0a256e9 100644 --- a/pkg/generator/coredhcp.go +++ b/pkg/generator/coredhcp.go @@ -1,12 +1,9 @@ -//go:build coredhcp || plugins -// +build coredhcp plugins - package generator import ( "fmt" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -24,6 +21,6 @@ func (g *CoreDhcp) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s' to generate config files. (WIP)", g.GetName()) } -func (g *CoreDhcp) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *CoreDhcp) Generate(config *config.Config, params Params) (FileMap, error) { return nil, fmt.Errorf("plugin does not implement generation function") } diff --git a/pkg/generator/dhcpd.go b/pkg/generator/dhcpd.go index f2ce520..cc32a48 100644 --- a/pkg/generator/dhcpd.go +++ b/pkg/generator/dhcpd.go @@ -4,6 +4,8 @@ import ( "fmt" configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -21,23 +23,18 @@ func (g *DHCPd) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s'.", g.GetName()) } -func (g *DHCPd) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *DHCPd) Generate(config *config.Config, params Params) (FileMap, error) { var ( - params = GetParams(opts...) - client = GetClient(params) - targetKey = params["target"].(string) - target = config.Targets[targetKey] - compute_nodes = "" - eths []configurator.EthernetInterface = nil - err error = nil + smdClient = client.NewSmdClient(params.ClientOpts...) + eths = []configurator.EthernetInterface{} + computeNodes = "" + err error = nil ) // - if client != nil { - eths, err = client.FetchEthernetInterfaces(opts...) - if err != nil { - return nil, fmt.Errorf("failed to fetch ethernet interfaces with client: %w", err) - } + eths, err = smdClient.FetchEthernetInterfaces(params.Verbose) + if err != nil { + return nil, fmt.Errorf("failed to fetch ethernet interfaces with client: %w", err) } // check if we have the required params first @@ -49,25 +46,19 @@ func (g *DHCPd) Generate(config *configurator.Config, opts ...util.Option) (File } // format output to write to config file - compute_nodes = "# ========== DYNAMICALLY GENERATED BY OPENCHAMI CONFIGURATOR ==========\n" + computeNodes = "# ========== DYNAMICALLY GENERATED BY OPENCHAMI CONFIGURATOR ==========\n" for _, eth := range eths { if len(eth.IpAddresses) == 0 { continue } - compute_nodes += fmt.Sprintf("host %s { hardware ethernet %s; fixed-address %s} ", eth.ComponentId, eth.MacAddress, eth.IpAddresses[0]) - } - compute_nodes += "# =====================================================================" - - if verbose, ok := params["verbose"].(bool); ok { - if verbose { - fmt.Printf("") - } + computeNodes += fmt.Sprintf("host %s { hardware ethernet %s; fixed-address %s} ", eth.ComponentId, eth.MacAddress, eth.IpAddresses[0]) } - return ApplyTemplateFromFiles(Mappings{ + computeNodes += "# =====================================================================" + return ApplyTemplates(Mappings{ "plugin_name": g.GetName(), "plugin_version": g.GetVersion(), "plugin_description": g.GetDescription(), - "compute_nodes": compute_nodes, + "compute_nodes": computeNodes, "node_entries": "", - }, target.TemplatePaths...) + }, params.Templates) } diff --git a/pkg/generator/dnsmasq.go b/pkg/generator/dnsmasq.go index ab5e648..83bf1d6 100644 --- a/pkg/generator/dnsmasq.go +++ b/pkg/generator/dnsmasq.go @@ -2,9 +2,10 @@ package generator import ( "fmt" - "strings" configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -22,7 +23,7 @@ func (g *DNSMasq) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s'.", g.GetName()) } -func (g *DNSMasq) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *DNSMasq) Generate(config *config.Config, params Params) (FileMap, error) { // make sure we have a valid config first if config == nil { return nil, fmt.Errorf("invalid config (config is nil)") @@ -30,20 +31,15 @@ func (g *DNSMasq) Generate(config *configurator.Config, opts ...util.Option) (Fi // set all the defaults for variables var ( - params = GetParams(opts...) - client = GetClient(params) - targetKey = params["target"].(string) // required param - target = config.Targets[targetKey] - eths []configurator.EthernetInterface = nil - err error = nil + smdClient = client.NewSmdClient(params.ClientOpts...) + eths = []configurator.EthernetInterface{} + err error = nil ) // if we have a client, try making the request for the ethernet interfaces - if client != nil { - eths, err = client.FetchEthernetInterfaces(opts...) - if err != nil { - return nil, fmt.Errorf("failed to fetch ethernet interfaces with client: %v", err) - } + eths, err = smdClient.FetchEthernetInterfaces(params.Verbose) + if err != nil { + return nil, fmt.Errorf("failed to fetch ethernet interfaces with client: %v", err) } // check if we have the required params first @@ -54,13 +50,6 @@ func (g *DNSMasq) Generate(config *configurator.Config, opts ...util.Option) (Fi return nil, fmt.Errorf("no ethernet interfaces found") } - // print message if verbose param found - if verbose, ok := params["verbose"].(bool); ok { - if verbose { - fmt.Printf("template: \n%s\nethernet interfaces found: %v\n", strings.Join(target.TemplatePaths, "\n\t"), len(eths)) - } - } - // format output to write to config file output := "# ========== DYNAMICALLY GENERATED BY OPENCHAMI CONFIGURATOR ==========\n" for _, eth := range eths { @@ -73,10 +62,10 @@ func (g *DNSMasq) Generate(config *configurator.Config, opts ...util.Option) (Fi output += "# =====================================================================" // apply template substitutions and return output as byte array - return ApplyTemplateFromFiles(Mappings{ + return ApplyTemplates(Mappings{ "plugin_name": g.GetName(), "plugin_version": g.GetVersion(), "plugin_description": g.GetDescription(), "dhcp-hosts": output, - }, target.TemplatePaths...) + }, params.Templates) } diff --git a/pkg/generator/example.go b/pkg/generator/example.go index b8b5c1d..f18abe4 100644 --- a/pkg/generator/example.go +++ b/pkg/generator/example.go @@ -1,13 +1,9 @@ -//go:build example || plugins -// +build example plugins - package generator import ( "fmt" - configurator "github.com/OpenCHAMI/configurator/pkg" - "github.com/OpenCHAMI/configurator/pkg/generator" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -27,11 +23,9 @@ func (g *Example) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s'.", g.GetName()) } -func (g *Example) Generate(config *configurator.Config, opts ...util.Option) (generator.FileMap, error) { +func (g *Example) Generate(config *config.Config, params Params) (FileMap, error) { g.Message = ` This is an example generator plugin. See the file in 'internal/generator/plugins/example/example.go' on information about constructing plugins and plugin requirements.` - return generator.FileMap{"example": []byte(g.Message)}, nil + return FileMap{"example": []byte(g.Message)}, nil } - -var Generator Example diff --git a/pkg/generator/generator.go b/pkg/generator/generator.go index e705e16..dfc39d7 100644 --- a/pkg/generator/generator.go +++ b/pkg/generator/generator.go @@ -1,7 +1,6 @@ package generator import ( - "bytes" "fmt" "io/fs" "os" @@ -9,9 +8,9 @@ import ( "plugin" configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" - "github.com/nikolalohinski/gonja/v2" - "github.com/nikolalohinski/gonja/v2/exec" "github.com/rs/zerolog/log" ) @@ -19,7 +18,6 @@ type ( Mappings map[string]any FileMap map[string][]byte FileList [][]byte - Template []byte // Generator interface used to define how files are created. Plugins can // be created entirely independent of the main driver program. @@ -27,17 +25,7 @@ type ( GetName() string GetVersion() string GetDescription() string - Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) - } - - // Params defined and used by the "generate" subcommand. - Params struct { - Args []string - TemplatePaths []string - PluginPath string - Target string - Client *configurator.SmdClient - Verbose bool + Generate(config *config.Config, params Params) (FileMap, error) } ) @@ -47,8 +35,7 @@ func createDefaultGenerators() map[string]Generator { var ( generatorMap = map[string]Generator{} generators = []Generator{ - &Conman{}, &DHCPd{}, &DNSMasq{}, &Hostfile{}, - &Powerman{}, &Syslog{}, &Warewulf{}, + &Conman{}, &DHCPd{}, &DNSMasq{}, &Warewulf{}, &Example{}, &CoreDhcp{}, } ) for _, g := range generators { @@ -129,11 +116,11 @@ func LoadPlugin(path string) (Generator, error) { // // Returns a map of generators. Each generator can be accessed by the name // returned by the generator.GetName() implemented. -func LoadPlugins(dirpath string, opts ...util.Option) (map[string]Generator, error) { +func LoadPlugins(dirpath string, opts ...Option) (map[string]Generator, error) { // check if verbose option is supplied var ( generators = make(map[string]Generator) - params = util.ToDict(opts...) + params = ToParams(opts...) ) // @@ -143,6 +130,11 @@ func LoadPlugins(dirpath string, opts ...util.Option) (map[string]Generator, err return nil } + // only try loading if file has .so extension + if filepath.Ext(path) != ".so" { + return nil + } + // load the generator plugin from current path gen, err := LoadPlugin(path) if err != nil { @@ -150,8 +142,8 @@ func LoadPlugins(dirpath string, opts ...util.Option) (map[string]Generator, err } // show the plugins found if verbose flag is set - if params.GetVerbose() { - fmt.Printf("-- found plugin '%s'\n", gen.GetName()) + if params.Verbose { + log.Info().Str("plugin_name", gen.GetName()).Msg("found plugin") } // map each generator plugin by name for lookup @@ -163,239 +155,34 @@ func LoadPlugins(dirpath string, opts ...util.Option) (map[string]Generator, err return nil, fmt.Errorf("failed to walk directory: %w", err) } - // items, _ := os.ReadDir(dirpath) - // for _, item := range items { - // if item.IsDir() { - // subitems, _ := os.ReadDir(item.Name()) - // for _, subitem := range subitems { - // if !subitem.IsDir() { - // gen, err := LoadPlugin(subitem.Name()) - // if err != nil { - // fmt.Printf("failed to load generator in directory '%s': %v\n", item.Name(), err) - // continue - // } - // if verbose, ok := params["verbose"].(bool); ok { - // if verbose { - // fmt.Printf("-- found plugin '%s'\n", item.Name()) - // } - // } - // gens[gen.GetName()] = gen - // } - // } - // } else { - // gen, err := LoadPlugin(dirpath + item.Name()) - // if err != nil { - // fmt.Printf("failed to load plugin: %v\n", err) - // continue - // } - // if verbose, ok := params["verbose"].(bool); ok { - // if verbose { - // fmt.Printf("-- found plugin '%s'\n", dirpath+item.Name()) - // } - // } - // gens[gen.GetName()] = gen - // } - // } - return generators, nil } -func LoadTemplate(path string) (Template, error) { - // skip loading template if path is a directory with no error - if isDir, err := util.IsDirectory(path); err == nil && isDir { - return nil, nil - } else if err != nil { - return nil, fmt.Errorf("failed to test if template path is directory: %w", err) - } - - // try and read the contents of the file - // NOTE: we don't care if this is actually a Jinja template - // or not...at least for now. - return os.ReadFile(path) -} - -func LoadTemplates(paths []string, opts ...util.Option) (map[string]Template, error) { - var ( - templates = make(map[string]Template) - params = util.ToDict(opts...) - ) - - for _, path := range paths { - err := filepath.Walk(path, func(path string, info fs.FileInfo, err error) error { - // skip trying to load generator plugin if directory or error - if info.IsDir() || err != nil { - return nil - } - - // load the contents of the template - template, err := LoadTemplate(path) - if err != nil { - return fmt.Errorf("failed to load generator in directory '%s': %w", path, err) - } - - // show the templates loaded if verbose flag is set - if params.GetVerbose() { - fmt.Printf("-- loaded tempalte '%s'\n", path) - } - - // map each template by the path it was loaded from - templates[path] = template - return nil - }) - - if err != nil { - return nil, fmt.Errorf("failed to walk directory: %w", err) - } - } - - return templates, nil -} - -// Option to specify "target" in parameter map. This is used to set which generator -// to use to generate a config file. -func WithTarget(target string) util.Option { - return func(p util.Params) { - if p != nil { - p["target"] = target - } - } -} - -// Option to specify "type" in parameter map. This is not currently used. -func WithType(_type string) util.Option { - return func(p util.Params) { - if p != nil { - p["type"] = _type - } - } -} - -// Option to the plugin to load -func WithPlugin(path string) util.Option { - return func(p util.Params) { - if p != nil { - plugin, err := LoadPlugin(path) - if err != nil { - return - } - p["plugin"] = plugin - } - } -} - -func WithTemplates(paths []string) util.Option { - return func(p util.Params) { - if p != nil { - templates, err := LoadTemplates(paths) - if err != nil { - - } - p["templates"] = templates - } - } -} - -// Option to a specific client to include in implementing plugin generator.Generate(). -// -// NOTE: This may be changed to pass some kind of client interface as an argument in -// the future instead. -func WithClient(client configurator.SmdClient) util.Option { - return func(p util.Params) { - p["client"] = client - } -} - -// Helper function to get client in generator.Generate() plugin implementations. -func GetClient(params util.Params) *configurator.SmdClient { - return util.Get[configurator.SmdClient](params, "client") -} - -// Helper function to get the target in generator.Generate() plugin implementations. -func GetTarget(config *configurator.Config, key string) configurator.Target { - return config.Targets[key] -} - -// Helper function to load all options set with With*() into parameter map. -func GetParams(opts ...util.Option) util.Params { - params := util.Params{} - for _, opt := range opts { - opt(params) - } - return params -} - -// Wrapper function to slightly abstract away some of the nuances with using gonja -// into a single function call. This function is *mostly* for convenience and -// simplication. If no paths are supplied, then no templates will be applied and -// there will be no output. +// Generate() is the main function to generate a collection of files and returns them as a map. +// This function only expects a path to a plugin and paths to a collection of templates to +// be used. This function will only load the plugin on-demand and fetch resources as needed. // -// The "FileList" returns a slice of byte arrays in the same order as the argument -// list supplied, but with the Jinja templating applied. -func ApplyTemplates(mappings Mappings, contents ...[]byte) (FileList, error) { - var ( - data = exec.NewContext(mappings) - outputs = FileList{} - ) - - for _, b := range contents { - // load jinja template from file - t, err := gonja.FromBytes(b) - if err != nil { - return nil, fmt.Errorf("failed to read template from file: %w", err) - } - - // execute/render jinja template - b := bytes.Buffer{} - if err = t.Execute(&b, data); err != nil { - return nil, fmt.Errorf("failed to execute: %w", err) - } - outputs = append(outputs, b.Bytes()) - } - - return outputs, nil -} - -// Wrapper function similiar to "ApplyTemplates" but takes file paths as arguments. -// This function will load templates from a file instead of using file contents. -func ApplyTemplateFromFiles(mappings Mappings, paths ...string) (FileMap, error) { +// This function requires that a target and plugin path be set at minimum. +func Generate(plugin string, params Params) (FileMap, error) { var ( - data = exec.NewContext(mappings) - outputs = FileMap{} + gen Generator + ok bool + err error ) - for _, path := range paths { - // load jinja template from file - t, err := gonja.FromFile(path) + // check if generator is built-in first before loading external plugin + log.Debug().Any("generators", DefaultGenerators).Msg("available generators") + gen, ok = DefaultGenerators[plugin] + if !ok { + // only load the plugin needed for this target if we don't find default + log.Error().Str("plugin", plugin).Msg("could not find target in default generators") + gen, err = LoadPlugin(plugin) if err != nil { - return nil, fmt.Errorf("failed to read template from file: %w", err) + return nil, fmt.Errorf("failed to load plugin from file: %v", err) } - - // execute/render jinja template - b := bytes.Buffer{} - if err = t.Execute(&b, data); err != nil { - return nil, fmt.Errorf("failed to execute: %w", err) - } - outputs[path] = b.Bytes() } - return outputs, nil -} - -// Generate() is the main function to generate a collection of files and returns them as a map. -// This function only expects a path to a plugin and paths to a collection of templates to -// be used. This function will only load the plugin on-demand and fetch resources as needed. -func Generate(config *configurator.Config, params Params) (FileMap, error) { - var ( - gen Generator - client = configurator.NewSmdClient() - ) - - return gen.Generate( - config, - WithPlugin(params.PluginPath), - WithTemplates(params.TemplatePaths), - WithClient(client), - ) + return gen.Generate(nil, params) } // Main function to generate a collection of files as a map with the path as the key and @@ -407,59 +194,66 @@ func Generate(config *configurator.Config, params Params) (FileMap, error) { // It is also call when running the configurator as a service with the "/generate" route. // // TODO: Separate loading plugins so we can load them once when running as a service. -func GenerateWithTarget(config *configurator.Config, params Params) (FileMap, error) { +func GenerateWithTarget(config *config.Config, target string) (FileMap, error) { // load generator plugins to generate configs or to print var ( - client configurator.SmdClient - target configurator.Target - generator Generator - err error - ok bool + opts []client.Option + targetInfo configurator.Target + generator Generator + params Params + err error + ok bool ) - // check if we have a client from params first and create new one if not - if params.Client == nil { - client = configurator.NewSmdClient( - configurator.WithHost(config.SmdClient.Host), - configurator.WithPort(config.SmdClient.Port), - configurator.WithAccessToken(config.AccessToken), - configurator.WithCertPoolFile(config.CertPath), - ) - } else { - client = *params.Client - } - // check if a target is supplied - if len(params.Args) == 0 && params.Target == "" { + if target == "" { return nil, fmt.Errorf("must specify a target") } // load target information from config - target, ok = config.Targets[params.Target] + targetInfo, ok = config.Targets[target] if !ok { - return nil, fmt.Errorf("target not found in config") + log.Warn().Str("target", target).Msg("target not found in config") } - // if plugin path specified from CLI, use that instead - if params.PluginPath != "" { - target.PluginPath = params.PluginPath + // if no plugin supplied in config target, then using the target supplied + if targetInfo.Plugin == "" { + targetInfo.Plugin = target } // check if generator is built-in first before loading - generator, ok = DefaultGenerators[params.Target] + generator, ok = DefaultGenerators[target] if !ok { // only load the plugin needed for this target if we don't find default - log.Error().Msg("did not find target in default generators") - generator, err = LoadPlugin(target.PluginPath) + log.Warn().Str("target", target).Msg("could not find target in default generators") + generator, err = LoadPlugin(targetInfo.Plugin) if err != nil { - return nil, fmt.Errorf("failed to load plugin: %w", err) + return nil, fmt.Errorf("failed to load plugin: %v", err) } } + // prepare params to pass into generator + params.Templates = map[string]Template{} + for _, templatePath := range targetInfo.TemplatePaths { + template := Template{} + template.LoadFromFile(templatePath) + params.Templates[templatePath] = template + } + + // set the client options + if config.AccessToken != "" { + params.ClientOpts = append(opts, client.WithAccessToken(config.AccessToken)) + } + if config.CertPath != "" { + params.ClientOpts = append(opts, client.WithCertPoolFile(config.CertPath)) + } + + // load files that are not to be copied + params.Files, err = LoadFiles(targetInfo.FilePaths...) + if err != nil { + return nil, fmt.Errorf("failed to load files to copy: %v", err) + } + // run the generator plugin from target passed - return generator.Generate( - config, - WithTarget(generator.GetName()), - WithClient(client), - ) + return generator.Generate(config, params) } diff --git a/pkg/generator/hostfile.go b/pkg/generator/hostfile.go index e998714..7ce26c8 100644 --- a/pkg/generator/hostfile.go +++ b/pkg/generator/hostfile.go @@ -3,7 +3,7 @@ package generator import ( "fmt" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -21,6 +21,6 @@ func (g *Hostfile) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s'.", g.GetName()) } -func (g *Hostfile) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *Hostfile) Generate(config *config.Config, opts ...Option) (FileMap, error) { return nil, fmt.Errorf("plugin does not implement generation function") } diff --git a/pkg/generator/params.go b/pkg/generator/params.go new file mode 100644 index 0000000..e54420a --- /dev/null +++ b/pkg/generator/params.go @@ -0,0 +1,43 @@ +package generator + +import ( + configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" +) + +type ( + // Params used by the generator + Params struct { + Templates map[string]Template + Files map[string][]byte + ClientOpts []client.Option + Verbose bool + } + Option func(Params) +) + +func ToParams(opts ...Option) Params { + params := Params{} + for _, opt := range opts { + opt(params) + } + return params +} + +func WithClientOpts(opts ...client.Option) Option { + return func(p Params) { + p.ClientOpts = opts + } +} + +func WithTemplates(templates map[string]Template) Option { + return func(p Params) { + p.Templates = templates + } +} + +// Helper function to get the target in generator.Generate() plugin implementations. +func GetTarget(config *config.Config, key string) configurator.Target { + return config.Targets[key] +} diff --git a/pkg/generator/powerman.go b/pkg/generator/powerman.go index 36be6fc..08745e5 100644 --- a/pkg/generator/powerman.go +++ b/pkg/generator/powerman.go @@ -3,7 +3,7 @@ package generator import ( "fmt" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -21,6 +21,6 @@ func (g *Powerman) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s'.", g.GetName()) } -func (g *Powerman) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *Powerman) Generate(config *config.Config, opts ...Option) (FileMap, error) { return nil, fmt.Errorf("plugin does not implement generation function") } diff --git a/pkg/generator/syslog.go b/pkg/generator/syslog.go index 463f727..67b28cf 100644 --- a/pkg/generator/syslog.go +++ b/pkg/generator/syslog.go @@ -3,7 +3,7 @@ package generator import ( "fmt" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" ) @@ -21,6 +21,6 @@ func (g *Syslog) GetDescription() string { return fmt.Sprintf("Configurator generator plugin for '%s'.", g.GetName()) } -func (g *Syslog) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *Syslog) Generate(config *config.Config, opts ...Option) (FileMap, error) { return nil, fmt.Errorf("plugin does not implement generation function") } diff --git a/pkg/generator/templates.go b/pkg/generator/templates.go new file mode 100644 index 0000000..6d4ae5d --- /dev/null +++ b/pkg/generator/templates.go @@ -0,0 +1,95 @@ +package generator + +import ( + "bytes" + "fmt" + "os" + + "github.com/OpenCHAMI/configurator/pkg/util" + "github.com/nikolalohinski/gonja/v2" + "github.com/nikolalohinski/gonja/v2/exec" +) + +type Template struct { + Contents []byte `json:"contents"` +} + +func (t *Template) LoadFromFile(path string) error { + // skip loading template if path is a directory with no error + if isDir, err := util.IsDirectory(path); err == nil && isDir { + return nil + } else if err != nil { + return fmt.Errorf("failed to test if template path is directory: %w", err) + } + + // try and read the contents of the file + // NOTE: we don't care if this is actually a Jinja template + // or not...at least for now. + contents, err := os.ReadFile(path) + if err != nil { + return fmt.Errorf("failed to read file: %v", err) + } + t.Contents = contents + return nil +} + +func (t *Template) IsEmpty() bool { + return len(t.Contents) <= 0 +} + +// Wrapper function to slightly abstract away some of the nuances with using gonja +// into a single function call. This function is *mostly* for convenience and +// simplication. If no paths are supplied, then no templates will be applied and +// there will be no output. +// +// The "FileList" returns a slice of byte arrays in the same order as the argument +// list supplied, but with the Jinja templating applied. +func ApplyTemplates(mappings Mappings, templates map[string]Template) (FileMap, error) { + var ( + data = exec.NewContext(mappings) + outputs = FileMap{} + ) + + for path, template := range templates { + // load jinja template from file + t, err := gonja.FromBytes(template.Contents) + if err != nil { + return nil, fmt.Errorf("failed to read template from file: %w", err) + } + + // execute/render jinja template + b := bytes.Buffer{} + if err = t.Execute(&b, data); err != nil { + return nil, fmt.Errorf("failed to execute: %w", err) + } + outputs[path] = b.Bytes() + } + + return outputs, nil +} + +// Wrapper function similiar to "ApplyTemplates" but takes file paths as arguments. +// This function will load templates from a file instead of using file contents. +func ApplyTemplateFromFiles(mappings Mappings, paths ...string) (FileMap, error) { + var ( + data = exec.NewContext(mappings) + outputs = FileMap{} + ) + + for _, path := range paths { + // load jinja template from file + t, err := gonja.FromFile(path) + if err != nil { + return nil, fmt.Errorf("failed to read template from file: %w", err) + } + + // execute/render jinja template + b := bytes.Buffer{} + if err = t.Execute(&b, data); err != nil { + return nil, fmt.Errorf("failed to execute: %w", err) + } + outputs[path] = b.Bytes() + } + + return outputs, nil +} diff --git a/pkg/generator/warewulf.go b/pkg/generator/warewulf.go index 49b0c2c..bdfbda1 100644 --- a/pkg/generator/warewulf.go +++ b/pkg/generator/warewulf.go @@ -5,8 +5,10 @@ import ( "maps" "strings" - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/util" + "github.com/rs/zerolog/log" ) type Warewulf struct{} @@ -23,22 +25,16 @@ func (g *Warewulf) GetDescription() string { return "Configurator generator plugin for 'warewulf' config files." } -func (g *Warewulf) Generate(config *configurator.Config, opts ...util.Option) (FileMap, error) { +func (g *Warewulf) Generate(config *config.Config, params Params) (FileMap, error) { var ( - params = GetParams(opts...) - client = GetClient(params) - targetKey = params["target"].(string) - target = config.Targets[targetKey] - outputs = make(FileMap, len(target.FilePaths)+len(target.TemplatePaths)) + smdClient = client.NewSmdClient(params.ClientOpts...) + outputs = make(FileMap, len(params.Templates)) + nodeEntries = "" + paths = []string{} ) - // check if our client is included and is valid - if client == nil { - return nil, fmt.Errorf("invalid client (client is nil)") - } - // if we have a client, try making the request for the ethernet interfaces - eths, err := client.FetchEthernetInterfaces(opts...) + eths, err := smdClient.FetchEthernetInterfaces(params.Verbose) if err != nil { return nil, fmt.Errorf("failed to fetch ethernet interfaces with client: %v", err) } @@ -51,15 +47,8 @@ func (g *Warewulf) Generate(config *configurator.Config, opts ...util.Option) (F return nil, fmt.Errorf("no ethernet interfaces found") } - // print message if verbose param found - if verbose, ok := params["verbose"].(bool); ok { - if verbose { - fmt.Printf("template: \n%s\n ethernet interfaces found: %v\n", strings.Join(target.TemplatePaths, "\n\t"), len(eths)) - } - } - // fetch redfish endpoints and handle errors - eps, err := client.FetchRedfishEndpoints(opts...) + eps, err := smdClient.FetchRedfishEndpoints(params.Verbose) if err != nil { return nil, fmt.Errorf("failed to fetch redfish endpoints: %v", err) } @@ -67,33 +56,23 @@ func (g *Warewulf) Generate(config *configurator.Config, opts ...util.Option) (F return nil, fmt.Errorf("no redfish endpoints found") } - // format output for template substitution - nodeEntries := "" - - // load files and templates and copy to outputs - files, err := LoadFiles(target.FilePaths...) - if err != nil { - return nil, fmt.Errorf("failed to load files: %v", err) - } - templates, err := ApplyTemplateFromFiles(Mappings{ + templates, err := ApplyTemplates(Mappings{ "node_entries": nodeEntries, - }, target.TemplatePaths...) + }, params.Templates) if err != nil { return nil, fmt.Errorf("failed to load templates: %v", err) } - maps.Copy(outputs, files) + maps.Copy(outputs, params.Files) maps.Copy(outputs, templates) // print message if verbose param is found - if verbose, ok := params["verbose"].(bool); ok { - if verbose { - fmt.Printf("templates and files loaded: \n") - for path, _ := range outputs { - fmt.Printf("\t%s", path) - } + if params.Verbose { + for path, _ := range outputs { + paths = append(paths, path) } } + log.Info().Str("paths", strings.Join(paths, ":")).Msg("templates and files loaded: \n") return outputs, err } diff --git a/pkg/server/server.go b/pkg/server/server.go index 9afc7db..7da26d7 100644 --- a/pkg/server/server.go +++ b/pkg/server/server.go @@ -4,22 +4,21 @@ package server import ( - "crypto/tls" - "crypto/x509" "encoding/json" "fmt" - "net" + "io" "net/http" "os" "time" configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/client" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/generator" "github.com/OpenCHAMI/jwtauth/v5" "github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5/middleware" "github.com/rs/zerolog" - "github.com/sirupsen/logrus" openchami_authenticator "github.com/openchami/chi-middleware/auth" openchami_logger "github.com/openchami/chi-middleware/log" @@ -36,62 +35,48 @@ type Jwks struct { } type Server struct { *http.Server - Config *configurator.Config + Config *config.Config Jwks Jwks `yaml:"jwks"` GeneratorParams generator.Params TokenAuth *jwtauth.JWTAuth + Targets map[string]Target +} + +type Target struct { + Name string `json:"name"` + PluginPath string `json:"plugin"` + Templates []generator.Template `json:"templates"` } // Constructor to make a new server instance with an optional config. -func New(config *configurator.Config) *Server { +func New(conf *config.Config) *Server { // create default config if none supplied - if config == nil { - c := configurator.NewConfig() - config = &c - } - // return based on config values - return &Server{ - Config: config, - Server: &http.Server{ - Addr: fmt.Sprintf("%s:%d", config.Server.Host, config.Server.Port), - }, + if conf == nil { + c := config.New() + conf = &c + } + newServer := &Server{ + Config: conf, + Server: &http.Server{Addr: conf.Server.Host}, Jwks: Jwks{ - Uri: config.Server.Jwks.Uri, - Retries: config.Server.Jwks.Retries, + Uri: conf.Server.Jwks.Uri, + Retries: conf.Server.Jwks.Retries, }, } + // load templates for server from config + newServer.loadTargets() + log.Debug().Any("targets", newServer.Targets).Msg("new server targets") + return newServer } // Main function to start up configurator as a service. -func (s *Server) Serve(cacertPath string) error { - // create client just for the server to use to fetch data from SMD - client := &configurator.SmdClient{ - Host: s.Config.SmdClient.Host, - Port: s.Config.SmdClient.Port, - } - - // add cert to client if `--cacert` flag is passed - if cacertPath != "" { - cacert, _ := os.ReadFile(cacertPath) - certPool := x509.NewCertPool() - certPool.AppendCertsFromPEM(cacert) - client.Transport = &http.Transport{ - TLSClientConfig: &tls.Config{ - RootCAs: certPool, - InsecureSkipVerify: true, - }, - DisableKeepAlives: true, - Dial: (&net.Dialer{ - Timeout: 120 * time.Second, - KeepAlive: 120 * time.Second, - }).Dial, - TLSHandshakeTimeout: 120 * time.Second, - ResponseHeaderTimeout: 120 * time.Second, - } - } +func (s *Server) Serve() error { + // Setup logger + zerolog.TimeFieldFormat = zerolog.TimeFormatUnix + logger := log.Output(zerolog.ConsoleWriter{Out: os.Stderr}) // set the server address with config values - s.Server.Addr = fmt.Sprintf("%s:%d", s.Config.Server.Host, s.Config.Server.Port) + s.Server.Addr = s.Config.Server.Host // fetch JWKS public key from authorization server if s.Config.Server.Jwks.Uri != "" && tokenAuth == nil { @@ -99,16 +84,18 @@ func (s *Server) Serve(cacertPath string) error { var err error tokenAuth, err = configurator.FetchPublicKeyFromURL(s.Config.Server.Jwks.Uri) if err != nil { - logrus.Errorf("failed to fetch JWKS: %v", err) + log.Error().Err(err).Msgf("failed to fetch JWKS") continue } break } } - // Setup logger - zerolog.TimeFieldFormat = zerolog.TimeFormatUnix - logger := log.Output(zerolog.ConsoleWriter{Out: os.Stderr}) + // create client with opts to use to fetch data from SMD + opts := []client.Option{ + client.WithAccessToken(s.Config.AccessToken), + client.WithCertPoolFile(s.Config.CertPath), + } // create new go-chi router with its routes router := chi.NewRouter() @@ -127,13 +114,13 @@ func (s *Server) Serve(cacertPath string) error { ) // protected routes if using auth - r.HandleFunc("/generate", s.Generate(client)) - r.HandleFunc("/templates", s.ManageTemplates) + r.HandleFunc("/generate", s.Generate(opts...)) + r.Post("/targets", s.createTarget) }) } else { // public routes without auth - router.HandleFunc("/generate", s.Generate(client)) - router.HandleFunc("/templates", s.ManageTemplates) + router.HandleFunc("/generate", s.Generate(opts...)) + router.Post("/targets", s.createTarget) } // always available public routes go here (none at the moment) @@ -150,22 +137,39 @@ func (s *Server) Close() { // This is the corresponding service function to generate templated files, that // works similarly to the CLI variant. This function takes similiar arguments as // query parameters that are included in the HTTP request URL. -func (s *Server) Generate(client *configurator.SmdClient) func(w http.ResponseWriter, r *http.Request) { - +func (s *Server) Generate(opts ...client.Option) func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) { // get all of the expect query URL params and validate - s.GeneratorParams.Target = r.URL.Query().Get("target") - s.GeneratorParams.Client = client - if s.GeneratorParams.Target == "" { - writeErrorResponse(w, "must specify a target") + var ( + targetParam string = r.URL.Query().Get("target") + target *Target = s.getTarget(targetParam) + outputs generator.FileMap + err error + ) + s.GeneratorParams = parseGeneratorParams(r, opts...) + if targetParam == "" { + err = writeErrorResponse(w, "must specify a target") + log.Error().Err(err).Msg("failed to parse generator params") return } - // generate a new config file from supplied params - outputs, err := generator.GenerateWithTarget(s.Config, s.GeneratorParams) - if err != nil { - writeErrorResponse(w, "failed to generate file: %v", err) - return + // try to generate with target supplied by client first + if target != nil { + log.Debug().Any("target", target).Msg("target for Generate()") + outputs, err = generator.Generate(target.PluginPath, s.GeneratorParams) + if err != nil { + log.Error().Err(err).Msg("failed to generate file") + return + } + } else { + // try and generate a new config file from supplied params + log.Debug().Str("target", targetParam).Msg("target for GenerateWithTarget()") + outputs, err = generator.GenerateWithTarget(s.Config, targetParam) + if err != nil { + writeErrorResponse(w, "failed to generate file") + log.Error().Err(err).Msgf("failed to generate file with target '%s'", target) + return + } } // marshal output to JSON then send response to client @@ -173,32 +177,131 @@ func (s *Server) Generate(client *configurator.SmdClient) func(w http.ResponseWr b, err := json.Marshal(tmp) if err != nil { writeErrorResponse(w, "failed to marshal output: %v", err) + log.Error().Err(err).Msg("failed to marshal output") return } _, err = w.Write(b) if err != nil { writeErrorResponse(w, "failed to write response: %v", err) + log.Error().Err(err).Msg("failed to write response") return } } } -// Incomplete WIP function for managing templates remotely. There is currently no -// internal API to do this yet. +func (s *Server) loadTargets() { + // make sure the map is initialized first + if s.Targets == nil { + s.Targets = make(map[string]Target) + } + // add default generator targets + for name, _ := range generator.DefaultGenerators { + serverTarget := Target{ + Name: name, + PluginPath: name, + } + s.Targets[name] = serverTarget + } + // add targets from config to server (overwrites default targets) + for name, target := range s.Config.Targets { + serverTarget := Target{ + Name: name, + } + // only overwrite plugin path if it's set + if target.Plugin != "" { + serverTarget.PluginPath = target.Plugin + } else { + serverTarget.PluginPath = name + } + // add templates using template paths from config + for _, templatePath := range target.TemplatePaths { + template := generator.Template{} + template.LoadFromFile(templatePath) + serverTarget.Templates = append(serverTarget.Templates, template) + } + s.Targets[name] = serverTarget + } +} + +// Create a new target with name, generator, templates, and files. +// +// Example: +// +// curl -X POST /target?name=test&plugin=dnsmasq // // TODO: need to implement template managing API first in "internal/generator/templates" or something -func (s *Server) ManageTemplates(w http.ResponseWriter, r *http.Request) { - _, err := w.Write([]byte("this is not implemented yet")) +func (s *Server) createTarget(w http.ResponseWriter, r *http.Request) { + var ( + target = Target{} + bytes []byte + err error + ) + if r == nil { + err = writeErrorResponse(w, "request is invalid") + log.Error().Err(err).Msg("request == nil") + return + } + + bytes, err = io.ReadAll(r.Body) if err != nil { - writeErrorResponse(w, "failed to write response: %v", err) + writeErrorResponse(w, "failed to read response body: %v", err) + log.Error().Err(err).Msg("failed to read response body") + return + } + defer r.Body.Close() + + err = json.Unmarshal(bytes, &target) + if err != nil { + writeErrorResponse(w, "failed to unmarshal target: %v", err) + log.Error().Err(err).Msg("failed to unmarshal target") + return + } + + // make sure a plugin and at least one template is supplied + if target.Name == "" { + err = writeErrorResponse(w, "target name is required") + log.Error().Err(err).Msg("set target as a URL query parameter") + return + } + if target.PluginPath == "" { + err = writeErrorResponse(w, "generator name is required") + log.Error().Err(err).Msg("must supply a generator name") return } + if len(target.Templates) <= 0 { + writeErrorResponse(w, "requires at least one template") + log.Error().Err(err).Msg("must supply at least one template") + return + } + + s.Targets[target.Name] = target + +} + +func (s *Server) getTarget(name string) *Target { + t, ok := s.Targets[name] + if ok { + return &t + } + return nil } // Wrapper function to simplify writting error message responses. This function // is only intended to be used with the service and nothing else. func writeErrorResponse(w http.ResponseWriter, format string, a ...any) error { errmsg := fmt.Sprintf(format, a...) - w.Write([]byte(errmsg)) + bytes, _ := json.Marshal(map[string]any{ + "level": "error", + "time": time.Now().Unix(), + "message": errmsg, + }) + http.Error(w, string(bytes), http.StatusInternalServerError) return fmt.Errorf(errmsg) } + +func parseGeneratorParams(r *http.Request, opts ...client.Option) generator.Params { + var params = generator.Params{ + ClientOpts: opts, + } + return params +} diff --git a/tests/generate_local.hurl b/tests/generate_local.hurl new file mode 100644 index 0000000..d198e46 --- /dev/null +++ b/tests/generate_local.hurl @@ -0,0 +1,22 @@ +## +## Run these tests after starting server with `configurator serve...` +## + +# Generate a `example` config with default plugin and template +GET http://127.0.0.1:3334/generate?target=example +HTTP 200 + +# Create a new target using the API +POST http://127.0.0.1:3334/targets +{ + "name": "test", + "plugin": "example", + "templates": [{ + "contents": "This is an example template used with the example plugin." + }] +} +HTTP 200 + +# Test the new target just add from POST above +GET http://127.0.0.1:3334/generate?target=example +HTTP 200 \ No newline at end of file diff --git a/tests/generate_test.go b/tests/generate_test.go index 39ba06e..21ff978 100644 --- a/tests/generate_test.go +++ b/tests/generate_test.go @@ -3,17 +3,26 @@ package tests import ( "encoding/json" "fmt" + "log" "net/http" "os" "os/exec" + "path/filepath" "testing" configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/generator" "github.com/OpenCHAMI/configurator/pkg/server" "github.com/OpenCHAMI/configurator/pkg/util" ) +var ( + workDir string + replaceDir string + err error +) + // A valid test generator that implements the `Generator` interface. type TestGenerator struct{} @@ -22,98 +31,96 @@ func (g *TestGenerator) GetVersion() string { return "v1.0.0" } func (g *TestGenerator) GetDescription() string { return "This is a plugin created for running tests." } -func (g *TestGenerator) Generate(config *configurator.Config, opts ...util.Option) (generator.FileMap, error) { +func (g *TestGenerator) Generate(config *config.Config, params generator.Params) (generator.FileMap, error) { // Jinja 2 template file - files := [][]byte{ - []byte(` + files := map[string]generator.Template{ + "test1": generator.Template{ + Contents: []byte(` Name: {{plugin_name}} Version: {{plugin_version}} Description: {{plugin_description}} This is the first test template file. - `), - []byte(` + `), + }, + "test2": generator.Template{ + Contents: []byte(` This is another testing Jinja 2 template file using {{plugin_name}}. - `), + `), + }, } // apply Jinja templates to file - fileList, err := generator.ApplyTemplates(generator.Mappings{ + fileMap, err := generator.ApplyTemplates(generator.Mappings{ "plugin_name": g.GetName(), "plugin_version": g.GetVersion(), "plugin_description": g.GetDescription(), - }, files...) + }, files) if err != nil { return nil, fmt.Errorf("failed to apply templates: %v", err) } - // make sure we're able to receive certain arguments when passed - params := generator.GetParams(opts...) - if len(params) <= 0 { - return nil, fmt.Errorf("expect at least one params, but found none") - } - // make sure we have a valid config we can access if config == nil { return nil, fmt.Errorf("invalid config (config is nil)") } - // make sure we're able to get a valid client as well - client := generator.GetClient(params) - if client == nil { - return nil, fmt.Errorf("invalid client (client is nil)") - } - // TODO: make sure we can get a target // make sure we have the same number of files in file list - if len(files) != len(fileList) { - return nil, fmt.Errorf("file list output count is not the same as the input") - } - - // convert file list to file map - fileMap := make(generator.FileMap, len(fileList)) - for i, contents := range fileList { - fileMap[fmt.Sprintf("t-%d.txt", i)] = contents + var ( + fileInputCount = len(files) + fileOutputCount = len(fileMap) + ) + if fileInputCount != fileOutputCount { + return nil, fmt.Errorf("file output count (%d) is not the same as the input (%d)", fileOutputCount, fileInputCount) } return fileMap, nil } +func init() { + workDir, err = os.Getwd() + if err != nil { + log.Fatalf("failed to get working directory: %v", err) + } + replaceDir = fmt.Sprintf("%s", filepath.Dir(workDir)) +} + // Test building and loading plugins func TestPlugin(t *testing.T) { var ( testPluginDir = t.TempDir() testPluginPath = fmt.Sprintf("%s/test-plugin.so", testPluginDir) testPluginSourcePath = fmt.Sprintf("%s/test-plugin.go", testPluginDir) - testPluginSource = []byte(` -package main + testPluginSource = []byte( + `package main import ( - configurator "github.com/OpenCHAMI/configurator/pkg" + "github.com/OpenCHAMI/configurator/pkg/config" "github.com/OpenCHAMI/configurator/pkg/generator" - "github.com/OpenCHAMI/configurator/pkg/util" ) type TestGenerator struct{} -func (g *TestGenerator) GetName() string { return "test" } -func (g *TestGenerator) GetVersion() string { return "v1.0.0" } -func (g *TestGenerator) GetDescription() string { return "This is a plugin creating for running tests." } -func (g *TestGenerator) Generate(config *configurator.Config, opts ...util.Option) (generator.FileMap, error) { +func (g *TestGenerator) GetName() string { return "test" } +func (g *TestGenerator) GetVersion() string { return "v1.0.0" } +func (g *TestGenerator) GetDescription() string { + return "This is a plugin creating for running tests." +} +func (g *TestGenerator) Generate(config *config.Config, params generator.Params) (generator.FileMap, error) { return generator.FileMap{"test": []byte("test")}, nil } -var Generator TestGenerator - `) + +var Generator TestGenerator`) ) - wd, err := os.Getwd() - if err != nil { - t.Errorf("failed to get working directory: %v", err) - } + // get directory to replace remote pkg with local + // _, filename, _, _ := runtime.Caller(0) + // replaceDir := fmt.Sprintf("%s", filepath.Dir(workDir)) // show all paths to make sure we're using the correct ones - fmt.Printf("(TestPlugin) working directory: %v\n", wd) + fmt.Printf("(TestPlugin) working directory: %v\n", workDir) fmt.Printf("(TestPlugin) plugin directory: %v\n", testPluginDir) fmt.Printf("(TestPlugin) plugin path: %v\n", testPluginPath) fmt.Printf("(TestPlugin) plugin source path: %v\n", testPluginSourcePath) @@ -151,6 +158,12 @@ var Generator TestGenerator t.Fatalf("failed to execute command: %v\n%s", err, string(output)) } + // use the local `pkg` instead of the release one + cmd = exec.Command("bash", "-c", fmt.Sprintf("go mod edit -replace=github.com/OpenCHAMI/configurator=%s", replaceDir)) + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("failed to execute command: %v\n%s", err, string(output)) + } + // run `go mod tidy` for dependencies cmd = exec.Command("bash", "-c", "go mod tidy") if output, err := cmd.CombinedOutput(); err != nil { @@ -186,7 +199,7 @@ var Generator TestGenerator GetName() string GetVersion() string GetDescription() string - Generate(*configurator.Config, ...util.Option) (generator.FileMap, error) + Generate(*config.Config, generator.Params) (generator.FileMap, error) }); !ok { t.Error("plugin does not implement all of the generator interface") } @@ -203,7 +216,7 @@ var Generator TestGenerator GetName() string GetVersion() string GetDescription() string - Generate(*configurator.Config, ...util.Option) (generator.FileMap, error) + Generate(*config.Config, generator.Params) (generator.FileMap, error) }); !ok { t.Error("plugin does not implement all of the generator interface") } @@ -233,16 +246,15 @@ var Generator InvalidGenerator `) ) - wd, err := os.Getwd() - if err != nil { - t.Errorf("failed to get working directory: %v", err) - } // show all paths to make sure we're using the correct ones - fmt.Printf("(TestPluginWithInvalidOrNoSymbol) working directory: %v\n", wd) + fmt.Printf("(TestPluginWithInvalidOrNoSymbol) working directory: %v\n", workDir) fmt.Printf("(TestPluginWithInvalidOrNoSymbol) plugin directory: %v\n", testPluginDir) fmt.Printf("(TestPluginWithInvalidOrNoSymbol) plugin path: %v\n", testPluginPath) fmt.Printf("(TestPluginWithInvalidOrNoSymbol) plugin source path: %v\n", testPluginSourcePath) + // get directory to replace remote pkg with local + // _, filename, _, _ := runtime.Caller(0) + // make temporary directory to test plugin err = os.MkdirAll(testPluginDir, os.ModeDir) if err != nil { @@ -276,6 +288,12 @@ var Generator InvalidGenerator t.Fatalf("failed to execute command: %v\n%s", err, string(output)) } + // use the local `pkg` instead of the release one + cmd = exec.Command("bash", "-c", fmt.Sprintf("go mod edit -replace=github.com/OpenCHAMI/configurator=%s", replaceDir)) + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("failed to execute command: %v\n%s", err, string(output)) + } + // run `go mod tidy` for dependencies cmd = exec.Command("bash", "-c", "go mod tidy") if output, err := cmd.CombinedOutput(); err != nil { @@ -314,9 +332,8 @@ var Generator InvalidGenerator // we're not doing it here since that's not what is being tested. func TestGenerateExample(t *testing.T) { var ( - config = configurator.NewConfig() - client = configurator.NewSmdClient() - gen = TestGenerator{} + conf = config.New() + gen = TestGenerator{} ) // make sure our generator returns expected strings @@ -333,11 +350,7 @@ func TestGenerateExample(t *testing.T) { }) // try to generate a file with templating applied - fileMap, err := gen.Generate( - &config, - generator.WithTarget("test"), - generator.WithClient(client), - ) + fileMap, err := gen.Generate(&conf, generator.Params{}) if err != nil { t.Fatalf("failed to generate file: %v", err) } @@ -356,8 +369,7 @@ func TestGenerateExample(t *testing.T) { // try and load the plugin from a lib here either. func TestGenerateExampleWithServer(t *testing.T) { var ( - config = configurator.NewConfig() - client = configurator.NewSmdClient() + conf = config.New() gen = TestGenerator{} headers = make(map[string]string, 0) ) @@ -365,16 +377,14 @@ func TestGenerateExampleWithServer(t *testing.T) { // NOTE: Currently, the server needs a config to know where to get load plugins, // and how to handle targets/templates. This will be simplified in the future to // decoupled the server from required a config altogether. - config.Targets["test"] = configurator.Target{ + conf.Targets["test"] = configurator.Target{ TemplatePaths: []string{}, FilePaths: []string{}, } // create new server, add test generator, and start in background - server := server.New(&config) - server.GeneratorParams.Generators = map[string]generator.Generator{ - "test": &gen, - } + server := server.New(&conf) + generator.DefaultGenerators["test"] = &gen go server.Serve() // make request to server to generate a file @@ -390,10 +400,7 @@ func TestGenerateExampleWithServer(t *testing.T) { // // NOTE: we don't actually use the config in this plugin implementation, // but we do check that a valid config was passed. - fileMap, err := gen.Generate( - &config, - generator.WithClient(client), - ) + fileMap, err := gen.Generate(&conf, generator.Params{}) if err != nil { t.Fatalf("failed to generate file: %v", err) }