Update README, bug fixes, and initial targets API

README.md

@@ -1,90 +1,106 @@
 # OpenCHAMI Configurator
 
-The `configurator` (portmanteau of config + generator) is an extensible tool that fetchs data from an instance of [SMD](https://github.com/OpenCHAMI/smd) to generate commonly used config files based on Jinja 2 template files. The tool and generator plugins are written in Go and plugins can be written by following the ["Creating Generator Plugins"](#creating-generator-plugins) section of this README.
+The `configurator` is an extensible tool that is capable of dynamically generating files on the fly. The tool includes a built-in generator that fetchs data from an instance of [SMD](https://github.com/OpenCHAMI/smd) to generate files based on Jinja 2 template files. The tool and generator plugins are written in Go and plugins can be written by following the ["Creating Generator Plugins"](#creating-generator-plugins) section of this README.
 
 ## Building and Usage
 
-The `configurator` is built using standard `go` build tools. The project separates the client and server with build tags. To get started, clone the project, download the dependencies, and build the project:
+The `configurator` is built using standard `go` build tools. The project separates the client, server, and generator components using build tags. To get started, clone the project, download the dependencies, and build the project:
 
 ```bash
 git clone https://github.com/OpenCHAMI/configurator.git
 go mod tidy
 go build --tags all # equivalent to `go build --tags client,server``
-
-## ...or just run `make` in project directory
 ```
 
-This will build the main driver program, but also requires generator plugins to define how new config files are generated. The default plugins can be built using the following build command:
+This will build the main driver program with the default generators that are found in the `pkg/generators` directory.
+
+> [!WARNING]
+> Not all of the plugins have completed generation implementations and are a WIP.
+
+### Running Configurator with CLI
+
+After you build the program, run the following command to use the tool:
 
 ```bash
-go build -buildmode=plugin -o lib/conman.so internal/generator/plugins/conman/conman.go
-go build -buildmode=plugin -o lib/coredhcp.so internal/generator/plugins/coredhcp/coredhcp.go
-go build -buildmode=plugin -o lib/dnsmasq.so internal/generator/plugins/dnsmasq/dnsmasq.go
-go build -buildmode=plugin -o lib/powerman.so internal/generator/plugins/powerman/powerman.go
-go build -buildmode=plugin -o lib/syslog.so internal/generator/plugins/syslog/syslog.go
+export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs...
+./configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert ochami.pem
 ```
 
-**NOTE: Not all of the plugins have completed generation implementations and are WIP.**
+This will generate a new `coredhcp` config file based on the Jinja 2 template specified in the config file for "coredhcp". The files will be written to `coredhcp.conf` as specified with the `-o/--output` flag. The `--target` flag specifies the type of config file to generate by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details).
 
-These commands will build the default plugins and store them in the "lib" directory by default. Alternatively, the plugins can be built using `make plugins` if GNU make is installed and available. After you build the plugins, run the following to use the tool:
+In other words, there should be an entry in the config file that looks like this:
+
+```yaml
+...
+targets:
+  coredhcp:
+    plugin: "lib/coredhcp.so"  # optional, if we want to use an external plugin instead
+    templates:
+      - templates/coredhcp.j2
+...
 
-```bash
-./configurator generate --config config.yaml --target dnsmasq -o dnsmasq.conf
 ```
 
-This will generate a new `dnsmasq` config file based on the Jinja 2 template specified in the config file for "dnsmasq". The `--target` flag specifies the type of config file to generate by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). The `configurator` tool requires a valid access token when making requests to an instance of SMD that has protected routes.
+> [!NOTE]
+> The `configurator` tool requires a valid access token when making requests to an instance of SMD that has protected routes.
+
+### Running Configurator as a Service
 
 The tool can also run as a service to generate files for clients:
 
 ```bash
+export CONFIGURATOR_JWKS_URL="http://my.openchami.cluster:8443/key"
 ./configurator serve --config config.yaml
 ```
 
 Once the server is up and listening for HTTP requests, you can try making a request to it with `curl` or `configurator fetch`. Both commands below are essentially equivalent:
 
 ```bash
-curl http://127.0.0.1:3334/generate?target=dnsmasq -H "Authorization: Bearer $ACCESS_TOKEN"
+export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs...
+curl http://127.0.0.1:3334/generate?target=coredhcp -X GET -H "Authorization: Bearer $ACCESS_TOKEN" --cacert ochami.pem
 # ...or...
-./configurator fetch --target dnsmasq --host http://127.0.0.1 --port 3334 
+./configurator fetch --target coredhcp --host http://127.0.0.1:3334 --cacert ochami.pem
 ```
 
-This will do the same thing as the `generate` subcommand, but remotely. The access token is only required if the `CONFIGURATOR_JWKS_URL` environment variable is set. The `ACCESS_TOKEN` environment variable passed to `curl` and it's corresponding CLI argument both expects a token as a JWT.
+This will do the same thing as the `generate` subcommand, but through a GET request where the file contents is returned in the response. The access token is only required if the `CONFIGURATOR_JWKS_URL` environment variable is set when starting the server with `serve`. The `ACCESS_TOKEN` environment variable is passed to `curl` using the `Authorization` header and expects a token as a JWT.
 
 ### Docker
 
-New images can be built and tested using the `Dockerfile` provided in the project. However, the binary executable and the generator plugins must first be built before building the image since the Docker build copies the binary over. Therefore, build all of the binaries first by following the first section of ["Building and Usage"](#building-and-usage). If you run the `make docker`, this will be done for you. Otherwise, run the `docker build` command after building the executable and libraries.
+New images can be built and tested using the `Dockerfile` provided in the project. However, the binary executable and the generator plugins must first be built before building the image since the Docker build copies the binary over. Therefore, build all of the binaries first by following the first section of ["Building and Usage"](#building-and-usage). Running `make docker` from the Makefile will automate this process. Otherwise, run the `docker build` command after building the executable and libraries.
 
 ```bash
 docker build -t configurator:testing path/to/configurator/Dockerfile
-# ...or
-make docker
 ```
 
-Keep in mind that all plugins included in the project are build in the `lib/` directory and copied from there. If you want to easily include your own external generator plugins, you can build it and copy the `lib.so` file to that location. Make sure that the `Generator` interface is implemented correct as described in the ["Creating Generator Plugins"](#creating-generator-plugins) or the plugin will not load. Additionally, the name string returned from the `GetName()` method is used for looking up the plugin after all plugins have been loaded by the main driver.
+If you want to easily include your own external generator plugins, you can build it and copy the `lib.so` file to `lib/`. Make sure that the `Generator` interface is implemented correctly as described in the ["Creating Generator Plugins"](#creating-generator-plugins) or the plugin will not load (you should get an error that specifically says this). Additionally, the name string returned from the `GetName()` method is used for looking up the plugin with the `--target` flag by the main driver program.
 
 Alternatively, pull the latest existing image/container from the GitHub container repository.
 
 ```bash
 docker pull ghcr.io/openchami/configurator:latest
 ```
 
-Then, run the container similarly to the binary.
+Then, run the Docker container similarly to running the binary.
 
-```
-docker run ghcr.io/openchami/configurator:latest configurator generate --config config.yaml --target dnsmasq
+```bash
+export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs...
+docker run ghcr.io/openchami/configurator:latest configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert configurator.pem
 ```
 
 ### Creating Generator Plugins
 
-The `configurator` uses generator plugins to define how config files are generated using a `Generator` interface.  The interface is defined like so:
+The `configurator` uses built-in and user-defined generators that implement the `Generator` interface to describe how config files should be generated. The interface is defined like so:
 
 ```go
-type Files = map[string][]byte
+// maps the file path to its contents
+type FileMap = map[string][]byte
+
+// interface for generator plugins
 type Generator interface {
   GetName() string
   GetVersion() string
   GetDescription() string
-  Generate(config *configurator.Config, opts ...util.Option) (Files, error)
+  Generate(config *configurator.Config, opts ...util.Option) (FileMap, error)
 }
 ```
 
@@ -93,23 +109,30 @@ A new plugin can be created by implementing the methods from interface and expor
 ```go
 package main
 
-type MyGenerator struct {}
+type MyGenerator struct {
+  PluginInfo map[string]any
+}
+
+var pluginInfo map[string]any
+
+// this function is not a part of the `Generator` interface
+func (g *MyGenerator) LoadFromFile() map[string]any{ /*...*/ }
 
 func (g *MyGenerator) GetName() string {
   // just an example...this can be done however you want
-  pluginInfo := LoadFromFile("path/to/plugin/info.json")
-  return pluginInfo["name"]
+  g.PluginInfo := LoadFromFile("path/to/plugin/info.json")
+  return g.PluginInfo["name"]
 }
 
 func (g *MyGenerator) GetVersion() string {
-  return "v1.0.0"
+  return g.PluginInfo["version"] // "v1.0.0"
 }
 
 func (g *MyGenerator) GetDescription() string {
-  return "This is an example plugin."
+  return g.PluginInfo["description"] // "This is an example plugin."
 }
 
-func (g *MyGenerator) Generate(config *configurator.Config, opts ...util.Option) (generator.Files, error) {
+func (g *MyGenerator) Generate(config *configurator.Config, opts ...util.Option) (generator.FileMap, error) {
   // do config generation stuff here...
   var (
     params = generator.GetParams(opts...)
@@ -121,7 +144,7 @@ func (g *MyGenerator) Generate(config *configurator.Config, opts ...util.Option)
     // ... blah, blah, blah, check error, format output, and so on...
   }
 
-  // apply the substitutions to Jinja template and return output as byte array
+  // apply the substitutions to Jinja template and return output as FileMap (i.e. path and it's contents)
   return generator.ApplyTemplate(path, generator.Mappings{
     "plugin_name":        g.GetName(),
     "plugin_version":     g.GetVersion(),
@@ -140,7 +163,7 @@ Finally, build the plugin and put it somewhere specified by `plugins` in your co
 go build -buildmode=plugin -o lib/mygenerator.so path/to/mygenerator.go
 ```
 
-Now your plugin should be available to use with the `configurator` main driver. If you get an error about not loading the correct symbol type, make sure that you generator function definitions match the `Generator` interface exactly.
+Now your plugin should be available to use with the `configurator` main driver program. If you get an error about not loading the correct symbol type, make sure that your generator function definitions match the `Generator` interface entirely and that you don't have a partially implemented interface.
 
 ## Configuration
 
@@ -150,46 +173,33 @@ Here is an example config file to start using configurator:
 server:         # Server-related parameters when using as service
   host: 127.0.0.1
   port: 3334
-  jwks:         # Set the JWKS uri to protect /generate route
+  jwks:         # Set the JWKS uri for protected routes
     uri: ""
     retries: 5
-smd: .          # SMD-related parameters
-  host: http://127.0.0.1
-  port: 27779
+smd:            # SMD-related parameters
+  host: http://127.0.0.1:27779
 plugins:        # path to plugin directories
   - "lib/"
 targets:        # targets to call with --target flag
-  dnsmasq:
+  coredhcp:
     templates:
-      - templates/dnsmasq.jinja
-  warewulf:
-    templates:  # files using Jinja templating
-      - templates/warewulf/vnfs/dhcpd-template.jinja
-      - templates/warewulf/vnfs/dnsmasq-template.jinja
+      - templates/coredhcp.j2
     files:      # files to be copied without templating
-      - templates/warewulf/defaults/provision.jinja
-      - templates/warewulf/defaults/node.jinja
-      - templates/warewulf/filesystem/examples/*
-      - templates/warewulf/vnfs/*
-      - templates/warewulf/bootstrap.jinja
-      - templates/warewulf/database.jinja
-    targets:    # additional targets to run 
+      - extra/nodes.conf
+    targets:    # additional targets to run (does not run recursively)
       - dnsmasq
 ```
 
-The `server` section sets the properties for running the `configurator` tool as a service and is not required if you're only using the CLI. Also note that the `jwks-uri` parameter is only needs for protecting endpoints. If it is not set, then the API is entirely public. The `smd` section tells the `configurator` tool where to find SMD to pull state management data used by the internal client. The `templates` section is where the paths are mapped to each generator plugin by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). The `plugins` is a list of paths to load generator plugins.
+The `server` section sets the properties for running the `configurator` tool as a service and is not required if you're only using the CLI. Also note that the `jwks.uri` parameter is only needed for protecting endpoints. If it is not set, then all API routes are entirely public. The `smd` section tells the `configurator` tool where to find the SMD service to pull state management data used internally by the client's generator. The `templates` section is where the paths are mapped to each generator by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). The `plugins` is a list of paths to search for and load external generator plugins.
 
 ## Running the Tests
 
-The `configurator` project includes a collection of tests focused on verifying plugin behavior and generating files. The tests do not currently test fetching information from SMD (or whatever remote source). The tests can be ran with either of the following commands:
+The `configurator` project includes a collection of tests focused on verifying plugin behavior and generating files. The tests do not include fetching information from any remote sources, can be ran with the following command:
 
 ```bash
 go test ./tests/generate_test.go --tags=all
-# ...or alternatively with GNU make...
-make test
 ```
 
-
 ## Known Issues
 
 - Adds a new `OAuthClient` with every token request

cmd/config.go

@@ -1,11 +1,10 @@
 package cmd
 
 import (
-	"fmt"
-
+	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
 
-	configurator "github.com/OpenCHAMI/configurator/pkg"
+	"github.com/OpenCHAMI/configurator/pkg/config"
 	"github.com/OpenCHAMI/configurator/pkg/util"
 )
 
@@ -20,10 +19,10 @@ var configCmd = &cobra.Command{
 		for _, path := range args {
 			// check and make sure something doesn't exist first
 			if exists, err := util.PathExists(path); exists || err != nil {
-				fmt.Printf("file or directory exists\n")
+				log.Error().Err(err).Msg("file or directory exists")
 				continue
 			}
-			configurator.SaveDefaultConfig(path)
+			config.SaveDefault(path)
 		}
 	},
 }

cmd/fetch.go

@@ -24,16 +24,21 @@ var fetchCmd = &cobra.Command{
 			return
 		}
 
+		// check if we actually have any targets to run
+		if len(targets) <= 0 {
+			log.Error().Msg("must specify a target")
+		}
+
 		// check to see if an access token is available from env
-		if config.AccessToken == "" {
+		if conf.AccessToken == "" {
 			// check if OCHAMI_ACCESS_TOKEN env var is set if no access token is provided and use that instead
 			accessToken := os.Getenv("ACCESS_TOKEN")
 			if accessToken != "" {
-				config.AccessToken = accessToken
+				conf.AccessToken = accessToken
 			} else {
 				// TODO: try and fetch token first if it is needed
 				if verbose {
-					fmt.Printf("No token found. Attempting to generate config without one...\n")
+					log.Warn().Msg("No token found. Attempting to generate config without one...")
 				}
 			}
 		}
@@ -46,25 +51,23 @@ var fetchCmd = &cobra.Command{
 
 		for _, target := range targets {
 			// make a request for each target
-			url := fmt.Sprintf("%s:%d/generate?target=%s", remoteHost, remotePort, target)
+			url := fmt.Sprintf("%s/generate?target=%s", remoteHost, target)
 			res, body, err := util.MakeRequest(url, http.MethodGet, nil, headers)
 			if err != nil {
-				log.Error().Err(err).Msg("failed to make request")
+				log.Error().Err(err).Msg("failed to fetch files")
 				return
 			}
 			// handle getting other error codes other than a 200
 			if res != nil {
-				if res.StatusCode == http.StatusOK {
-					log.Info().Msgf("%s\n", string(body))
-				}
+				// NOTE: the server responses are already marshaled to JSON
+				fmt.Print(string(body))
 			}
 		}
 	},
 }
 
 func init() {
-	fetchCmd.Flags().StringVar(&remoteHost, "host", "", "set the remote configurator host")
-	fetchCmd.Flags().IntVar(&remotePort, "port", 3334, "set the remote configurator port")
+	fetchCmd.Flags().StringVar(&remoteHost, "host", "", "set the remote configurator host and port")
 	fetchCmd.Flags().StringSliceVar(&targets, "target", nil, "set the target configs to make")
 	fetchCmd.Flags().StringVarP(&outputPath, "output", "o", "", "set the output path for config targets")
 	fetchCmd.Flags().StringVar(&accessToken, "access-token", "o", "set the output path for config targets")