Add API to manage groups in meta-data

[davidallendj]

This PR addresses #17 adds a groups API to cloud-init with the following endpoints:

• POST /groups adds data from request body to metadata.groups
• PUT /groups update data from request body to metadata.groups if it exists
• GET /groups returns and prints existing metadata.groups as YAML
• DELETE /groups removes existing metadata.groups for id

The endpoints above should be used for managing the entire group data structure. Groups with the structure can be further fine-tuned using an identifier:

• POST /groups/{id} add data for a specific group if the data does not exist for id
• PUT /groups/{id} updates data if it already exists for specific group
• GET /groups/{id} returns and prints existing data for specific group as YAML
• DELETE /groups/{id} removes existing data for group specified with id

One thing to keep in mind is that to access the meta-data in the payload, you must specify the identifier with the way cloud-init data is structured. The /groups API only allows for setting the entire meta-data.groups data and not individual key-value pairs. This implementation requires the meta-data to already exists and assumes a specific ##groups tag for the citypes.CI.Name (which can be changed). To set specific key-value pairs in a group, use the /groups/{id} endpoints as shown below.

Creating Groups Without the Groups API

Currently, groups can be created using the following without using the API added in this PR with the following:

curl 'http://127.0.0.1:27777/cloud-init/' -d @data.json -X POST

where data.json is something like:

{
  "name": "IDENTIFIER",
  "cloud-init": {
    "userdata": {
      "write_files": [
        {
          "content": "hello world",
          "path": "/etc/hello"
        }
      ]
    },
    "metadata": {
      "groups": {...}
    }
  }
}

The internal data structure requires that there be a name identifier included in the data. The groups API assumes a specific name as mentioned above to get around this for now.

Creating Groups Using the Groups API

Groups can be added and modified using the /groups endpoints to modify all the data or /groups/{id} endpoints to modify specific key-value pairs within a group:

curl 'http://127.0.0.1:27777/cloud-init/groups' -d @groups.json -X POST  # adds group data
curl 'http://127.0.0.1:27777/cloud-init/groups' -d @groups.json -X PUT    # updates group data

The JSON in groups.json only contains the data to include in the groups:

{
  "x3000": {
    "syslog_aggregator": "127.0.0.1"
  },
  "canary-123": {
    "syslog_aggregator": "172.16.0.105"
  }
}

To confirm that the data was added and return as YAML:

curl 'http://127.0.0.1:27777/cloud-init/groups'

This should return something like this:

x3000: 
    syslog_aggregator: "127.0.0.1"
canary-123: 
    syslog_aggregator: "172.16.0.105"

If instead you only want to add key-value pairs to your group data, you can do something similar to the /groups endpoint, but you must specify an identifier associated with the group:

curl 'http://127.0.0.1:27777/cloud-init/groups/test' -d @groups.json -X POST  # adds group key-value data
curl 'http://127.0.0.1:27777/cloud-init/groups/test' -d @groups.json -X PUT    # updates group key-value data

The data from groups.json will now be added as a value with a test key and should return this:

test:
  x3000: 
      syslog_aggregator: "127.0.0.1"
  canary-123: 
      syslog_aggregator: "172.16.0.105"

The commands above have been tested with a local instance of cloud-init built with the changes in this branch.

[alexlovelltroy]

Please move IDENTIFIER after cloud-init in the path

[travisbcotton]

I built and ran a container based on this branch and I can't get the /groups endpoint to work

This is the payload:

{
  "x3000": {
    "syslog_aggregator": "127.0.0.1"
  },
  "canary-123": {
    "syslog_aggregator": "172.16.0.105"
  }
}

I then try to POST to the /groups endpoint with

curl 'https://stratus.openchami.cluster:8443/cloud-init/groups' -d @groups2.json -X POST

But this returns not found
The logs show this for the POST attempt

3:51PM INF Request bytes_in=113 bytes_out=11 duration=0.039503 method=POST remote_addr=172.16.0.254 request_id=cloud-init-server/x0daBEQElZ-000010 request_uri=/cloud-init/groups status="Internal Server Error" status_code=500 user_agent=curl/7.61.1
2024/11/07 15:51:00 [cloud-init-server/x0daBEQElZ-000010] "POST http://stratus.openchami.cluster:8443/cloud-init/groups HTTP/1.1" from 172.16.0.254 - 500 11B in 128.479µs

[davidallendj]

I built and ran a container based on this branch and I can't get the /groups endpoint to work

This is the payload:

{
  "x3000": {
    "syslog_aggregator": "127.0.0.1"
  },
  "canary-123": {
    "syslog_aggregator": "172.16.0.105"
  }
}

I then try to POST to the /groups endpoint with

curl 'https://stratus.openchami.cluster:8443/cloud-init/groups' -d @groups2.json -X POST

But this returns not found The logs show this for the POST attempt

3:51PM INF Request bytes_in=113 bytes_out=11 duration=0.039503 method=POST remote_addr=172.16.0.254 request_id=cloud-init-server/x0daBEQElZ-000010 request_uri=/cloud-init/groups status="Internal Server Error" status_code=500 user_agent=curl/7.61.1
2024/11/07 15:51:00 [cloud-init-server/x0daBEQElZ-000010] "POST http://stratus.openchami.cluster:8443/cloud-init/groups HTTP/1.1" from 172.16.0.254 - 500 11B in 128.479µs

How is the container built? I'm wondering if something weird is happening with copying the binary. Also, is it possible to try again on bare metal?

[travisbcotton]

I'm using the Dockerfile

go build ./cmd/cloud-init-server/
podman build -t cloud-init-server:test -f Dockerfile .