Discussion: IAC / Declarative Configuration for tokens and policies

[NiklasBeierl]

I would like to discuss whether there is interest for adding "Infrastructure as Code"-Style support for tokens and policies to this project and if yes, how it should look like.

Lets start off with the observation that we already have an IAC feature: Importing / Exporting Zone files. But there is more to desec than just dns records!

The two features of the desec api that are an absolute killer feature to me are tokens and policies. Desec is, to the best of my knowledge, the only DNS provider outside of big pubic cloud providers like AWS and CloudFlare that allows fine grained access control for managing dns records. However the tooling support is rather thin at the moment, not even desecs own web-gui supports setting policies yet.

With desec-dns we have a cli to manage policies but I would like to take it a little further. I am envisioning something like docker compose for desec-tokens, that I want to demonstrate based on the snippets below.

Imagine having a yaml file like this:

# tokens.yml
policies:
  disallow_all:
    perm_write: false

policysets:
  developer:
    - disallow_all # String -> Look up in `policies` 
      # Allow developer tokens to set TXT records for "local.dev.mysite.com" 
      # so they can get htttps on their dev setups via DNS-challenges.
    - domain: mysite.com 
      subname: local.dev
      type: txt
      perm_write: true
  
tokens:
  admin:
    name: "Personal token of our admin"
    perm_manage_tokens: true
  webserver:
    file: ./webserverToken.txt # The token (as in secret) gets written to a file instead of stdout.
    allowed_subnets: 
      - 1.2.3.4/32
    policies:
      - disallow_all
      - domain: mysite.com 
        subname: www
        type: txt
        perm_write: true
 bob:
   policies: developer # String -> Look up in `policysets`
 alice:
   name: "Personal token of Alice"
   policies: developer

Then, you run desec commands in a dir with a tokens.yml or pass one explicitly with some flag.

$ desec tokens --config ~/my-tokens.yml up
Created token webserver, written to ./webserverToken.txt
Token "Personal token of our admin" already exists, policies are up-to-date.
Token bob already exists, updating policies.
Created token "Personal token of alice": abcdefgh123

Rolling tokens:

$ desec tokens -c ~/my-tokens.yml roll bob
Deleted old token.
Creating new token.
Setting policies.
Created token bob: abcdefgh123

Deleting:

$ desec tokens delete bob
This will delete token <id>, are you sure? [y/N]: y
Deleted token bob.

Pruning:

$ desec tokens prune
The following tokens are not described in `./tokens.yml`:
- <id> "Developer who left last week"
- <id> "Server that was removed last month"
Are you sure you want to delete them? [y/N]:

Why not implement that in terraform?

I am sympathetic to that idea as well, especially since there already is a terraform-provider for desec. As far as I understand though, it doesn't have support for managing tokens and policies yet. Also I am really not sure how one would implement features like the proposed desec tokens roll and I am not sure how much control providers have over the output of terraform for printing the secret in case one doesn't want to put it in a file.

[peterthomassen]

(deSEC hat on)

However the tooling support is rather thin at the moment, not even desecs own web-gui supports setting policies yet.

Yup. That's mainly because there would be several REST requests required (e.g., to list the policies of a token, with each policy having a dropdown for the domain it relates to, and this dropdown in turn listing the domains for selection). This requires holding more complex state; OTOH, we're reluctant to implementing this right now, as our GUI is not extremely up-to-date from a framework perspective and we should first migrate to Vuetify 3. That's a whole other can of worms and that's why not much is happing right now.

Long story short, I just wanted to say that the lack of GUI should not be mistaken as a sign that the token policy API isn't stable -- in fact, it is.

manage policies but I would like to take it a little further. [...] for desec-tokens

That sounds like a fabulous idea, and if there's anything we can help with (especially conceptual or API discussions), happy to engage.

[s-hamann]

I would like to discuss whether there is interest for adding "Infrastructure as Code"-Style support for tokens and policies to this project and if yes, how it should look like.

There certainly is.

Lets start off with the observation that we already have an IAC feature: Importing / Exporting Zone files. But there is more to desec than just dns records!

For the record: There is also import and export of records as JSON. Same data though, just a different file format.

The two features of the desec api that are an absolute killer feature to me are tokens and policies. Desec is, to the best of my knowledge, the only DNS provider outside of big pubic cloud providers like AWS and CloudFlare that allows fine grained access control for managing dns records. However the tooling support is rather thin at the moment, not even desecs own web-gui supports setting policies yet.

With desec-dns we have a cli to manage policies but I would like to take it a little further. I am envisioning something like docker compose for desec-tokens, that I want to demonstrate based on the snippets below.

I'm in on that 🙂
Token and policy management currently involves copy&pasting various non-human-friendly UUIDs around, which is something that has been bothering me for a while. Your approach solves that nicely.
However, I'd like to take these improvements to the rest of the CLI to the extent possible.
For example, all commands should be able to reference tokens by their name rather than UUID (if the name happens to be unique).
And we could implement a roll-token subcommand, I suppose.

Naming policies and the concept of policy sets seem great for more human-friendly policy management. I don't think the API currently provides any support for this, so they will need to be stored in a local file, such a your proposed tokens.yml. I'm not quite sure how to apply that to existing commands, but we can surely come up with something.

I'd appreciate implementing and submitting the small changes first, if that makes any sense to you. Helps making the pull requests more easily reviewable.

YAML is the modern go-to for infrastructure as code, so I suppose that's pretty much set.
Unfortunately it's not supported by python core. We'll need to add another dependency 😐
And the existing import and export subcommands use JSON. Adding YAML to the mix seems somewhat cluttered. I think it might be best to change them to YAML import/export.

A detail question, though:
How do you want to check, if the tokens from the YAML file are known to the API? Storing their UUIDs locally conflicts with the infrastructure as code approach. I think that only leaves the name attribute, which you let the user to some arbitrary value. Is that sensible?

And finally a note on the time line: Implementing and testing a big change like this takes its time anyway, but I'd like to get a package published to PyPI before that. So that'll be my personal priority for now. However, don't let that stop you from having a go at this.

[peterthomassen]

reference tokens by their name rather than UUID (if the name happens to be unique)

Do you think uniqueness should be enforced on token names?

[s-hamann]

reference tokens by their name rather than UUID (if the name happens to be unique)

Do you think uniqueness should be enforced on token names?

From the API view point: No. Token names are optional anyway and I always regarded them as kind of comment/annotation. UUIDs are technically sufficient to uniquely identify a token, so why have another unique identifier?

From the API consumer view point: Names are a nice alternative to UUIDs when working with tokens. But they are only as reliable as the user's own naming convention. That makes them tricky to use in software, but I'm OK with supporting them only for users who choose to keep (a subset of) their token names unique.

[NiklasBeierl]

Oh wow, great to see the enthusiasm here! :)

However, I'd like to take these improvements to the rest of the CLI to the extent possible.
I'd appreciate implementing and submitting the small changes first, if that makes any sense to you. Helps making the pull requests more easily reviewable.

By that you mean first implementing a config file format and corresponding operations for domains and records? :)

And the existing import and export subcommands use JSON. Adding YAML to the mix seems somewhat cluttered. I think it might be best to change them to YAML import/export.

Hmm, I think it wouldn't be too hard to support both. YAML and JSON are quite interoperable.

How do you want to check, if the tokens from the YAML file are known to the API? Storing their UUIDs locally conflicts with the infrastructure as code approach. I think that only leaves the name attribute, which you let the user to some arbitrary value.

Yes the name of tokens would be the way to map between the yaml and the API. From the example:

tokens
 bob:
   # Since no name attribute is set, we use the "symbolic" name in the markup
   #  => token.name = bob
   policies: developer
 alice:
   name: "Personal token of Alice" 
   # => token.name = "Personal token of Alice"
   policies: developer

Do you think uniqueness should be enforced on token names?

I think we should just abort if we can't uniquely resolve a name from a config file to a token from the api. What would be nice is the ability to get tokens "by name". Think: GET api/v1/auth/tokens?name=..., returning a list of tokens. If the result has more than one element the CLI aborts.

I would try to avoid having a local "mapping" that can get out of sync as far as we can.

And finally a note on the time line: Implementing and testing a big change like this takes its time anyway, but I'd like to get a package published to PyPI before that. So that'll be my personal priority for now. However, don't let that stop you from having a go at this.

Yes, I think the first step would be a "spec" for this anyway.

[peterthomassen]

Yes the name of tokens would be the way to map between the yaml and the API. From the example:

I suggest adding a prefix (s-hamann/desec-dns: or some such thing) to prevent collisions with (potentially older) user-specified names.

What would be nice is the ability to get tokens "by name". Think: GET api/v1/auth/tokens?name=..., returning a list of tokens.

Mh. That requires an extra database index, but why not. Care to file a PR? ;-)

[s-hamann]

However, I'd like to take these improvements to the rest of the CLI to the extent possible.

By that you mean first implementing a config file format and corresponding operations for domains and records? :)

Yes, something like a function to roll (or copy) tokens or to resolve a token name to the UUID (assuming no changes in the API).

And the existing import and export subcommands use JSON. Adding YAML to the mix seems somewhat cluttered. I think it might be best to change them to YAML import/export.

Hmm, I think it wouldn't be too hard to support both. YAML and JSON are quite interoperable.

Technically, yes. But I wonder if it justifies the more complex UI.

Yes the name of tokens would be the way to map between the yaml and the API. From the example:

tokens
 bob:
   # Since no name attribute is set, we use the "symbolic" name in the markup
   #  => token.name = bob
   policies: developer
 alice:
   name: "Personal token of Alice" 
   # => token.name = "Personal token of Alice"
   policies: developer

That actually moves the "key" that lets us reference the token inside the YAML mapping/dict. Users might get the idea that they can add/remove/change the name and still reference the same "alice" token, just as they can do with policy sets, etc.
I think we should just enforce using the (prefixed) YAML key as the API-side name. That takes the API's name attribute away from users, but they can comment the YAML file for mostly the same purpose.

I suggest adding a prefix (s-hamann/desec-dns: or some such thing) to prevent collisions with (potentially older) user-specified names.

Makes sense but also makes me wish I had chosen a better name back when I started this project 😉

What would be nice is the ability to get tokens "by name". Think: GET api/v1/auth/tokens?name=..., returning a list of tokens.

Mh. That requires an extra database index, but why not. Care to file a PR? ;-)

I don't have time for a PR and it's easy enough to do downstream. But if anyone else wants to do it, I'd happily make use of it 🙂

[NiklasBeierl]

That actually moves the "key" that lets us reference the token inside the YAML mapping/dict. Users might get the idea that they can add/remove/change the name and still reference the same "alice" token, just as they can do with policy sets, etc. I think we should just enforce using the (prefixed) YAML key as the API-side name. That takes the API's name attribute away from users, but they can comment the YAML file for mostly the same purpose.

Hmm, I am either missunderstanding you here or I want to challenge that notion. In any case I think the discussion could benefit from an explicit distinction. There is the name we give resources inside the yaml / config file. In the example we have bob and alice as tokens, disallow_all as a policy and developer as a policyset. For lack of a better word, I would like to call these symbolic names . Then there is the "name" attribute of tokens, lets call that api name and note that it is not necessarily unique.

Finally we have the actual identifiers used by the api, simply called id:

• Domain: Name of the domain (e.g. example.com)
• RRSets: Tuple: (domain, subname, type)
• Tokens: Their: id
• Token Policies: Tuple: (token.id,policy.id)

@peterthomassen please correct me if I got any of this wrong!

Now, when "applying" (parts of) a config file, we will always undergo these steps:

1. Parse the configuration, Resolving any symbolic names and get a model of the "desired state"
2. Map the model resources onto existing resources
3. Make updates

For now my focus is on step 1 and 2:

In principle, any referencing between resources within the config file should always happen by symbolic name. Note that in the current example, all resources that you might want to reference have a symbolic name given by their yaml mapping key.

We potentially have concepts in our config file that do not exist in the api, such as policysets. These only need to be handled in step 1 by their symbolic name. This is similar to how docker compose has a notion of projects and services, whereas docker itself has no real notion of either of those. (Usually a service maps to exactly one container, but there are many cases where this doesn't hold).

Domain and RRSet will still require carefull design of the config language, but for them it is a bit easier since they have "natural" ids. There will not be any touching point between symbolic names and the api.

Tokens and policies are different because they have api-side ids.

Let's do policies first, because they are easier:
During stage 1, we get a list of "desired" policies for all our tokens. If the set is in any way different from the current state, we update the existing one. There is no way to update policies on the api side. Only add / remove them to tokens and they are always scoped within their token, so the ids are really only needed during the update process.

Which leaves to tokens which I think that is the only tricky part: But I think a principled solution could be this:
As mentioned above, the config file will only ever reference this token by its symbolic name, so that is all we need for stage 1. For stage 2 we need to figure out the id of the token.

Case 1: The token config only has a symbolic name
We modify / create a token with <prefix>/<symbolic name> as name attribute.

tokens:
  alice:
    policies:
      ... 
  # equivalent:
  alice:
    name: desec-dns/alice
    policies:
      ...

Case 2: The token config has a name attribute set
We look for / create a token with that name. That allows desec-dns to be used with other tools that might have other predispositions on the name attribute and it accommodates users that don't like our naming scheme.

tokens:
  alice:
    name: "Alice's personal token"
    policies:
      ... 

Case 3: The token config has an id set
This could be used as a last resort if there is a token that has no name and you want to manage its policies without creating a new one. roll token will not work for this token.

tokens:
  some_legacy_token:
    id: 3a6b94b5-d20e-40bd-a7cc-521f5c79fab3
    policies:
      ... 

Users might get the idea that they can add/remove/change the name and still reference the same "alice" token, just as they can do with policy sets, etc.

Well, they can change the symbolic name, as long as they set name explicitly. But these kinds of problems are almost inevitable. 🤷 We could again use docker compose as inspiration and print "orphan"-warnings if we see tokens in the api that have our prefix but don't match anything in our model.

[s-hamann]

Yes, what you describe is basically what I envision, too.

Well, they can change the symbolic name, as long as they set name explicitly.

My concern is not about users changing the symbolic name on a whim. It's a YAML key, so I believe it's fairly intuitive to assume that changing it breaks existing references, i.e. the mapping between symbolic name and API name.
My concern is that changing the name attribute within the YAML file looks benign, although actually has the same consequence as changing the symbolic name of a case-1-token.

An example to make my point clear:
User has and deployed the following configuration:

tokens:
  alice:
    policies: ...

User finds out about the name attribute and wants to give that token a nice name. Config changes to:

tokens:
  alice:
    name: "Alice's personal token"
    policies: ...

On deployment, the user may be surprised when the API name of Alice's existing token does not change but a new token is created instead. Most likely, that surprise comes after deleting the weirdly named old token manually or by using desec token prune.
Of course, there will be documentation that clearly warned about this. But - knowing myself - I don't expect most users to actually read it.

Not implementing case 2 in the first place would prevent this kind of surprise. Unfortunately, it also limits interoperability with other tools (if we enforce a prefix on the API name).

[NiklasBeierl]

My concern is not about users changing the symbolic name on a whim. It's a YAML key, so I believe it's fairly intuitive to assume that changing it breaks existing references, i.e. the mapping between symbolic name and API name.
My concern is that changing the name attribute within the YAML file looks benign, although actually has the same consequence as changing the symbolic name of a case-1-token.

Now I understand. Two suggestions:

1. More explicit key for the name attribute. E.g. api_name or identifying_name?

2. Make it clear that this attribute is used as an identifier through nesting, can also be applied to case 3:

tokens:
  alice:
    identifier:
      id: 3a6b94b5-d20e-40bd-a7cc-521f5c79fab3
    policies:
      ...
  alice:
    identifier:
     name: desec-dns/alice
    policies:
      ...
  alice:   # Implicitly equivalent to the one above
    policies:
      ...

[s-hamann]

Two suggestions:

1. More explicit key for the name attribute. E.g. api_name or identifying_name?

2. Make it clear that this attribute is used as an identifier through nesting, can also be applied to case 3:

Good ideas, both. I think, I like the second one better.

[peterthomassen]

There is no way to update policies on the api side.

You should also be able to PATCH/PUT them.

Not implementing case 2 in the first place would prevent this kind of surprise. Unfortunately, it also limits interoperability with other tools

You could call the attribute foreign_name to indicate that it's for interoperability. (Relates to @NiklasBeierl's suggestion 1.)