From 6acc0fcd13c651876bd41794879448d2cc48f611 Mon Sep 17 00:00:00 2001 From: gve Date: Sat, 20 Apr 2019 17:30:12 +0200 Subject: [PATCH 1/9] test org mode markdown readme --- README.md => README.org | 159 ++++++++++++++++++++-------------------- 1 file changed, 79 insertions(+), 80 deletions(-) rename README.md => README.org (68%) diff --git a/README.md b/README.org similarity index 68% rename from README.md rename to README.org index c195444..723598e 100644 --- a/README.md +++ b/README.org @@ -1,125 +1,120 @@ -# vaultify - -Our greater purpose is to **never leave any secret file on a persisting -filesystem.** - -`vaultify` allows to integrate various means of secret provision into -any runtime environment, may it be some `containerd`, virtualized box or -bare-metal. Currently, we provide three means of fetching secrets and three -means of consuming them. - - ->see the table in the feature section for more - -## architecture - -vaultify is an extensible program, which is driven by a notion of -providers and consumers. Providers are classes, that are meant to -fetch/generate/decrypt secrets and pass them as a dictionary to a -consumer. The consumer will then digest that dictionary in the way it -has been designed: writing it as a `.env` file, some `.json` or spawn -a new process within vaultify-enriched environment. - - -> You want to store secret environment keys in a gpg encrypted file -> and then provide to it to the environment: use -> the `GPGProvider`-provider and the `EnvRunner`-consumer, e.g. - -vaultify will try to compile all secrets into one dictionary, and then -write those to either a volatile file system or just enrich the -current environment. - -What will be done actually is configurable via the environment, that -executes vaultify. - -## os dependencies - -ubuntu: -``` -apt-get install entr make -``` - -## usage - -### first steps +#+TITLE: vaultify usage +#+OPTIONS: +#+AUTHOR: Georg vom Endt (krysopath@gmail.com) + +* Installation +This section explains different installation methods. +** Requirements + - python3.6, python3.7 + - pip + - docker (obsoletes the above) + +** from pypi +Most implementers will fetch the package from pypi and build their own +things on another layer. +#+BEGIN_SRC shell +pip3 install vaultify +#+END_SRC + +** from git +Alternatively fetch the sources from github to develop your own +adapter classes. +#+BEGIN_SRC shell +git clone git@github.com:krysopath/vaultify.git +#+END_SRC +** os dependencies +These tools can greatly speed up local development, when used in +conjunction via ~make~, so install them. They can overwhelm you with +output, too. + +*** ubuntu +#+BEGIN_SRC shell +apt install make entr yamllint pylint bandit +#+END_SRC + +* usage + +** first steps 1. either create a virtualenv and install requirements.txt or 2. run `docker build .` Before using `vaultify` you need to export these into your environment: -```bash +#+BEGIN_SRC export VAULTIFY_PROVIDER= export VAULTIFY_CONSUMER= export VAULTIFY_LOGLEVEL= # default is info -``` +#+END_SRC >**r**ead **t**he **f**riendly **m**anual for the chosen Provider & Consumer, too Then just run: -```bash +#+BEGIN_SRC ./entry.py -``` +#+END_SRC -## feature overview + +** feature overview In this table you find an info about which Provider/Consumer pairs are supported: -``` - | GPG | OpenSSL | Vault | -|--------------|-----|---------|-------| -| DotEnvWriter | y | y | y | -| EnvRunner | y | y | y | -| JsonWriter | y | y | y | -``` +| pairs | GPG | OpenSSL | Vault | +|--------------+-----+---------+-------| +| DotEnvWriter | y | y | y | +| EnvRunner | y | y | y | +| JsonWriter | y | y | y | + Assuming the pattern holds, we expect always full compatibility between any Provider/Consumer pair. -### providers +** providers Providers are all classes, that create some `vaultify`-compliant dictionary, which then is used by `vaultify` consumers. -#### GPGProvider +*** GPGProvider This Provider can decrypt symmetrically encrypted files, created with `gpg`. Adhere to this format: -```bash +#+BEGIN_SRC KEY1=VAL1 [...] KEYN=VALN -``` +#+END_SRC To encrypt such a file, execute: -```bash +#+BEGIN_SRC gpg --symmetric -``` +#+END_SRC Below are environment variables, that are needed by this provider: -```bash +#+BEGIN_SRC # this will be used as the passphrase to gpg export VAULTIFY_SECRET= -``` +#+END_SRC -#### OpenSSLProvider +*** OpenSSLProvider This provider can decrypt symmetrically encrypted files, created with `openssl` Adhere to this format: -```bash +#+BEGIN_SRC KEY1=VAL1 [...] KEYN=VALN -``` +#+END_SRC To encrypt such a file, execute: ```bash +#+BEGIN_SRC openssl enc -aes-256-cbc -salt -a -in -out .enc +#+END_SRC ``` > Do not use aes-256-cbc, if there is aes-256-gcm available in your openssl. @@ -131,18 +126,20 @@ supports aes-256-gcm, please file a bug report against vaultify. Below are environment variables, that are needed by this provider: ```bash +#+BEGIN_SRC +#+END_SRC # this will be used as the passphrase to openssl export VAULTIFY_SECRET= ``` -#### VaultProvider +*** VaultProvider This provider fetches secrets from HashiCorp Vault API. Below are environment variables, that are needed by this provider: -```bash +#+BEGIN_SRC # set this to a reachable vault API export VAULT_ADDR= # set this to nodes in vaults kv engine, where you do have perms for READ @@ -150,38 +147,38 @@ export VAULT_PATHS= # if you do not set $VAULTIFY_SECRET, then export VAULT_TOKEN= -``` +#+END_SRC `VaultProvider` will use `VAULTIFY_SECRET` or `VAULT_TOKEN` for authentication, in that order. -### consumers +** consumers are all classes that operate on a `vaultify` compliant dictionary, to **somehow** use the secrets in there for the greater good. -#### DotEnvWriter +*** DotEnvWriter This simplest form of vaultification just creates a plaintext file with the form of -```bash +#+BEGIN_SRC export Key1=Value1 [...] export KeyN=ValueN -``` +#+END_SRC for all N keys in the provided dictionary. Below are environment variables, that are needed by this consumer: -```bash +#+BEGIN_SRC # this controls the location of the dotenv file export VAULTIFY_DESTFILE=/a/path/to/where/secrets.env -``` +#+END_SRC -#### JsonWriter +*** JsonWriter This consumer is very similar to the `DotEnvWriter`, but produces a json file instead. @@ -189,11 +186,13 @@ json file instead. Below are environment variables, that are needed by this consumer: ```bash +#+BEGIN_SRC +#+END_SRC # this controls the location of the dotenv file export VAULTIFY_DESTFILE=/a/path/to/where/secrets.json ``` -#### EnvRunner +*** EnvRunner If you want to just execute a process with some secrets, then `EnvRunner` consumer will run a subprocess with an enriched @@ -208,9 +207,9 @@ accidentally persist after a crash Below are environment variables, that are needed by this consumer: -```bash +#+BEGIN_SRC # this controls the invocation of the target process. export VAULTIFY_TARGET='/a/path/where/a/secret/hungry/binary --with-some flag wants-execution' -``` +#+END_SRC Currently `EnvRunner` does not support interactive commands. From 446c72a773a490cd68d6f1d507d4e7a125f344c2 Mon Sep 17 00:00:00 2001 From: gve Date: Sat, 20 Apr 2019 17:59:32 +0200 Subject: [PATCH 2/9] update readme --- README.org | 196 ++++------------------------------------------------- 1 file changed, 12 insertions(+), 184 deletions(-) diff --git a/README.org b/README.org index 723598e..0d63c18 100644 --- a/README.org +++ b/README.org @@ -1,7 +1,15 @@ -#+TITLE: vaultify usage +#+TITLE: vaultify - hexagon of secrets #+OPTIONS: #+AUTHOR: Georg vom Endt (krysopath@gmail.com) +* Introduction +Vaultify is secret provisioning method. It can be used to read/write +from/to resources and pass environment values to spawned processes. + +Originally a simple bash script fetching from HashiCorp Vault and +writing as dotenv file, it quickly became an API for providing secrets +from any arbitrary URI to consume the secrets via another arbitrary URI + * Installation This section explains different installation methods. ** Requirements @@ -27,189 +35,9 @@ These tools can greatly speed up local development, when used in conjunction via ~make~, so install them. They can overwhelm you with output, too. -*** ubuntu +*** debian #+BEGIN_SRC shell -apt install make entr yamllint pylint bandit -#+END_SRC - -* usage - -** first steps - -1. either create a virtualenv and install requirements.txt or -2. run `docker build .` - -Before using `vaultify` you need to export these into your environment: - -#+BEGIN_SRC -export VAULTIFY_PROVIDER= -export VAULTIFY_CONSUMER= -export VAULTIFY_LOGLEVEL= # default is info -#+END_SRC - ->**r**ead **t**he **f**riendly **m**anual for the chosen Provider & Consumer, too - -Then just run: -#+BEGIN_SRC -./entry.py -#+END_SRC - - - -** feature overview - -In this table you find an info about which Provider/Consumer -pairs are supported: - -| pairs | GPG | OpenSSL | Vault | -|--------------+-----+---------+-------| -| DotEnvWriter | y | y | y | -| EnvRunner | y | y | y | -| JsonWriter | y | y | y | - - -Assuming the pattern holds, we expect always full compatibility -between any Provider/Consumer pair. - -** providers - -Providers are all classes, that create some `vaultify`-compliant dictionary, -which then is used by `vaultify` consumers. - -*** GPGProvider - -This Provider can decrypt symmetrically encrypted files, created with `gpg`. - -Adhere to this format: -#+BEGIN_SRC -KEY1=VAL1 -[...] -KEYN=VALN -#+END_SRC - -To encrypt such a file, execute: -#+BEGIN_SRC -gpg --symmetric -#+END_SRC - -Below are environment variables, that are needed by this provider: - -#+BEGIN_SRC -# this will be used as the passphrase to gpg -export VAULTIFY_SECRET= -#+END_SRC - -*** OpenSSLProvider - -This provider can decrypt symmetrically encrypted files, created with `openssl` - -Adhere to this format: -#+BEGIN_SRC -KEY1=VAL1 -[...] -KEYN=VALN -#+END_SRC - -To encrypt such a file, execute: -```bash -#+BEGIN_SRC -openssl enc -aes-256-cbc -salt -a -in -out .enc -#+END_SRC -``` - -> Do not use aes-256-cbc, if there is aes-256-gcm available in your openssl. -This prevents Padding Oracle attacks against the cipher text. Currently -setting the aes cipher is not possible in `vaultify` but will be made, when -the default openssl library ships with AEAD compiled. If your OpenSSL CLI -supports aes-256-gcm, please file a bug report against vaultify. - -Below are environment variables, that are needed by this provider: - -```bash -#+BEGIN_SRC -#+END_SRC -# this will be used as the passphrase to openssl -export VAULTIFY_SECRET= -``` - -*** VaultProvider - -This provider fetches secrets from HashiCorp Vault API. - - -Below are environment variables, that are needed by this provider: - -#+BEGIN_SRC -# set this to a reachable vault API -export VAULT_ADDR= -# set this to nodes in vaults kv engine, where you do have perms for READ -export VAULT_PATHS= - -# if you do not set $VAULTIFY_SECRET, then -export VAULT_TOKEN= -#+END_SRC - -`VaultProvider` will use `VAULTIFY_SECRET` or `VAULT_TOKEN` for authentication, -in that order. - -** consumers - -are all classes that operate on a `vaultify` compliant dictionary, to -**somehow** use the secrets in there for the greater good. - -*** DotEnvWriter - -This simplest form of vaultification just creates a plaintext file with -the form of - -#+BEGIN_SRC -export Key1=Value1 -[...] -export KeyN=ValueN -#+END_SRC - -for all N keys in the provided dictionary. - -Below are environment variables, that are needed by this consumer: - -#+BEGIN_SRC -# this controls the location of the dotenv file -export VAULTIFY_DESTFILE=/a/path/to/where/secrets.env -#+END_SRC - - -*** JsonWriter - -This consumer is very similar to the `DotEnvWriter`, but produces a -json file instead. - -Below are environment variables, that are needed by this consumer: - -```bash -#+BEGIN_SRC -#+END_SRC -# this controls the location of the dotenv file -export VAULTIFY_DESTFILE=/a/path/to/where/secrets.json -``` - -*** EnvRunner - -If you want to just execute a process with some secrets, then -`EnvRunner` consumer will run a subprocess with an enriched -environment for you. - ->In that sense `EnvRunner` doubles as an entry point for docker runtimes. - -Choose this, if you want to prevent any kind of secret persistence. - -> one might not like having docker `tmpfs` volumes swapped or -accidentally persist after a crash - -Below are environment variables, that are needed by this consumer: - -#+BEGIN_SRC -# this controls the invocation of the target process. -export VAULTIFY_TARGET='/a/path/where/a/secret/hungry/binary --with-some flag wants-execution' +apt-get install make entr yamllint pylint bandit +pip3 install black #+END_SRC -Currently `EnvRunner` does not support interactive commands. From ca158d332e75cf8b102fce8ebf2ae3206cf24339 Mon Sep 17 00:00:00 2001 From: gve Date: Sat, 20 Apr 2019 18:15:18 +0200 Subject: [PATCH 3/9] cleanup old files change gitignore --- assets/.gitignore | 1 + version | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) delete mode 100644 version diff --git a/assets/.gitignore b/assets/.gitignore index 73c00f8..db551d0 100644 --- a/assets/.gitignore +++ b/assets/.gitignore @@ -1,2 +1,3 @@ *.enc *.gpg +*.plain diff --git a/version b/version deleted file mode 100644 index 227cea2..0000000 --- a/version +++ /dev/null @@ -1 +0,0 @@ -2.0.0 From d9783b0efc64995777e3c68bbfcbd3ca621f0111 Mon Sep 17 00:00:00 2001 From: gve Date: Sat, 20 Apr 2019 18:16:16 +0200 Subject: [PATCH 4/9] change makefile to remove dist dirs --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 29bd103..6ff35f2 100644 --- a/Makefile +++ b/Makefile @@ -30,7 +30,7 @@ artifact/tag: vaultify:$(TAG) clean: - rm -rf tests/new* assets/* + rm -rf tests/new* assets/* build/ dist *vaultify.egg* run/tests: clean cp tests/test-config.env assets/secrets.plain From aae2483a039296937dc5c7c52727257c8429d25c Mon Sep 17 00:00:00 2001 From: gve Date: Tue, 23 Apr 2019 09:16:21 +0200 Subject: [PATCH 5/9] hasty changes --- Makefile | 9 ++++++--- README.rst | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ setup.cfg | 2 +- setup.py | 4 ++-- 4 files changed, 66 insertions(+), 6 deletions(-) create mode 100644 README.rst diff --git a/Makefile b/Makefile index 6ff35f2..8a25dd5 100644 --- a/Makefile +++ b/Makefile @@ -11,9 +11,12 @@ ci: run/tests\ dev/ci: ls vaultify/*py Dockerfile .vaultify.yml | entr -s 'make run/tests' -artifact/pkg: +artifact/pkg: clean python3 setup.py sdist bdist_wheel +pypi: artifact/pkg + twine upload --repository-url https://test.pypi.org/legacy/ dist/* + artifact/docker: docker build \ --build-arg BASE_IMAGE=$(BASE_IMAGE)\ @@ -53,13 +56,13 @@ run/tests: clean manual: @groff -man -Tascii man/vaultify.1 -dev/install/packaging: +dev/install/tools: python3\ -m pip\ install\ --user\ --upgrade\ - pip setuptools wheel + pip setuptools wheel twine black dev/install/os: sudo apt-get install gnupg openssl diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..b8007a6 --- /dev/null +++ b/README.rst @@ -0,0 +1,57 @@ +============================= +vaultify - hexagon of secrets +============================= + + :Author: Georg vom Endt (krysopath@gmail.com) + +.. contents:: + + + +1 Installation +-------------- + +This section explains different installation methods. + +1.1 Requirements +~~~~~~~~~~~~~~~~ + +- python3.6, python3.7 + +- pip + +- docker (obsoletes the above) + +1.2 from pypi +~~~~~~~~~~~~~ + +Most implementers will fetch the package from pypi and build their own +things on another layer. + +.. code:: shell + + pip3 install vaultify + +1.3 from git +~~~~~~~~~~~~ + +Alternatively fetch the sources from github to develop your own +adapter classes. + +.. code:: shell + + git clone git@github.com:krysopath/vaultify.git + +1.4 os dependencies +~~~~~~~~~~~~~~~~~~~ + +These tools can greatly speed up local development, when used in +conjunction via ``make``, so install them. They can overwhelm you with +output, too. + +1.4.1 ubuntu +^^^^^^^^^^^^ + +.. code:: shell + + apt install make entr yamllint pylint bandit diff --git a/setup.cfg b/setup.cfg index b88034e..5aef279 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,2 +1,2 @@ [metadata] -description-file = README.md +description-file = README.rst diff --git a/setup.py b/setup.py index 38c8705..13de0cd 100644 --- a/setup.py +++ b/setup.py @@ -7,12 +7,12 @@ setup( name='vaultify', packages=['vaultify'], - version='2.0.0', + version='0.1.2', description='A hexagon of secret provisoning', author='Georg vom Endt', author_email='krysopath@gmail.com', url='https://github.com/krysopath/vaultify', - download_url='https://github.com/author/repo/tarball/2.0.0', + download_url='https://github.com/krysopath/vaultify/tarball/0.1.1', keywords=['vault', 'gpg', 'secrets', 'docker', 'cli'], entry_points={ 'console_scripts': [ From 9612793d51138ed27aa6423b3735c2e41ad62ab3 Mon Sep 17 00:00:00 2001 From: gve Date: Tue, 23 Apr 2019 09:29:55 +0200 Subject: [PATCH 6/9] making black work some test commit apply black to complete all lol fix black for reals --- .githooks/pre-commit.d/00-black | 14 +++++++++++++- runtests.py | 21 +++++++++++---------- setup.py | 24 ++++++++++-------------- vaultify/consumers.py | 4 ++-- vaultify/vaultify.py | 10 +++++----- 5 files changed, 41 insertions(+), 32 deletions(-) diff --git a/.githooks/pre-commit.d/00-black b/.githooks/pre-commit.d/00-black index 21cfde7..d6769cc 100755 --- a/.githooks/pre-commit.d/00-black +++ b/.githooks/pre-commit.d/00-black @@ -1 +1,13 @@ -black ./vaultify +#!/bin/sh + +# Stash unstaged changes +git stash -q --keep-index + +black \ + --exclude\ + . + +# Stage updated files +git add -u +# Re-apply original unstaged changes +git stash pop -q diff --git a/runtests.py b/runtests.py index 467fb25..698be34 100644 --- a/runtests.py +++ b/runtests.py @@ -5,19 +5,20 @@ import os files = [] -root_dir = 'vaultify/' +root_dir = "vaultify/" for root, _, filenames in os.walk(root_dir): - for filename in filenames: - if (filename == '__init__.py' - or filename[-3:] != '.py' - or filename.startswith('.#') - ): + for filename in filenames: + if ( + filename == "__init__.py" + or filename[-3:] != ".py" + or filename.startswith(".#") + ): continue - f = os.path.join(root, filename) - f = f.replace('/', '.') - f = f[:-3] - files.append(f) + f = os.path.join(root, filename) + f = f.replace("/", ".") + f = f[:-3] + files.append(f) suite = unittest.TestSuite() for module in files: diff --git a/setup.py b/setup.py index 13de0cd..3fd07de 100644 --- a/setup.py +++ b/setup.py @@ -5,19 +5,15 @@ from setuptools import setup setup( - name='vaultify', - packages=['vaultify'], - version='0.1.2', - description='A hexagon of secret provisoning', - author='Georg vom Endt', - author_email='krysopath@gmail.com', - url='https://github.com/krysopath/vaultify', - download_url='https://github.com/krysopath/vaultify/tarball/0.1.1', - keywords=['vault', 'gpg', 'secrets', 'docker', 'cli'], - entry_points={ - 'console_scripts': [ - 'vaultify=vaultify.vaultify:main', - ], - }, + name="vaultify", + packages=["vaultify"], + version="0.1.2", + description="A hexagon of secret provisoning", + author="Georg vom Endt", + author_email="krysopath@gmail.com", + url="https://github.com/krysopath/vaultify", + download_url="https://github.com/krysopath/vaultify/tarball/0.1.1", + keywords=["vault", "gpg", "secrets", "docker", "cli"], + entry_points={"console_scripts": ["vaultify=vaultify.vaultify:main"]}, classifiers=[], ) diff --git a/vaultify/consumers.py b/vaultify/consumers.py index 93757af..2b0b79a 100644 --- a/vaultify/consumers.py +++ b/vaultify/consumers.py @@ -117,14 +117,14 @@ class EnvRunner(Consumer): """ This Consumer will update the environment and then run a subprocess in that altered environment. - + We carry our local environment over into the spawned process: >>> os.environ.update({"K1": "V1"}) >>> EnvRunner('./tests/echo-vars.sh').consume_secrets({"K2":"V2","K3":"V3"}) K1=V1 K2=V2 K3=V3 - + We fail when the command can not be found: >>> EnvRunner('nowhere.sh').consume_secrets({"K1":"V1"}) Traceback (most recent call last): diff --git a/vaultify/vaultify.py b/vaultify/vaultify.py index 2d38792..f90415f 100644 --- a/vaultify/vaultify.py +++ b/vaultify/vaultify.py @@ -29,7 +29,7 @@ class Vaultify(API): >>> from . import consumers, providers >>> vfy = Vaultify( - ... consumer=consumers.EnvRunner('env'), + ... consumer=consumers.EnvRunner('env'), ... provider=providers.GPGProvider('abc') ... ) >>> isinstance(vfy, Vaultify) @@ -48,12 +48,12 @@ def consume_secrets(self, data: dict) -> bool: def validate(self) -> t.Iterable: """ >>> vfy = Vaultify( - ... consumer="fancystring", + ... consumer="fancystring", ... provider={1: 2} ... ) >>> isinstance(vfy.validate(), list) True - >>> + >>> """ results = [] @@ -94,10 +94,10 @@ def factory(config_dict: dict, **kwargs) -> Vaultify: :param config_dict: an initialised cfg dict :param kwargs: user passed kwargs :return: - + >>> from . import configure >>> isinstance( - ... factory(configure()), + ... factory(configure()), ... Vaultify ... ) True From 4e4cbfafa4f0954ae422e4a8d2e4400b50d7b1fc Mon Sep 17 00:00:00 2001 From: gve Date: Tue, 23 Apr 2019 09:31:59 +0200 Subject: [PATCH 7/9] test black --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 3fd07de..a42ba31 100644 --- a/setup.py +++ b/setup.py @@ -11,7 +11,7 @@ description="A hexagon of secret provisoning", author="Georg vom Endt", author_email="krysopath@gmail.com", - url="https://github.com/krysopath/vaultify", + url='https://github.com/krysopath/vaultify', download_url="https://github.com/krysopath/vaultify/tarball/0.1.1", keywords=["vault", "gpg", "secrets", "docker", "cli"], entry_points={"console_scripts": ["vaultify=vaultify.vaultify:main"]}, From 1223c72bbb13fabd0966ac66eb01eb8110b00607 Mon Sep 17 00:00:00 2001 From: gve Date: Tue, 23 Apr 2019 09:33:12 +0200 Subject: [PATCH 8/9] test black --- .githooks/pre-commit.d/00-black | 2 +- setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.githooks/pre-commit.d/00-black b/.githooks/pre-commit.d/00-black index d6769cc..3d3c351 100755 --- a/.githooks/pre-commit.d/00-black +++ b/.githooks/pre-commit.d/00-black @@ -4,7 +4,7 @@ git stash -q --keep-index black \ - --exclude\ + --exclude venv\ . # Stage updated files diff --git a/setup.py b/setup.py index a42ba31..3fd07de 100644 --- a/setup.py +++ b/setup.py @@ -11,7 +11,7 @@ description="A hexagon of secret provisoning", author="Georg vom Endt", author_email="krysopath@gmail.com", - url='https://github.com/krysopath/vaultify', + url="https://github.com/krysopath/vaultify", download_url="https://github.com/krysopath/vaultify/tarball/0.1.1", keywords=["vault", "gpg", "secrets", "docker", "cli"], entry_points={"console_scripts": ["vaultify=vaultify.vaultify:main"]}, From cb5b5550725fbdb605041f9bd9ab7d2e0e4fb590 Mon Sep 17 00:00:00 2001 From: gve Date: Tue, 23 Apr 2019 09:40:45 +0200 Subject: [PATCH 9/9] add usage part --- docs/usage.org | 215 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 docs/usage.org diff --git a/docs/usage.org b/docs/usage.org new file mode 100644 index 0000000..723598e --- /dev/null +++ b/docs/usage.org @@ -0,0 +1,215 @@ +#+TITLE: vaultify usage +#+OPTIONS: +#+AUTHOR: Georg vom Endt (krysopath@gmail.com) + +* Installation +This section explains different installation methods. +** Requirements + - python3.6, python3.7 + - pip + - docker (obsoletes the above) + +** from pypi +Most implementers will fetch the package from pypi and build their own +things on another layer. +#+BEGIN_SRC shell +pip3 install vaultify +#+END_SRC + +** from git +Alternatively fetch the sources from github to develop your own +adapter classes. +#+BEGIN_SRC shell +git clone git@github.com:krysopath/vaultify.git +#+END_SRC +** os dependencies +These tools can greatly speed up local development, when used in +conjunction via ~make~, so install them. They can overwhelm you with +output, too. + +*** ubuntu +#+BEGIN_SRC shell +apt install make entr yamllint pylint bandit +#+END_SRC + +* usage + +** first steps + +1. either create a virtualenv and install requirements.txt or +2. run `docker build .` + +Before using `vaultify` you need to export these into your environment: + +#+BEGIN_SRC +export VAULTIFY_PROVIDER= +export VAULTIFY_CONSUMER= +export VAULTIFY_LOGLEVEL= # default is info +#+END_SRC + +>**r**ead **t**he **f**riendly **m**anual for the chosen Provider & Consumer, too + +Then just run: +#+BEGIN_SRC +./entry.py +#+END_SRC + + + +** feature overview + +In this table you find an info about which Provider/Consumer +pairs are supported: + +| pairs | GPG | OpenSSL | Vault | +|--------------+-----+---------+-------| +| DotEnvWriter | y | y | y | +| EnvRunner | y | y | y | +| JsonWriter | y | y | y | + + +Assuming the pattern holds, we expect always full compatibility +between any Provider/Consumer pair. + +** providers + +Providers are all classes, that create some `vaultify`-compliant dictionary, +which then is used by `vaultify` consumers. + +*** GPGProvider + +This Provider can decrypt symmetrically encrypted files, created with `gpg`. + +Adhere to this format: +#+BEGIN_SRC +KEY1=VAL1 +[...] +KEYN=VALN +#+END_SRC + +To encrypt such a file, execute: +#+BEGIN_SRC +gpg --symmetric +#+END_SRC + +Below are environment variables, that are needed by this provider: + +#+BEGIN_SRC +# this will be used as the passphrase to gpg +export VAULTIFY_SECRET= +#+END_SRC + +*** OpenSSLProvider + +This provider can decrypt symmetrically encrypted files, created with `openssl` + +Adhere to this format: +#+BEGIN_SRC +KEY1=VAL1 +[...] +KEYN=VALN +#+END_SRC + +To encrypt such a file, execute: +```bash +#+BEGIN_SRC +openssl enc -aes-256-cbc -salt -a -in -out .enc +#+END_SRC +``` + +> Do not use aes-256-cbc, if there is aes-256-gcm available in your openssl. +This prevents Padding Oracle attacks against the cipher text. Currently +setting the aes cipher is not possible in `vaultify` but will be made, when +the default openssl library ships with AEAD compiled. If your OpenSSL CLI +supports aes-256-gcm, please file a bug report against vaultify. + +Below are environment variables, that are needed by this provider: + +```bash +#+BEGIN_SRC +#+END_SRC +# this will be used as the passphrase to openssl +export VAULTIFY_SECRET= +``` + +*** VaultProvider + +This provider fetches secrets from HashiCorp Vault API. + + +Below are environment variables, that are needed by this provider: + +#+BEGIN_SRC +# set this to a reachable vault API +export VAULT_ADDR= +# set this to nodes in vaults kv engine, where you do have perms for READ +export VAULT_PATHS= + +# if you do not set $VAULTIFY_SECRET, then +export VAULT_TOKEN= +#+END_SRC + +`VaultProvider` will use `VAULTIFY_SECRET` or `VAULT_TOKEN` for authentication, +in that order. + +** consumers + +are all classes that operate on a `vaultify` compliant dictionary, to +**somehow** use the secrets in there for the greater good. + +*** DotEnvWriter + +This simplest form of vaultification just creates a plaintext file with +the form of + +#+BEGIN_SRC +export Key1=Value1 +[...] +export KeyN=ValueN +#+END_SRC + +for all N keys in the provided dictionary. + +Below are environment variables, that are needed by this consumer: + +#+BEGIN_SRC +# this controls the location of the dotenv file +export VAULTIFY_DESTFILE=/a/path/to/where/secrets.env +#+END_SRC + + +*** JsonWriter + +This consumer is very similar to the `DotEnvWriter`, but produces a +json file instead. + +Below are environment variables, that are needed by this consumer: + +```bash +#+BEGIN_SRC +#+END_SRC +# this controls the location of the dotenv file +export VAULTIFY_DESTFILE=/a/path/to/where/secrets.json +``` + +*** EnvRunner + +If you want to just execute a process with some secrets, then +`EnvRunner` consumer will run a subprocess with an enriched +environment for you. + +>In that sense `EnvRunner` doubles as an entry point for docker runtimes. + +Choose this, if you want to prevent any kind of secret persistence. + +> one might not like having docker `tmpfs` volumes swapped or +accidentally persist after a crash + +Below are environment variables, that are needed by this consumer: + +#+BEGIN_SRC +# this controls the invocation of the target process. +export VAULTIFY_TARGET='/a/path/where/a/secret/hungry/binary --with-some flag wants-execution' +#+END_SRC + +Currently `EnvRunner` does not support interactive commands.