From e87e18fd7a1cd3a5a28abb0cc5672546a4a72925 Mon Sep 17 00:00:00 2001 From: "Rainer M. Canavan" Date: Wed, 13 May 2020 17:52:12 +0200 Subject: [PATCH 01/47] document $FLAT_DEBUG_AUTH/$FLAT_DEBUG_ALLOW_HEADER --- administration/configuration.md | 3 ++- reference/debugging.md | 7 +++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/administration/configuration.md b/administration/configuration.md index ed909c7..7e033c7 100644 --- a/administration/configuration.md +++ b/administration/configuration.md @@ -11,7 +11,8 @@ The chapter [Defining Env Vars](/cookbook/envvars.md#defining-env-vars) in the C * `FLAT_SERVER_ROLE`: This setting is typically used to distinguish production systems from staging or development servers. Its value is accessible in the [`$server` variable](/reference/variables.md#predefined-variables) as `$server/role`. * `FLAT_STATUS_AUTH`: Username and password, separated by a `:` for access to the `php-fpm` and `httpd` status pages. The pages are disabled entirely if `FLAT_STATUS_AUTH` is not set. If enabled, the `httpd` status can be requested via HTTP at `/ServerStatus`, and the php-fpm status at `/FPMStatus?full`. - +* `FLAT_DEBUG_ALLOW_HEADER`: enable [per request debugging](/reference/debugging.md), defaults to `false` unless `FLAT_DEBUG_AUTH` is set. +* `FLAT_DEBUG_AUTH`: sets a password to protect [per request debugging](/reference/debugging.md). ### Request Timeouts diff --git a/reference/debugging.md b/reference/debugging.md index 79ccd52..80c1502 100644 --- a/reference/debugging.md +++ b/reference/debugging.md @@ -32,8 +32,11 @@ sending the desired parameters in the HTTP `Debug` header field, for example: $ curl -H "Debug: request:info:append" localhost:8080 ``` -Per request debugging may be password-protected via the `$FLAT_DEBUG_AUTH` environment variable. -If such authorization is required, the password has to be appended as `;auth=…` in the `Debug` header, for example: +To use per request debugging, you must enable it by setting the +`$FLAT_DEBUG_ALLOW_HEADER` environment variable to `true` or by turning on +password protection via the `$FLAT_DEBUG_AUTH` environment variable. +If authorization is required, the password has to be appended as `;auth=…` +in the `Debug` header, for example: ```bash $ curl -H "Debug: flow::append; auth=Pas5W0Rd" localhost:8080 From 520101f3fdd31625ef30327cb3f29648ef72cc11 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Afflerbach?= Date: Wed, 20 May 2020 12:38:51 +0200 Subject: [PATCH 02/47] Tutorial update, cosmetics --- administration/configuration.md | 6 +++--- tutorial/README.md | 5 ++++- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/administration/configuration.md b/administration/configuration.md index 7e033c7..49245af 100644 --- a/administration/configuration.md +++ b/administration/configuration.md @@ -1,7 +1,7 @@ # Configuration -Certain settings of the FLAT runner can be configured using environment variables. +Certain settings of the FLAT server can be configured using environment variables. The chapter [Defining Env Vars](/cookbook/envvars.md#defining-env-vars) in the Cookbook provides information on how to set environment variables. @@ -11,8 +11,8 @@ The chapter [Defining Env Vars](/cookbook/envvars.md#defining-env-vars) in the C * `FLAT_SERVER_ROLE`: This setting is typically used to distinguish production systems from staging or development servers. Its value is accessible in the [`$server` variable](/reference/variables.md#predefined-variables) as `$server/role`. * `FLAT_STATUS_AUTH`: Username and password, separated by a `:` for access to the `php-fpm` and `httpd` status pages. The pages are disabled entirely if `FLAT_STATUS_AUTH` is not set. If enabled, the `httpd` status can be requested via HTTP at `/ServerStatus`, and the php-fpm status at `/FPMStatus?full`. -* `FLAT_DEBUG_ALLOW_HEADER`: enable [per request debugging](/reference/debugging.md), defaults to `false` unless `FLAT_DEBUG_AUTH` is set. -* `FLAT_DEBUG_AUTH`: sets a password to protect [per request debugging](/reference/debugging.md). +* `FLAT_DEBUG_ALLOW_HEADER`: enable [per request debugging](/reference/debugging.md#request-debugging), defaults to `false` unless `FLAT_DEBUG_AUTH` is set. +* `FLAT_DEBUG_AUTH`: sets a password to protect [per request debugging](/reference/debugging.md#request-debugging). ### Request Timeouts diff --git a/tutorial/README.md b/tutorial/README.md index 8f5f6b2..e7dbb8b 100644 --- a/tutorial/README.md +++ b/tutorial/README.md @@ -1045,10 +1045,13 @@ If we also set the debug sink to `inline` or `append`, the output will be included in the HTTP response rather than the log file, for example: ```bash -$ curl --header Debug:time:info:inline localhost:8080/html +$ curl --header Debug:time:debug:inline localhost:8080/html ``` The latter is the only reasonable way to debug on a production system where we usually can't access the log file. +Header debugging is enabled by default in the [`flat` command line tool](/reference/flat-cli.md), +but has to be explicitly enabled in other environments, usually by setting +[`$FLAT_DEBUG_AUTH`](/administration/configuration.md). > 📎 > If the `$FLAT_DEBUG_AUTH` environment variable is set (this is a **must** on production systems!), FLAT requires a password to enable debugging by means of the `Debug` header, for example `--header 'Debug: *:warn:append; auth=Pa5sw0rd'`. From fa2679b58a8b523be6cd1b070305ef0b69bb8819 Mon Sep 17 00:00:00 2001 From: "Rainer M. Canavan" Date: Wed, 20 May 2020 16:58:52 +0200 Subject: [PATCH 03/47] not explicit #47211-debug-password-default --- tutorial/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorial/README.md b/tutorial/README.md index e7dbb8b..aadd160 100644 --- a/tutorial/README.md +++ b/tutorial/README.md @@ -1049,8 +1049,8 @@ $ curl --header Debug:time:debug:inline localhost:8080/html ``` The latter is the only reasonable way to debug on a production system where we usually can't access the log file. -Header debugging is enabled by default in the [`flat` command line tool](/reference/flat-cli.md), -but has to be explicitly enabled in other environments, usually by setting +While the [`flat` command line tool](/reference/flat-cli.md) enables header debugging by default, +the appropriate environment variables must be set to enable it in other environments, typically [`$FLAT_DEBUG_AUTH`](/administration/configuration.md). > 📎 From aa8ef2309d234b9a547aa0e5614803fee1ee13d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Afflerbach?= Date: Fri, 22 May 2020 10:49:31 +0200 Subject: [PATCH 04/47] Changelog entry from source repository --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f622d91..b281aae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased +### Added + +- [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#request-debugging) to enable debugging using the `Debug` request header, defaults to `false` + ## [20200519](https://hub.docker.com/r/sevenvaltechnologies/flatrunner/tags) ### Added From 7d7a804e133c51b825b6f166b0d9af524793d9a7 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Tue, 26 May 2020 16:07:08 +0200 Subject: [PATCH 05/47] new cookbook: x-flat-jwt with some references --- cookbook/jwt.md | 5 +- cookbook/x-flat-jwt.md | 324 ++++++++++++++++++++++++++++++++++ reference/OpenAPI/security.md | 4 + 3 files changed, 331 insertions(+), 2 deletions(-) create mode 100644 cookbook/x-flat-jwt.md diff --git a/cookbook/jwt.md b/cookbook/jwt.md index 1a9a24c..e79b5d5 100644 --- a/cookbook/jwt.md +++ b/cookbook/jwt.md @@ -101,5 +101,6 @@ Note that, with `RSASSA` based algorithms, you have to specify the algorithm in ## See also -* [`jwt-encode()`](/reference/functions/jwt-encode.md) -* [`jwt-decode()`](/reference/functions/jwt-decode.md) +* [`jwt-encode()`](/reference/functions/jwt-encode.md) (reference) +* [`jwt-decode()`](/reference/functions/jwt-decode.md) (reference) +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md new file mode 100644 index 0000000..7bc7b25 --- /dev/null +++ b/cookbook/x-flat-jwt.md @@ -0,0 +1,324 @@ +# Protecting Access using JWT Tokens + +Imagine, you have a [proxy](/reference/OpenAPI/routing.md#assigning-flat-proxies) to an API, e.g. httpbin.org. + +swagger.yml: +```yaml +swagger: "2.0" +basePath: / +paths: + /httpbin/**: + x-flat-proxy: + origin: https://httpbin.org + stripEndpoint: true +``` + +Sending a request to FLAT running on localhost port 1234 results in: + +``` +curl -i http://localhost:1234/httpbin/anything +HTTP/1.1 200 OK +… +Content-Type: application/json + +{ + "args": {}, + "data": "", + "files": {}, + "form": {}, + "headers": { + "Accept": "*/*", + "Accept-Encoding": "deflate, gzip", + "Host": "httpbin.org", + "User-Agent": "curl/7.29.0", + "X-Amzn-Trace-Id": "…" + }, + "json": null, + "method": "GET", + "origin": "…", + "url": "https://httpbin.org/anything" +} +``` + +That's what you would expect from httpbin.org, right? + +## Restricting access: Swagger Security and `x-flat-jwt` + +Now, you don't want everyone, but only authorized users to use this proxy. This is typically achieved with access tokens. Some tokens are JSON Web Tokens ([JWT](https://tools.ietf.org/html/rfc7519)), while others are opaque. + +Swagger has a two-part feature to describe protected access to routes: `securityDefinitions` (what sort of protection is applied …) and `security` (… to which routes), e.g.: + +```yaml +swagger: "2.0" +basePath: / +securityDefinitions: + JWTCookie: + type: apiKey + in: header + name: Cookie +paths: + /httpbin/**: + security: + - JWTCookie: [] +``` + +This defines a security scheme object (named `JWTCookie`), meaning that some sort of cookie is needed to access certain routes. This is applied to the [wildcard path](/reference/OpenAPI/differences.md#wildcard-paths) `/httpbin/**`. + +This documentation feature, with some extensions, is used to make FLAT actually allow access to the route only if a certain JWT token is presented. + +First, we specify the name of the cookie expected to accompany the API request: + +```yaml +… +securityDefinitions: + JWTCookie: + type: apiKey + in: header + name: Cookie + x-flat-cookiename: authtoken # ⬅ specify the cookie name +… +``` + +Then we give some [configuration](/reference/OpenAPI/security.md#the-x-flat-jwt-field) for decoding the JWT token: + +```yaml +… + JWTCookie: + type: apiKey + in: header + name: Cookie + x-flat-cookiename: authtoken + x-flat-jwt: # ⬅ specify the JWT configuration: + key: # ⬅ the key to decode the JWT … + file: secret.pem # ⬅ … is read from the file secret.pem + alg: RS256 # ⬅ the signing algorithm is RS256 +… +``` + +The specified key is a public key, read from the file secret.pem, e.g.: +``` +-----BEGIN PUBLIC KEY----- +MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGSd+sSTss2uOuVJKpumpFAaml +t1CWLMTAZNAabF71Ur0P6u833RhAIjXDSA/QeVitzvqvCZpNtbOJVegaREqLMJqv +FOUkFdLNRP3f9XjYFFvubo09tcjX6oGEREKDqLG2MfZ2Z8LVzuJc6SwZMgVFk/63 +rdAOci3W9u3zOSGj4QIDAQAB +-----END PUBLIC KEY----- +``` + +That's all. Here, only those tokens will be accepted that are signed with the RS256 algorithm in a way that they can be decoded using the given public key. + +(You may want to try this with the key and algorithm given to you by the party providing you with tokens, e.g. an OAuth2 authorization server.) + +Let's give it a try: + +``` +curl -i http://localhost:1234/httpbin/anything +HTTP/1.1 403 Forbidden +… +Content-Type: application/json + +{"error":{"message":"Security violation","status":403,"requestID":"Xsz@QAv8CEBZfWbmQhKLIQAAAII","info":["JWT Security (JWTCookie): No Cookie header sent"],"code":3206}} +``` + +Ah, yes, we forgot to present a token in the `authtoken` cookie. But we see, that the protection works. + +Let's try again with a token: + +``` +curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:1234/httpbin/anything +HTTP/1.1 403 Forbidden +… +Content-Type: application/json + +{"error":{"message":"Security violation","status":403,"requestID":"Xsz80Av8CEBZfWbmQhKLIAAAAIE","info":["JWT Security (JWTCookie): Invalid JWT: Token has expired."],"code":3206}} +``` + +Hmm, expired. So this one is too old. (Access tokens typically have a restricted period of use.) + +OK, let's use another token: + +``` +curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:1234/httpbin/anything +HTTP/1.1 200 OK +… +Content-Type: application/json + +{ + "args": {}, + "data": "", + "files": {}, + "form": {}, + "headers": { + "Accept": "*/*", + "Accept-Encoding": "deflate, gzip", + "Host": "httpbin.org", + "User-Agent": "curl/7.29.0", + "X-Amzn-Trace-Id": "…" + }, + "json": null, + "method": "GET", + "origin": "…", + "url": "https://httpbin.org/anything" +} +``` + +Tada! + +By the way, apart from Cookies, this also works similarly with the `Authorization: Bearer …` header: + +``` +… +securityDefinitions: + JWTBearer: + type: apiKey + in: header + name: Authorization # ⬅ + x-flat-jwt: + key: + file: secret.pem + alg: RS256 +paths: + /httpbin/**: + security: + - JWTBearer: [] +``` + +You can try that with + +``` +curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:1234/httpbin/anything +``` + +But there are two additional features that can be quite handy: `out-var` and `out-header`. + +## Storing JWT claims: `out-var` + +With `out-var` you can specify the name of the [variable](/reference/variables.md) where to store the JSON claims encoded in the JWT, in order to make them available to further processing. E.g. + + +```yaml +… + x-flat-jwt: + key: + file: secret.pem + alg: RS256 + out-var: $the_claims # ⬅ +… +``` + +We can log the claims by adding a `log` action to an [init flow](/reference/OpenAPI/routing.md#init-flow): + +```yaml +swagger: "2.0" +… +x-flat-init: init.xml +paths: +… +``` + +with init.xml: + +```xml + + + { + "JWT-Claims": {{ $the_claims }} + } + + +``` + +If you look at the [FLAT logs](/administration/logging.md) and try again with a valid token, you'll notice: + +``` +{…,"type":"flat_access",…,"JWT-Claims":{"sub":"some_user","iss":"some_provider"}} +``` + +Here you see the two claims from the JWT token. + +## Sending JWT claims upstream: `out-header` + +With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you can send the whole set of claims from the JWT upstream: + +```yaml: +… + x-flat-jwt: + key: + file: secret.pem + alg: RS256 + out-var: $the_claims + out-header: JWT # ⬅ the name of the request header with the JWT claims +… +``` + +``` +curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:1234/httpbin/anything +HTTP/1.1 200 OK +… +Content-Type: application/json + +{ +… + "headers": { +… + "Jwt": "{\"sub\":\"some_user\",\"iss\":\"some_provider\"}", +… + }, +… +} +``` + +## All files together + +swagger.yaml: +```yaml +swagger: "2.0" +basePath: / +securityDefinitions: + JWTCookie: + type: apiKey + in: header + name: Cookie + x-flat-cookiename: authtoken + x-flat-jwt: + key: + file: secret.pem + alg: RS256 + out-var: $the_claims + out-header: JWT +x-flat-init: init.xml +paths: + /httpbin/**: + security: + - JWTCookie: [] + x-flat-proxy: + origin: https://httpbin.org + stripEndpoint: true +``` + +secret.pem: +``` +-----BEGIN PUBLIC KEY----- +MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGSd+sSTss2uOuVJKpumpFAaml +t1CWLMTAZNAabF71Ur0P6u833RhAIjXDSA/QeVitzvqvCZpNtbOJVegaREqLMJqv +FOUkFdLNRP3f9XjYFFvubo09tcjX6oGEREKDqLG2MfZ2Z8LVzuJc6SwZMgVFk/63 +rdAOci3W9u3zOSGj4QIDAQAB +-----END PUBLIC KEY----- +``` + +init.xml: +```xml + + + { + "JWT-Claims": {{ $the_claims }} + } + + +``` + +## See also + +* [FLAT Security](/reference/OpenAPI/security.md) (reference) +* [Routing: Init Flow and FLAT Proxies](/reference/OpenAPI/routing.md) (reference) +* [Encoding and Decoding JWT](jwt.md) (cookbook) diff --git a/reference/OpenAPI/security.md b/reference/OpenAPI/security.md index 58aa7a7..279084d 100644 --- a/reference/OpenAPI/security.md +++ b/reference/OpenAPI/security.md @@ -226,3 +226,7 @@ paths: security: - JWTHeaderAuth: [] ``` + +## See also + +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) From 73ef7fbaad26dae5068c60cb42bc30e0bc2d87be Mon Sep 17 00:00:00 2001 From: "Rainer M. Canavan" Date: Tue, 26 May 2020 20:22:07 +0200 Subject: [PATCH 06/47] minor improvements for #47297-x-flat-jwt-cookbook --- cookbook/x-flat-jwt.md | 18 +++++++++--------- reference/OpenAPI/security.md | 6 ++++-- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 7bc7b25..895e7cd 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -1,6 +1,6 @@ # Protecting Access using JWT Tokens -Imagine, you have a [proxy](/reference/OpenAPI/routing.md#assigning-flat-proxies) to an API, e.g. httpbin.org. +Imagine you had a [proxy](/reference/OpenAPI/routing.md#assigning-flat-proxies) to an API, e.g. httpbin.org. swagger.yml: ```yaml @@ -44,7 +44,7 @@ That's what you would expect from httpbin.org, right? ## Restricting access: Swagger Security and `x-flat-jwt` -Now, you don't want everyone, but only authorized users to use this proxy. This is typically achieved with access tokens. Some tokens are JSON Web Tokens ([JWT](https://tools.ietf.org/html/rfc7519)), while others are opaque. +Now, you don't want anyone except authorized users to use this proxy. This is typically achieved with access tokens. Some tokens are JSON Web Tokens ([JWT](https://tools.ietf.org/html/rfc7519)), while others are opaque. Swagger has a two-part feature to describe protected access to routes: `securityDefinitions` (what sort of protection is applied …) and `security` (… to which routes), e.g.: @@ -64,9 +64,9 @@ paths: This defines a security scheme object (named `JWTCookie`), meaning that some sort of cookie is needed to access certain routes. This is applied to the [wildcard path](/reference/OpenAPI/differences.md#wildcard-paths) `/httpbin/**`. -This documentation feature, with some extensions, is used to make FLAT actually allow access to the route only if a certain JWT token is presented. +This documentation feature, with some extensions, is used to make FLAT actually permit access to the route only if a valid JWT token is presented. -First, we specify the name of the cookie expected to accompany the API request: +First, we define the name of the cookie expected to accompany the API request: ```yaml … @@ -79,7 +79,7 @@ securityDefinitions: … ``` -Then we give some [configuration](/reference/OpenAPI/security.md#the-x-flat-jwt-field) for decoding the JWT token: +Then we specify the [configuration](/reference/OpenAPI/security.md#the-x-flat-jwt-field) for decoding the JWT token: ```yaml … @@ -88,7 +88,7 @@ Then we give some [configuration](/reference/OpenAPI/security.md#the-x-flat-jwt- in: header name: Cookie x-flat-cookiename: authtoken - x-flat-jwt: # ⬅ specify the JWT configuration: + x-flat-jwt: # ⬅ our JWT configuration: key: # ⬅ the key to decode the JWT … file: secret.pem # ⬅ … is read from the file secret.pem alg: RS256 # ⬅ the signing algorithm is RS256 @@ -105,9 +105,9 @@ rdAOci3W9u3zOSGj4QIDAQAB -----END PUBLIC KEY----- ``` -That's all. Here, only those tokens will be accepted that are signed with the RS256 algorithm in a way that they can be decoded using the given public key. +That's all. Now FLAT will only permit requests if they supply a token that bear an RS256 signature that was created with the private key that matches the given public key. -(You may want to try this with the key and algorithm given to you by the party providing you with tokens, e.g. an OAuth2 authorization server.) +(You may want to try this with the key and algorithm provided to you by the party providing you with tokens, e.g. an OAuth2 authorization server.) Let's give it a try: @@ -193,7 +193,7 @@ But there are two additional features that can be quite handy: `out-var` and `ou ## Storing JWT claims: `out-var` -With `out-var` you can specify the name of the [variable](/reference/variables.md) where to store the JSON claims encoded in the JWT, in order to make them available to further processing. E.g. +With `out-var` you can specify the name of a [variable](/reference/variables.md) where FLAT will store the JSON claims encoded in the JWT, in order to make them available for further processing. E.g. ```yaml diff --git a/reference/OpenAPI/security.md b/reference/OpenAPI/security.md index 279084d..c3f8e1e 100644 --- a/reference/OpenAPI/security.md +++ b/reference/OpenAPI/security.md @@ -12,7 +12,7 @@ The `x-flat-jwt` field references an object with fields describing the expected * `key` - REQUIRED. The key to decode the JSON Web Signature (JWS). This can either be specified with a value, or by referencing a file (`file`) or an environment variable (`env`). * `alg` - The signing algorithm the JWS is expected to be created with. This can either be specified with a value, or by referencing a file (`file`) or an environment variable (`env`). See the [`algorithm` parameter for `jwt-decode()`](/reference/functions/jwt-decode.md) for more information. * `out-var` - The name of the variable in which the JWT is stored (must be a proper variable name, starting with `$`; default: `"$jwt"`). -* `out-header` - The name of the HTTP header that shall carry the JWT +* `out-header` - The name of an HTTP request header that shall carry the JWT in upstream requests. * `claims` - An object with claims the JWT payload is expected to contain. The field names are the claim names, the expected claim value is specified either with a value, or by referencing a file (`file`) or an environment variable (`env`). The token is considered valid if all of the following are true: @@ -22,6 +22,8 @@ The token is considered valid if all of the following are true: * the JWT contains the expected claims, if any are configured, * the JWT can be stored in a variable. +`$jwt` or the alternative variable specified in `out-var` and the header specified in `out-header` will be unset if the token is not valid. + ## JWT in `Authorization` Header Use a [Security Scheme Object](https://swagger.io/specification/v2/#securitySchemeObject) with `type: apiKey`, `in: header` and `name: Authorization` if the JWT is expected to be sent as a bearer token in an `Authorization` header; e.g.: @@ -158,7 +160,7 @@ paths: ## Forwarding JWT Upstream -The claims of an incoming JWT are stored in `$jwt` – or in any other global +The claims of an incoming JWT are stored in `$jwt` – or in the global [variable](/reference/variables.md) that you specify in the `out-var` property of `x-flat-jwt`. In a [`request`](/reference/actions/request.md) or From 517f3659bad4fd653c6b879e829e75cb58b59959 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Afflerbach?= Date: Wed, 27 May 2020 12:58:30 +0200 Subject: [PATCH 07/47] Fixes & cosmetics --- cookbook/custom-logging.md | 46 ++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/cookbook/custom-logging.md b/cookbook/custom-logging.md index bd762a7..a723d88 100644 --- a/cookbook/custom-logging.md +++ b/cookbook/custom-logging.md @@ -17,7 +17,7 @@ We want all of that! To play with logging, you need a FLAT project. If you don't have on, yet, have a look into the [tutorial](/tutorial/README.md) to [get started](/tutorial/README.md#getting-started). -## Reading logs +## Reading Logs Before start customizing our logs, we need to setup our workplace in order to see what we are doing. Where can we inspect FLAT logs? @@ -34,7 +34,7 @@ However, once you call your API with `curl` you will see JSON lines in your term $ curl localhost:8080/api/… ``` ```json -{"timestamp":"2019-10-15T15:49:13+00:00","type":"flat_access","requestID":"XaXqeccky00IowY@OUgG8AAAAAA","path":"/api/…","status":200,"method":"GET","agent":"curl/7.54.0","referrer":"","mime":"application/json"} +{"timestamp":"2019-10-15T15:49:13+00:00","type":"flat_access",…} ``` You can have the JSON colored and pretty printed by piping the output to the [`jq` command](https://stedolan.github.io/jq/): @@ -51,17 +51,23 @@ Now, the access log will be nicely readable on your terminal: "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", + "url": "http://localhost:8080/api/…", "path": "/api/…", "status": 200, "method": "GET", "agent": "curl/7.54.0", "referrer": "", - "mime": "application/json" + "mime": "application/json", + "realtime": 0.08935, + "bytes": 1387, + "requestbytes": 0, + "flow": "flow.xml", + "requestmime": "", + "location": "" } ``` - -## Adding a log field +## Adding a Log Field Now that we know where our logs go, we're finally ready to add fields! @@ -90,20 +96,14 @@ $ curl localhost:8080/api/… { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject" } ``` Our first custom log field has hit the terminal! -## Adding dynamic log fields +## Adding Dynamic Log Fields With [JSON templates](/reference/templating/README.md) you can use expressions to dynamically set field names and values. @@ -127,13 +127,7 @@ Notice, how we can provide default values for missing data. If _both_ of the var { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject", "stage": "unknown" } @@ -141,7 +135,7 @@ Notice, how we can provide default values for missing data. If _both_ of the var `stage` is always defined. But `location` is missing, because its expression has evaluated to `null`. Nulled fields don't show up in the log. -## Structured log fields +## Structured Log Fields Your custom log fields are not restricted to the top-level field-list. If you have more fields, that belong together, you can group them in an object. @@ -159,8 +153,6 @@ In our case, we might want to gather information about the environment: ``` -Notice, how we can provide default values for missing data. If _both_ of the variables are not set, a log may look like this: - ```json { "timestamp": "2019-10-15T15:49:13+00:00", @@ -180,9 +172,9 @@ Notice, how we can provide default values for missing data. If _both_ of the var } ``` -## Using request data +## Using Request Data -Client's provide a lot of useful information in the HTTP request headers. We can easily use them to augment our log: +Clients provide a lot of useful information in the HTTP request headers. We can easily use them to augment our log: ```xml @@ -233,7 +225,7 @@ For logging alone, the JSON armor prevents format breakouts and log attacks. HTT ## Testing -Log augmentation with the [`log` action](/reference/actions/log.md) belongs to our project code. Therefore, we would like to test that! This can be done by using the [`get-log()` function](/reference/functions/get-log.md) in a [FLAT test](/reference/testing/README.md). +Log augmentation with the [`log` action](/reference/actions/log.md) belongs to our project code. Therefore, we would like to test that! This can be done by using the [`get-log()` function](/reference/functions/get-log.md) in a [FLAT test](/reference/testing/README.md). Put this into `tests/loggging.xml`: ```xml @@ -260,7 +252,7 @@ $ flat test tests/logging.xml ok 1 tests/logging.xml: 1 assertions ``` -In a real project, you would rather send a request to your FLAT API and check if the log was written as expected +In a real project, you would rather send a request to your FLAT API and check if the log was written as expected: ```xml From e66951cb685f9f1c1f1c9fb752a7aec832668590 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Afflerbach?= Date: Wed, 27 May 2020 12:58:54 +0200 Subject: [PATCH 08/47] Changelog --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b281aae..25b8f72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ - [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#request-debugging) to enable debugging using the `Debug` request header, defaults to `false` +## Fixed + +- Empty objects are no longer [logged](/cookbook/custom-logging#adding-a-log-field) as empty arrays. + ## [20200519](https://hub.docker.com/r/sevenvaltechnologies/flatrunner/tags) ### Added From b0147cf939c04f845300b0ac9048b25f0d8bb9a3 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:16:42 +0200 Subject: [PATCH 09/47] changed port 1234 to 8080 --- cookbook/x-flat-jwt.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 895e7cd..c7b745d 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -13,10 +13,10 @@ paths: stripEndpoint: true ``` -Sending a request to FLAT running on localhost port 1234 results in: +Sending a request to FLAT running on localhost port 8080 results in: ``` -curl -i http://localhost:1234/httpbin/anything +curl -i http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json @@ -112,7 +112,7 @@ That's all. Now FLAT will only permit requests if they supply a token that bear Let's give it a try: ``` -curl -i http://localhost:1234/httpbin/anything +curl -i http://localhost:8080/httpbin/anything HTTP/1.1 403 Forbidden … Content-Type: application/json @@ -125,7 +125,7 @@ Ah, yes, we forgot to present a token in the `authtoken` cookie. But we see, tha Let's try again with a token: ``` -curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:1234/httpbin/anything +curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:8080/httpbin/anything HTTP/1.1 403 Forbidden … Content-Type: application/json @@ -138,7 +138,7 @@ Hmm, expired. So this one is too old. (Access tokens typically have a restricted OK, let's use another token: ``` -curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:1234/httpbin/anything +curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json @@ -186,7 +186,7 @@ paths: You can try that with ``` -curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:1234/httpbin/anything +curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything ``` But there are two additional features that can be quite handy: `out-var` and `out-header`. @@ -252,7 +252,7 @@ With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you ``` ``` -curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:1234/httpbin/anything +curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json From a14e55519daf66148f68ebdc459573aa7b020c99 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:19:09 +0200 Subject: [PATCH 10/47] changed "provided to you by" to "acquired from" --- cookbook/x-flat-jwt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index c7b745d..dc3fbba 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -107,7 +107,7 @@ rdAOci3W9u3zOSGj4QIDAQAB That's all. Now FLAT will only permit requests if they supply a token that bear an RS256 signature that was created with the private key that matches the given public key. -(You may want to try this with the key and algorithm provided to you by the party providing you with tokens, e.g. an OAuth2 authorization server.) +(You may want to try this with the key and algorithm acquired from the party providing you with tokens, e.g. an OAuth2 authorization server.) Let's give it a try: From 2fda4c224498ee4de35323aa39f51eca78189d47 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:21:02 +0200 Subject: [PATCH 11/47] Added prompt --- cookbook/x-flat-jwt.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index dc3fbba..f1fc2f7 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -16,7 +16,7 @@ paths: Sending a request to FLAT running on localhost port 8080 results in: ``` -curl -i http://localhost:8080/httpbin/anything +$ curl -i http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json @@ -112,7 +112,7 @@ That's all. Now FLAT will only permit requests if they supply a token that bear Let's give it a try: ``` -curl -i http://localhost:8080/httpbin/anything +$ curl -i http://localhost:8080/httpbin/anything HTTP/1.1 403 Forbidden … Content-Type: application/json @@ -125,7 +125,7 @@ Ah, yes, we forgot to present a token in the `authtoken` cookie. But we see, tha Let's try again with a token: ``` -curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:8080/httpbin/anything +$ curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:8080/httpbin/anything HTTP/1.1 403 Forbidden … Content-Type: application/json @@ -138,7 +138,7 @@ Hmm, expired. So this one is too old. (Access tokens typically have a restricted OK, let's use another token: ``` -curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +$ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json @@ -186,7 +186,7 @@ paths: You can try that with ``` -curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +$ curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything ``` But there are two additional features that can be quite handy: `out-var` and `out-header`. @@ -252,7 +252,7 @@ With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you ``` ``` -curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +$ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json From 8baf7ddae2a02d73cfbaba228475fc12d95d11d5 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:22:21 +0200 Subject: [PATCH 12/47] link to log action docu --- cookbook/x-flat-jwt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index f1fc2f7..5f13a31 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -206,7 +206,7 @@ With `out-var` you can specify the name of a [variable](/reference/variables.md) … ``` -We can log the claims by adding a `log` action to an [init flow](/reference/OpenAPI/routing.md#init-flow): +We can log the claims by adding a [`log` action](/reference/actions/log.md) to an [init flow](/reference/OpenAPI/routing.md#init-flow): ```yaml swagger: "2.0" From ce0f5b64e5c627a7eee45e3a193f0abb97c43669 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:24:56 +0200 Subject: [PATCH 13/47] rename public key file --- cookbook/x-flat-jwt.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 5f13a31..e5426ee 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -90,12 +90,12 @@ Then we specify the [configuration](/reference/OpenAPI/security.md#the-x-flat-jw x-flat-cookiename: authtoken x-flat-jwt: # ⬅ our JWT configuration: key: # ⬅ the key to decode the JWT … - file: secret.pem # ⬅ … is read from the file secret.pem + file: pubkey.pem # ⬅ … is read from the file pubkey.pem alg: RS256 # ⬅ the signing algorithm is RS256 … ``` -The specified key is a public key, read from the file secret.pem, e.g.: +The specified key is a public key, read from the file pubkey.pem, e.g.: ``` -----BEGIN PUBLIC KEY----- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGSd+sSTss2uOuVJKpumpFAaml @@ -175,7 +175,7 @@ securityDefinitions: name: Authorization # ⬅ x-flat-jwt: key: - file: secret.pem + file: pubkey.pem alg: RS256 paths: /httpbin/**: @@ -200,7 +200,7 @@ With `out-var` you can specify the name of a [variable](/reference/variables.md) … x-flat-jwt: key: - file: secret.pem + file: pubkey.pem alg: RS256 out-var: $the_claims # ⬅ … @@ -244,7 +244,7 @@ With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you … x-flat-jwt: key: - file: secret.pem + file: pubkey.pem alg: RS256 out-var: $the_claims out-header: JWT # ⬅ the name of the request header with the JWT claims @@ -282,7 +282,7 @@ securityDefinitions: x-flat-cookiename: authtoken x-flat-jwt: key: - file: secret.pem + file: pubkey.pem alg: RS256 out-var: $the_claims out-header: JWT @@ -296,7 +296,7 @@ paths: stripEndpoint: true ``` -secret.pem: +pubkey.pem: ``` -----BEGIN PUBLIC KEY----- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGSd+sSTss2uOuVJKpumpFAaml From 87d1140dfa7e1e243401bfc679e3f23796139399 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:38:13 +0200 Subject: [PATCH 14/47] list and explain sample tokens --- cookbook/x-flat-jwt.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index e5426ee..005cab6 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -107,7 +107,17 @@ rdAOci3W9u3zOSGj4QIDAQAB That's all. Now FLAT will only permit requests if they supply a token that bear an RS256 signature that was created with the private key that matches the given public key. -(You may want to try this with the key and algorithm acquired from the party providing you with tokens, e.g. an OAuth2 authorization server.) +Usually, you would get the key and algorithm from your identity provider (e.g. an OAuth2 authorization server). That service would be responsible for issuing JWT tokens for your users. + +For this tutorial we have prepared a couple of JWT tokens for you to try out different situations: + +``` +eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4 +``` + +``` +eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU +``` Let's give it a try: @@ -122,7 +132,7 @@ Content-Type: application/json Ah, yes, we forgot to present a token in the `authtoken` cookie. But we see, that the protection works. -Let's try again with a token: +Let's try again with the first token: ``` $ curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:8080/httpbin/anything @@ -135,7 +145,7 @@ Content-Type: application/json Hmm, expired. So this one is too old. (Access tokens typically have a restricted period of use.) -OK, let's use another token: +OK, let's use the other token: ``` $ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything From 4e2ededf1b10bdbeda1aad255ad25653782c4d5f Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:39:09 +0200 Subject: [PATCH 15/47] Move out-header section up --- cookbook/x-flat-jwt.md | 64 +++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 005cab6..2e3ea75 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -201,6 +201,38 @@ $ curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiL But there are two additional features that can be quite handy: `out-var` and `out-header`. +## Sending JWT claims upstream: `out-header` + +With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you can send the whole set of claims from the JWT upstream: + +```yaml: +… + x-flat-jwt: + key: + file: pubkey.pem + alg: RS256 + out-var: $the_claims + out-header: JWT # ⬅ the name of the request header with the JWT claims +… +``` + +``` +$ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +HTTP/1.1 200 OK +… +Content-Type: application/json + +{ +… + "headers": { +… + "Jwt": "{\"sub\":\"some_user\",\"iss\":\"some_provider\"}", +… + }, +… +} +``` + ## Storing JWT claims: `out-var` With `out-var` you can specify the name of a [variable](/reference/variables.md) where FLAT will store the JSON claims encoded in the JWT, in order to make them available for further processing. E.g. @@ -246,38 +278,6 @@ If you look at the [FLAT logs](/administration/logging.md) and try again with a Here you see the two claims from the JWT token. -## Sending JWT claims upstream: `out-header` - -With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you can send the whole set of claims from the JWT upstream: - -```yaml: -… - x-flat-jwt: - key: - file: pubkey.pem - alg: RS256 - out-var: $the_claims - out-header: JWT # ⬅ the name of the request header with the JWT claims -… -``` - -``` -$ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything -HTTP/1.1 200 OK -… -Content-Type: application/json - -{ -… - "headers": { -… - "Jwt": "{\"sub\":\"some_user\",\"iss\":\"some_provider\"}", -… - }, -… -} -``` - ## All files together swagger.yaml: From b3e9253c2c3bc1b232304f4ec16b622177d8fc60 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:40:25 +0200 Subject: [PATCH 16/47] "Storing" -> "Accessing" --- cookbook/x-flat-jwt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 2e3ea75..73af307 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -233,7 +233,7 @@ Content-Type: application/json } ``` -## Storing JWT claims: `out-var` +## Accessing JWT claims: `out-var` With `out-var` you can specify the name of a [variable](/reference/variables.md) where FLAT will store the JSON claims encoded in the JWT, in order to make them available for further processing. E.g. From 0a0399dc26776dd2a4b059f6f6930e04cf1fac25 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Wed, 27 May 2020 17:44:10 +0200 Subject: [PATCH 17/47] more links --- reference/functions/jwt-decode.md | 5 +++-- reference/functions/jwt-encode.md | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/reference/functions/jwt-decode.md b/reference/functions/jwt-decode.md index c0ce4fc..b70c9f4 100644 --- a/reference/functions/jwt-decode.md +++ b/reference/functions/jwt-decode.md @@ -48,5 +48,6 @@ The unpacked web token is stored in `$jwt`, which provides easy access to its co ## See also -* [`jwt-encode()`](jwt-encode.md) -* [Encoding and Decoding JWT](/cookbook/jwt.md) +* [`jwt-encode()`](jwt-encode.md) (reference) +* [Encoding and Decoding JWT](/cookbook/jwt.md) (cookbook) +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) diff --git a/reference/functions/jwt-encode.md b/reference/functions/jwt-encode.md index 9218616..e236a2c 100644 --- a/reference/functions/jwt-encode.md +++ b/reference/functions/jwt-encode.md @@ -46,5 +46,6 @@ After 600 seconds the token becomes invalid: ## See also -* [`jwt-decode()`](jwt-decode.md) -* [Encoding and Decoding JWT](/cookbook/jwt.md) +* [`jwt-decode()`](jwt-decode.md) (reference) +* [Encoding and Decoding JWT](/cookbook/jwt.md) (cookbook) +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) From 8b56c38f2a8334af86265e3bb30bd14736d54361 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 08:44:53 +0200 Subject: [PATCH 18/47] correct valid token --- cookbook/x-flat-jwt.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 73af307..df1678e 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -116,7 +116,7 @@ eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3B ``` ``` -eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU +eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU ``` Let's give it a try: @@ -148,7 +148,7 @@ Hmm, expired. So this one is too old. (Access tokens typically have a restricted OK, let's use the other token: ``` -$ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +$ curl -i -H "Cookie: authtoken=eeyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json From 63995320f8c3d4ece8beacd7395bec6a1a1aeef1 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 10:17:21 +0200 Subject: [PATCH 19/47] pretty-print error JSON --- cookbook/x-flat-jwt.md | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index df1678e..eeddc22 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -127,7 +127,17 @@ HTTP/1.1 403 Forbidden … Content-Type: application/json -{"error":{"message":"Security violation","status":403,"requestID":"Xsz@QAv8CEBZfWbmQhKLIQAAAII","info":["JWT Security (JWTCookie): No Cookie header sent"],"code":3206}} +{ + "error": { + "message": "Security violation", + "status": 403, + "requestID": "Xsz@QAv8CEBZfWbmQhKLIQAAAII", + "info": [ + "JWT Security (JWTCookie): No Cookie header sent" + ], + "code": 3206 + } +} ``` Ah, yes, we forgot to present a token in the `authtoken` cookie. But we see, that the protection works. @@ -140,7 +150,17 @@ HTTP/1.1 403 Forbidden … Content-Type: application/json -{"error":{"message":"Security violation","status":403,"requestID":"Xsz80Av8CEBZfWbmQhKLIAAAAIE","info":["JWT Security (JWTCookie): Invalid JWT: Token has expired."],"code":3206}} +{ + "error": { + "message": "Security violation", + "status": 403, + "requestID": "Xsz80Av8CEBZfWbmQhKLIAAAAIE", + "info": [ + "JWT Security (JWTCookie): Invalid JWT: Token has expired." + ], + "code": 3206 + } +} ``` Hmm, expired. So this one is too old. (Access tokens typically have a restricted period of use.) From adbe5a969d118ede71edfc88c521c7d50e1c0733 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 12:12:49 +0200 Subject: [PATCH 20/47] shortened JSON log --- cookbook/custom-logging.md | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/cookbook/custom-logging.md b/cookbook/custom-logging.md index a723d88..cabbdc7 100644 --- a/cookbook/custom-logging.md +++ b/cookbook/custom-logging.md @@ -157,13 +157,7 @@ In our case, we might want to gather information about the environment: { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject", "env": { "stage": "production", @@ -201,13 +195,7 @@ $ curl -H "X-Forwarded-Proto: https" localhost:8080/api/… { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject", "env": { "stage": "production", From e0001a467e9c112765a0c01a07f69138a51d26da Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Afflerbach?= Date: Thu, 28 May 2020 12:55:57 +0200 Subject: [PATCH 21/47] Fix link --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 25b8f72..bde4b51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ ## Fixed -- Empty objects are no longer [logged](/cookbook/custom-logging#adding-a-log-field) as empty arrays. +- Empty objects are no longer [logged](/cookbook/custom-logging.md#adding-a-log-field) as empty arrays. ## [20200519](https://hub.docker.com/r/sevenvaltechnologies/flatrunner/tags) From 0db2b7054f27dbd0ae56b10e5aadd07363c60088 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Afflerbach?= Date: Thu, 28 May 2020 12:56:56 +0200 Subject: [PATCH 22/47] System log fields can no longer be overridden. --- CHANGELOG.md | 5 +++++ reference/actions/log.md | 2 ++ 2 files changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bde4b51..bc5c165 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,11 @@ - Empty objects are no longer [logged](/cookbook/custom-logging.md#adding-a-log-field) as empty arrays. +## Changed + +The [`log` action](/reference/actions/log.md) can no longer override [system log fields](/administration/logging.md#fields). + + ## [20200519](https://hub.docker.com/r/sevenvaltechnologies/flatrunner/tags) ### Added diff --git a/reference/actions/log.md b/reference/actions/log.md index 5ade2b8..8d558dd 100644 --- a/reference/actions/log.md +++ b/reference/actions/log.md @@ -19,6 +19,8 @@ The action takes a JSON object as its argument. The object may contain nested fi All name/value pairs of the object are registered for logging. When the system writes the `flat_access` event the registered fields are included in that log line. +[System log fields](/administration/logging.md#fields) like `timestamp` cannot be overriden. + You can call the action multiple times. Fields of the same name are overwritten. However, nested fields are merged into the previously registered log fields. ```xml From 3b4e30c6ddd1a9511e92c19dc439d3386db1b500 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 14:50:02 +0200 Subject: [PATCH 23/47] json-to-csv() allows null value as array entry value --- reference/functions/json-to-csv.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/reference/functions/json-to-csv.md b/reference/functions/json-to-csv.md index 279d2f7..78570a4 100644 --- a/reference/functions/json-to-csv.md +++ b/reference/functions/json-to-csv.md @@ -6,7 +6,7 @@ string json-to-csv(OXN array) The `json-to-csv` function translates the given [OXN `array`](/reference/templating/oxn.md) into CSV as described in [RFC 4180](https://tools.ietf.org/html/rfc4180). -The array entries must either be arrays or "flat" objects with `number`, `boolean` or `string` values. +The array entries must either be arrays or "flat" objects with `number`, `boolean`, `string`, or `null` values. If any errors occur, an empty `string` is returned. @@ -19,7 +19,8 @@ Example: array of arrays [ [ 1, " foo ", true ], [ 2, "ba, r", false ], - [ 3.21, "q\"u\"x", true ] + [ 3.21, "q\"u\"x", true ], + [ '', null, '' ] ] json-to-csv($arr) @@ -33,6 +34,7 @@ creates the following output: 1, foo ,true 2,"ba, r",false 3.21,"q""u""x",true +,, ``` @@ -45,7 +47,8 @@ Example: array of "flat" objects [ { "A": 1, "B": " foo ", "C": true }, { "A": 2, "B": "ba, r", "C": false }, - { "A": 3.21, "B": "q\"u\"x", "C": true } + { "A": 3.21, "B": "q\"u\"x", "C": true }, + { "A": "", "B": null, "C": "" } ] json-to-csv($arr) From 04fd4ff1eabbf764e02d2d1364d63c8d1a26fd57 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 14:52:49 +0200 Subject: [PATCH 24/47] json-to-csv(): Changelog for null values --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bc5c165..8a2b7be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ ## Fixed - Empty objects are no longer [logged](/cookbook/custom-logging.md#adding-a-log-field) as empty arrays. +- The [`json-to-csv()` function](/reference/functions/json-to-csv.md) allows `null` values in array entry objects. ## Changed From ff8967b42511a044a77682ac807568bd644e0067 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 17:20:22 +0200 Subject: [PATCH 25/47] Add new recipe to list --- cookbook/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cookbook/README.md b/cookbook/README.md index a3da9ac..7ceeba1 100644 --- a/cookbook/README.md +++ b/cookbook/README.md @@ -1,5 +1,7 @@ # Cookbook +## [Protecting Access using JWT Tokens](x-flat-jwt.md) + ## [How can I inspect the client request?](see-client-request.md) ## [How can I pass an arbitrary header field to an upstream system?](pass-header-field-upstream.md) From 8fad8c4e00dc447f2f617728fc9974d4d440a1b4 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 17:21:00 +0200 Subject: [PATCH 26/47] New label for JWT signing recipe --- cookbook/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cookbook/README.md b/cookbook/README.md index 7ceeba1..24b3302 100644 --- a/cookbook/README.md +++ b/cookbook/README.md @@ -16,7 +16,7 @@ ## [Sending POST requests](upstream-post-requests.md) -## [Working with JWT](jwt.md) +## [Signing JWT](jwt.md) ## [Auto Docs with Swagger UI](swagger-docs.md) From b5c31de1a554e9b006fe5d713a32e458dbece49d Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 17:23:26 +0200 Subject: [PATCH 27/47] Changed order in recipe list --- cookbook/README.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/cookbook/README.md b/cookbook/README.md index 24b3302..86d55de 100644 --- a/cookbook/README.md +++ b/cookbook/README.md @@ -2,38 +2,38 @@ ## [Protecting Access using JWT Tokens](x-flat-jwt.md) -## [How can I inspect the client request?](see-client-request.md) +## [Proxying Requests to Upstream APIs (Swagger)](proxy-requests.md) -## [How can I pass an arbitrary header field to an upstream system?](pass-header-field-upstream.md) +## [Testing Templates](test-templates.md) -## [Proxying Requests to Upstream APIs (Swagger)](proxy-requests.md) +## [Testing API Requests](test-api-request.md) -## [Forwarding a Request to an Upstream API (Flow)](forward-request-upstream.md) +## [Testing with Backend Requests](test-backend.md) -## [How can I pass response headers to the client?](pass-header-field-downstream.md) +## [File Serving](file-serving.md) -## [How can I increase the request timeout to deal with a slow upstream system?](request-timeout.md) +## [Error Handling](error-flow.md) -## [Sending POST requests](upstream-post-requests.md) +## [Extracting Common Initialization Flow Tasks](init-flow.md) -## [Signing JWT](jwt.md) +## [Using Environment Variables](envvars.md) -## [Auto Docs with Swagger UI](swagger-docs.md) +## [Logging Custom Fields](custom-logging.md) -## [Using the Built-in Mocking](builtin-mocking.md) +## [Forwarding a Request to an Upstream API (Flow)](forward-request-upstream.md) -## [Extracting Common Initialization Flow Tasks](init-flow.md) +## [How can I pass an arbitrary header field to an upstream system?](pass-header-field-upstream.md) -## [Testing Templates](test-templates.md) +## [How can I pass response headers to the client?](pass-header-field-downstream.md) -## [Testing API Requests](test-api-request.md) +## [How can I increase the request timeout to deal with a slow upstream system?](request-timeout.md) -## [Testing with Backend Requests](test-backend.md) +## [Sending POST requests](upstream-post-requests.md) -## [Using Environment Variables](envvars.md) +## [Signing JWT](jwt.md) -## [File Serving](file-serving.md) +## [How can I inspect the client request?](see-client-request.md) -## [Error Handling](error-flow.md) +## [Auto Docs with Swagger UI](swagger-docs.md) -## [Logging Custom Fields](custom-logging.md) +## [Using the Built-in Mocking](builtin-mocking.md) From 29f8e4498ad20bfba1f29ab55165616322df0677 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 28 May 2020 17:34:29 +0200 Subject: [PATCH 28/47] Changed order of out-header and out-var also reflected in text and examples --- cookbook/x-flat-jwt.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index eeddc22..2bf705e 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -194,7 +194,7 @@ Content-Type: application/json Tada! -By the way, apart from Cookies, this also works similarly with the `Authorization: Bearer …` header: +By the way, apart from cookies, this also works similarly with the `Authorization: Bearer …` header: ``` … @@ -219,7 +219,7 @@ You can try that with $ curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything ``` -But there are two additional features that can be quite handy: `out-var` and `out-header`. +But there are two additional features that can be quite handy: `out-header` and `out-var`. ## Sending JWT claims upstream: `out-header` @@ -231,7 +231,6 @@ With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you key: file: pubkey.pem alg: RS256 - out-var: $the_claims out-header: JWT # ⬅ the name of the request header with the JWT claims … ``` @@ -264,6 +263,7 @@ With `out-var` you can specify the name of a [variable](/reference/variables.md) key: file: pubkey.pem alg: RS256 + out-header: JWT out-var: $the_claims # ⬅ … ``` From 35317ac7561ec7fae64dbbab75681822286fd496 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Afflerbach?= Date: Tue, 2 Jun 2020 09:32:49 +0200 Subject: [PATCH 29/47] Shell highlighting --- cookbook/x-flat-jwt.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 2bf705e..0c8e642 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -15,7 +15,7 @@ paths: Sending a request to FLAT running on localhost port 8080 results in: -``` +```sh $ curl -i http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … @@ -121,7 +121,7 @@ eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3B Let's give it a try: -``` +```sh $ curl -i http://localhost:8080/httpbin/anything HTTP/1.1 403 Forbidden … @@ -144,7 +144,7 @@ Ah, yes, we forgot to present a token in the `authtoken` cookie. But we see, tha Let's try again with the first token: -``` +```sh $ curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:8080/httpbin/anything HTTP/1.1 403 Forbidden … @@ -167,7 +167,7 @@ Hmm, expired. So this one is too old. (Access tokens typically have a restricted OK, let's use the other token: -``` +```sh $ curl -i -H "Cookie: authtoken=eeyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … @@ -215,7 +215,7 @@ paths: You can try that with -``` +```sh $ curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything ``` @@ -235,7 +235,7 @@ With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you … ``` -``` +```sh $ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … From fbf4f8e1f55d07dabd7640ea7b44eb3807d219ec Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 9 Jul 2020 16:02:42 +0200 Subject: [PATCH 30/47] documentation for request option force-cache-refresh --- reference/actions/request.md | 1 + 1 file changed, 1 insertion(+) diff --git a/reference/actions/request.md b/reference/actions/request.md index 6d312f2..31bf9e2 100644 --- a/reference/actions/request.md +++ b/reference/actions/request.md @@ -306,6 +306,7 @@ The `options` property sets the request options. Its value must be a JSON object * `proxy-credentials` - Authentication data for the proxy server (type: `string`) * `use-http-cache` - Whether to enable the HTTP cache (type: `boolean`, default: `false`). This option is mutually exclusive with `force-cache-ttl`. * `force-cache-ttl` - All resources are cached with a fixed time-to-live, ignoring all cache response headers and other obstacles for caching (type: `integer`). A number > 0 enables this kind of caching, specifying a lifetime in seconds. This option is mutually exclusive with `use-http-cache`. +* `force-cache-refresh` - Whether to send the request upstream, even if a cached response is available (type: `boolean`, default: `false`). This works with either `use-http-cache` or `force-cache-ttl`. * `respect-client-cache-headers` - Whether to respect incoming cache headers (type: `boolean`, default: `false`) * `follow-redirects` - Whether to automatically follow redirects (type: `boolean`, default: `false`) * `max-redirects` - Maximum number of consecutive redirects to follow (type: `integer`) From e1de401db3964ac242bd12bb98a1ffeff737143e Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 9 Jul 2020 16:05:13 +0200 Subject: [PATCH 31/47] changelog entry for request option force-cache-refresh --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a2b7be..be2892c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ ### Added - [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#request-debugging) to enable debugging using the `Debug` request header, defaults to `false` +- The [request option](/reference/actions/request.md#options) `force-cache-refresh` ## Fixed From 0a3e9124a8158dcaa023a7217efeb661e1231aec Mon Sep 17 00:00:00 2001 From: Felix Hassert Date: Wed, 12 Aug 2020 14:21:11 +0200 Subject: [PATCH 32/47] fix typo in jwt token eey -> ey --- cookbook/x-flat-jwt.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md index 0c8e642..7bd4d5a 100644 --- a/cookbook/x-flat-jwt.md +++ b/cookbook/x-flat-jwt.md @@ -4,7 +4,7 @@ Imagine you had a [proxy](/reference/OpenAPI/routing.md#assigning-flat-proxies) swagger.yml: ```yaml -swagger: "2.0" +swagger: "2.0" basePath: / paths: /httpbin/**: @@ -31,12 +31,12 @@ Content-Type: application/json "Accept-Encoding": "deflate, gzip", "Host": "httpbin.org", "User-Agent": "curl/7.29.0", - "X-Amzn-Trace-Id": "…" + "X-Amzn-Trace-Id": "…" }, "json": null, "method": "GET", "origin": "…", - "url": "https://httpbin.org/anything" + "url": "https://httpbin.org/anything" } ``` @@ -49,7 +49,7 @@ Now, you don't want anyone except authorized users to use this proxy. This is ty Swagger has a two-part feature to describe protected access to routes: `securityDefinitions` (what sort of protection is applied …) and `security` (… to which routes), e.g.: ```yaml -swagger: "2.0" +swagger: "2.0" basePath: / securityDefinitions: JWTCookie: @@ -168,7 +168,7 @@ Hmm, expired. So this one is too old. (Access tokens typically have a restricted OK, let's use the other token: ```sh -$ curl -i -H "Cookie: authtoken=eeyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +$ curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything HTTP/1.1 200 OK … Content-Type: application/json @@ -183,12 +183,12 @@ Content-Type: application/json "Accept-Encoding": "deflate, gzip", "Host": "httpbin.org", "User-Agent": "curl/7.29.0", - "X-Amzn-Trace-Id": "…" + "X-Amzn-Trace-Id": "…" }, "json": null, "method": "GET", "origin": "…", - "url": "https://httpbin.org/anything" + "url": "https://httpbin.org/anything" } ``` @@ -271,7 +271,7 @@ With `out-var` you can specify the name of a [variable](/reference/variables.md) We can log the claims by adding a [`log` action](/reference/actions/log.md) to an [init flow](/reference/OpenAPI/routing.md#init-flow): ```yaml -swagger: "2.0" +swagger: "2.0" … x-flat-init: init.xml paths: @@ -302,7 +302,7 @@ Here you see the two claims from the JWT token. swagger.yaml: ```yaml -swagger: "2.0" +swagger: "2.0" basePath: / securityDefinitions: JWTCookie: From 2b3600f5b76b4795f0d451d2cfb27b050bc5839a Mon Sep 17 00:00:00 2001 From: "Rainer M. Canavan" Date: Wed, 19 Aug 2020 17:48:37 +0200 Subject: [PATCH 33/47] force-cache-refresh doku beautification #47442_force-cache-refresh --- reference/actions/request.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/actions/request.md b/reference/actions/request.md index 31bf9e2..8e56489 100644 --- a/reference/actions/request.md +++ b/reference/actions/request.md @@ -306,7 +306,7 @@ The `options` property sets the request options. Its value must be a JSON object * `proxy-credentials` - Authentication data for the proxy server (type: `string`) * `use-http-cache` - Whether to enable the HTTP cache (type: `boolean`, default: `false`). This option is mutually exclusive with `force-cache-ttl`. * `force-cache-ttl` - All resources are cached with a fixed time-to-live, ignoring all cache response headers and other obstacles for caching (type: `integer`). A number > 0 enables this kind of caching, specifying a lifetime in seconds. This option is mutually exclusive with `use-http-cache`. -* `force-cache-refresh` - Whether to send the request upstream, even if a cached response is available (type: `boolean`, default: `false`). This works with either `use-http-cache` or `force-cache-ttl`. +* `force-cache-refresh` - if `true`, fetch a fresh response from upstream, even if a valid cached response is available (type: `boolean`, default: `false`). This works with both `use-http-cache` or `force-cache-ttl`. * `respect-client-cache-headers` - Whether to respect incoming cache headers (type: `boolean`, default: `false`) * `follow-redirects` - Whether to automatically follow redirects (type: `boolean`, default: `false`) * `max-redirects` - Maximum number of consecutive redirects to follow (type: `integer`) From 2beffa0de29eef80f0577b02809e021ef8cac224 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Fri, 21 Aug 2020 17:08:24 +0200 Subject: [PATCH 34/47] documentation for the ldap-lookup() function --- reference/functions/README.md | 1 + reference/functions/ldap-lookup.md | 43 ++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) create mode 100644 reference/functions/ldap-lookup.md diff --git a/reference/functions/README.md b/reference/functions/README.md index 94a32d5..3d4c572 100644 --- a/reference/functions/README.md +++ b/reference/functions/README.md @@ -85,6 +85,7 @@ * [`has-class()`](has-class.md) * [`html-parse()`](html-parse.md) * [`id()` ↗](https://developer.mozilla.org/en/XPath/Functions/id) +* [`ldap-lookup()`](ldap-lookup.md) * [`lang()` ↗](https://developer.mozilla.org/en/XPath/Functions/lang) * [`last()` ↗](https://developer.mozilla.org/en/XPath/Functions/last) * [`local-name()` ↗](https://developer.mozilla.org/en/XPath/Functions/local-name) diff --git a/reference/functions/ldap-lookup.md b/reference/functions/ldap-lookup.md new file mode 100644 index 0000000..cd97d04 --- /dev/null +++ b/reference/functions/ldap-lookup.md @@ -0,0 +1,43 @@ +# `ldap-lookup()` + +``` +OXN-node-set ldap-lookup(string url, string rdn, string rdnPassword, string base_dn, string userSearch, string userPassword, string attributes) +``` + +The `ldap-lookup()` function connects to an LDAP server with the given `url`, `rdn` and `rdnPassword`. +It then searches for a user by the given `userSearch`. +If a user is found, it connects with the user's DN and the given `userPassword`. +If the password is correct, an [OXN](/reference/templating/oxn.md) JSON document is returned with at least the user's `dn` and additional attributes given by `attributes`. +Otherwise an empty node-set is returned. + +## Parameters + +* `url` The ldap URL (string) +* `rdn` The (relative) distinguished name of the (system) user (string) +* `rdnPassword` The password of the (system) user (string) +* `base_dn` The base distinguished name for the search (string) +* `userSearch` The filter for searching a user (string) +* `userPassword` The user's password (string) +* `attributes` A comma-separated list of attributes to return (string) + + +## Example + +In the following example, the LDAP server is connected with the DN given in `$ldap_settings/bind_dn` and the password from $env/FLAT_SYSTEM_PASSWORD. +Zhe given filter is used to search for an entry of a person which is a member of a group "Users" and has the email address "john.doe@example.com". +In addition to the (default) `dn`, the `sAMAccountName` and `mail` from the entry are added to the result. + +```xml + + concat("(&(objectClass=person)(memberOf=CN=Users,ou=People,dc=example,dc=com)(mail=john.doe@example.com))") + "sAMAccountName,mail" + + ldap-lookup($ldap_settings/url, $ldap_settings/bind_dn, $env/FLAT_SYSTEM_PASSWORD, $ldap_settings/base_dn, $userSearch, "myP4s5w0rD, $attributes) + + { + "status": 403, + "message": "ldap-lookup() failed" + } + + +``` From a349bb17415dd2499b1b1d9db10362dc84a4d9a3 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Fri, 21 Aug 2020 17:09:58 +0200 Subject: [PATCH 35/47] changelog for the ldap-lookup() function --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index be2892c..71270b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ - [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#request-debugging) to enable debugging using the `Debug` request header, defaults to `false` - The [request option](/reference/actions/request.md#options) `force-cache-refresh` +- The [`ldap-lookup()` function](/reference/functions/ldap-lookup.md) ## Fixed From df0fe401dd43bbb860180adb6ca2a93a72901d7c Mon Sep 17 00:00:00 2001 From: Felix Hassert Date: Wed, 26 Aug 2020 18:53:45 +0200 Subject: [PATCH 36/47] typo --- reference/functions/ldap-lookup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/functions/ldap-lookup.md b/reference/functions/ldap-lookup.md index cd97d04..a970663 100644 --- a/reference/functions/ldap-lookup.md +++ b/reference/functions/ldap-lookup.md @@ -24,7 +24,7 @@ Otherwise an empty node-set is returned. ## Example In the following example, the LDAP server is connected with the DN given in `$ldap_settings/bind_dn` and the password from $env/FLAT_SYSTEM_PASSWORD. -Zhe given filter is used to search for an entry of a person which is a member of a group "Users" and has the email address "john.doe@example.com". +The given filter is used to search for an entry of a person which is a member of a group "Users" and has the email address "john.doe@example.com". In addition to the (default) `dn`, the `sAMAccountName` and `mail` from the entry are added to the result. ```xml From 8de9ad56854518bb2d3a3a63ab2c7b02e574b8fa Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 27 Aug 2020 08:44:58 +0200 Subject: [PATCH 37/47] JSON response in case of success --- reference/functions/ldap-lookup.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/reference/functions/ldap-lookup.md b/reference/functions/ldap-lookup.md index a970663..8f31cee 100644 --- a/reference/functions/ldap-lookup.md +++ b/reference/functions/ldap-lookup.md @@ -41,3 +41,12 @@ In addition to the (default) `dn`, the `sAMAccountName` and `mail` from the entr ``` + +The result in the case of success, is +```json +{ + "dn": "cn=John Doe,ou=People,dc=example,dc=com", + "sAMAccountName": "john.doe", + "mail": "john.doe@example.com" +} +``` From c53a3a1d68a395a6e38072ba9cef653b19954b18 Mon Sep 17 00:00:00 2001 From: Felix Hassert Date: Thu, 27 Aug 2020 09:04:58 +0200 Subject: [PATCH 38/47] remove traces of sources.xml --- reference/actions/request.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/reference/actions/request.md b/reference/actions/request.md index 8e56489..d710ce3 100644 --- a/reference/actions/request.md +++ b/reference/actions/request.md @@ -335,11 +335,6 @@ Example: } ``` -> 📎 -> Any request options set in a `conf/sources.xml` file that match the requested domain and path -> will also be applied to your request. Options directly set in the action have precedence, though. -> We do **not** recommend using `conf/sources.xml`. - ## See also From 6c824b445d9f95b3d8383c451480bc3209c87353 Mon Sep 17 00:00:00 2001 From: Felix Hassert Date: Thu, 27 Aug 2020 09:58:10 +0200 Subject: [PATCH 39/47] ldap-docs: add jwt hints + markup fixes --- reference/functions/ldap-lookup.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/reference/functions/ldap-lookup.md b/reference/functions/ldap-lookup.md index 8f31cee..35a80c2 100644 --- a/reference/functions/ldap-lookup.md +++ b/reference/functions/ldap-lookup.md @@ -23,8 +23,8 @@ Otherwise an empty node-set is returned. ## Example -In the following example, the LDAP server is connected with the DN given in `$ldap_settings/bind_dn` and the password from $env/FLAT_SYSTEM_PASSWORD. -The given filter is used to search for an entry of a person which is a member of a group "Users" and has the email address "john.doe@example.com". +In the following example, the LDAP server is connected with the DN given in `$ldap_settings/bind_dn` and the password from `$env/FLAT_SYSTEM_PASSWORD`. +The given filter is used to search for an entry of a person which is a member of a group `Users` and has the email address `john.doe@example.com`. In addition to the (default) `dn`, the `sAMAccountName` and `mail` from the entry are added to the result. ```xml @@ -50,3 +50,12 @@ The result in the case of success, is "mail": "john.doe@example.com" } ``` + +In a real setup you would read the user (here `john.doe@example.com`) and password parameters from user input, such as the JSON request body (e.g. `$body/json/username` and `$body/json/password`). + +The attributes returned from the function can then be used to set claims in a JWT token with [`jwt-encode()`](/reference/functions/jwt-encode.md). + +## See also + +* [`jwt-encode()`](jwt-encode.md) +* [Encoding and Decoding JWT](/cookbook/jwt.md) From 8fdf980c6b04c2ff4d4b54bd74225c8df36ba381 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 27 Aug 2020 11:19:02 +0200 Subject: [PATCH 40/47] a few addition words for base_dn; base_dn string in example, missing quote --- reference/functions/ldap-lookup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/functions/ldap-lookup.md b/reference/functions/ldap-lookup.md index 35a80c2..01f94d5 100644 --- a/reference/functions/ldap-lookup.md +++ b/reference/functions/ldap-lookup.md @@ -15,7 +15,7 @@ Otherwise an empty node-set is returned. * `url` The ldap URL (string) * `rdn` The (relative) distinguished name of the (system) user (string) * `rdnPassword` The password of the (system) user (string) -* `base_dn` The base distinguished name for the search (string) +* `base_dn` The base distinguished name for the directory, used for the search (string) * `userSearch` The filter for searching a user (string) * `userPassword` The user's password (string) * `attributes` A comma-separated list of attributes to return (string) @@ -32,7 +32,7 @@ In addition to the (default) `dn`, the `sAMAccountName` and `mail` from the entr concat("(&(objectClass=person)(memberOf=CN=Users,ou=People,dc=example,dc=com)(mail=john.doe@example.com))") "sAMAccountName,mail" - ldap-lookup($ldap_settings/url, $ldap_settings/bind_dn, $env/FLAT_SYSTEM_PASSWORD, $ldap_settings/base_dn, $userSearch, "myP4s5w0rD, $attributes) + ldap-lookup($ldap_settings/url, $ldap_settings/bind_dn, $env/FLAT_SYSTEM_PASSWORD, "dc=example,dc=com", $userSearch, "myP4s5w0rD", $attributes) { "status": 403, From 3c0fcb70b90fd4764c301b2c78bf40f3e4d289db Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 27 Aug 2020 17:07:53 +0200 Subject: [PATCH 41/47] docu for variables $body and $upstream --- reference/variables.md | 64 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/reference/variables.md b/reference/variables.md index c1b10ee..cbb5783 100644 --- a/reference/variables.md +++ b/reference/variables.md @@ -47,7 +47,7 @@ The actions [`request`](actions/request.md) and [`requests`](actions/requests.md { - "ok": { + "ok": { "url": "https://httpbin.org/status/200" }, "failure": { @@ -73,6 +73,7 @@ The actions [`request`](actions/request.md) and [`requests`](actions/requests.md "ok": { "url": "https://httpbin.org/status/200", "status": 200, + "cacheHit": false, "headers": { … } }, "failure": { @@ -159,6 +160,25 @@ conditions and produces the string `null` in placeholders: ``` +## `$body` + +The `$body` variable contains the request body. + +If the request body is JSON (`Content-Type: application/json`) `$body` contains the parsed JSON. You can access its properties with XPath expressions with a `json` segment before the top-level properties. E.g. + +```json +{ + "foo": 1, + "bar": { + "baz": true + } +} +``` +The value for `foo` can be accessed by `$body/json/foo`, the value for `baz` by `$body/json/bar/baz`. + +In other cases the content is stored in `$body` as a string and cannot be accessed by XPath. + + ## `$request` The `$request` variable contains information about the incoming client request, such as the URL, the request header fields and possibly the query component or cookies, if any were sent. @@ -228,6 +248,48 @@ paths: | https://example.com/api/bar | /** | /api/bar | /api | +## `upstream` + +The `$upstream` variable contains information about upstream responses. The properties for each upstream response are stored with the request ID ([`id` property](/reference/actions/request.md#id) or [`content` attribute](/reference/actions/request.md#syntax)). + +* `url` - The request URL (string) +* `status` - The response status code (integer) +* `cacheHit` - `true` if the response was served from a cache (see [`use-http-cache` or `force-cache-ttl` request options](/reference/actions/request.md#options)), `false` otherwise +* ``headers - The response headers, each with a lower-cased field name + +Example: +```xml + + + https://httpbin.org/status/200 + 200 + false + + Thu, 27 Aug 2020 14:12:33 GMT + text/html; charset=utf-8 + keep-alive + gunicorn/19.9.0 + * + true + + + + https://httpbin.org/status/500 + 500 + false + + Thu, 27 Aug 2020 14:12:33 GMT + text/html; charset=utf-8 + keep-alive + gunicorn/19.9.0 + * + true + + + +``` + + ## `$error` Client request and response validation, upstream connection and request and response validation errors, and those triggered by the [`error` action](/reference/actions/error.md) will store information about the error in `$error`. While initially empty, `$error` will have the following properties containing information about the most recent error, unless it is triggered by the `error` action: From caf91e46c0e2b4bd77d50b90cab9803fead502a2 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Thu, 27 Aug 2020 17:41:01 +0200 Subject: [PATCH 42/47] changed request IDs in example --- reference/variables.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/variables.md b/reference/variables.md index cbb5783..0ac90c1 100644 --- a/reference/variables.md +++ b/reference/variables.md @@ -260,7 +260,7 @@ The `$upstream` variable contains information about upstream responses. The prop Example: ```xml - + https://httpbin.org/status/200 200 false @@ -272,8 +272,8 @@ Example: * true - - + + https://httpbin.org/status/500 500 false @@ -285,7 +285,7 @@ Example: * true - + ``` From 2ec33c1612a9abf740395fc9e4f6cb1c071632f3 Mon Sep 17 00:00:00 2001 From: Felix Hassert Date: Fri, 28 Aug 2020 08:59:18 +0200 Subject: [PATCH 43/47] link variable list --- reference/variables.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/variables.md b/reference/variables.md index 0ac90c1..688ba81 100644 --- a/reference/variables.md +++ b/reference/variables.md @@ -8,12 +8,12 @@ A valid variable name starts with `$` followed by a letter `a`…`z` or `A`…`Z The following predefined variables exist: -* `$body`: client request body +* [`$body`](#usdbody): client request body * `$env`: [environment variables](/cookbook/envvars.md) -* `$request`: client request information +* [`$request`](#usdrequest): client request information * `$server`: server information -* `$upstream`: upstream response information -* `$error`: Contains information regarding the most recent error, but is initially empty. +* [`$upstream`](#usdupstream): upstream response information +* [`$error`](#usderror): Contains information regarding the most recent error, but is initially empty. Try the following flow with @@ -248,7 +248,7 @@ paths: | https://example.com/api/bar | /** | /api/bar | /api | -## `upstream` +## `$upstream` The `$upstream` variable contains information about upstream responses. The properties for each upstream response are stored with the request ID ([`id` property](/reference/actions/request.md#id) or [`content` attribute](/reference/actions/request.md#syntax)). From bd4af1eee8d5ffd7bada92f7046e3af1b14b1279 Mon Sep 17 00:00:00 2001 From: Felix Hassert Date: Fri, 28 Aug 2020 09:03:16 +0200 Subject: [PATCH 44/47] $upstream: typo and example --- reference/variables.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/reference/variables.md b/reference/variables.md index 688ba81..b69379b 100644 --- a/reference/variables.md +++ b/reference/variables.md @@ -255,7 +255,7 @@ The `$upstream` variable contains information about upstream responses. The prop * `url` - The request URL (string) * `status` - The response status code (integer) * `cacheHit` - `true` if the response was served from a cache (see [`use-http-cache` or `force-cache-ttl` request options](/reference/actions/request.md#options)), `false` otherwise -* ``headers - The response headers, each with a lower-cased field name +* `headers` - The response headers, each with a lower-cased field name Example: ```xml @@ -289,6 +289,12 @@ Example: ``` +To check, for example, if the status code of the response with `id="myRequest"` is successful you can use the following XPath expression: + +``` +$upstream/myRequest/status = 200 +``` + ## `$error` From d8e302d16b706782cde47d6b3558f0bdc664168f Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Fri, 28 Aug 2020 09:14:45 +0200 Subject: [PATCH 45/47] just ID instead of "markup" --- reference/variables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/variables.md b/reference/variables.md index b69379b..02a0d61 100644 --- a/reference/variables.md +++ b/reference/variables.md @@ -289,7 +289,7 @@ Example: ``` -To check, for example, if the status code of the response with `id="myRequest"` is successful you can use the following XPath expression: +To check, for example, if the status code of the response with the ID `myRequest` is successful you can use the following XPath expression: ``` $upstream/myRequest/status = 200 From 60ad2b62f666b80520083c6a6d093caee2604734 Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Fri, 28 Aug 2020 09:24:48 +0200 Subject: [PATCH 46/47] changelog entry for cacheHit --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 71270b1..35729d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ - [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#request-debugging) to enable debugging using the `Debug` request header, defaults to `false` - The [request option](/reference/actions/request.md#options) `force-cache-refresh` - The [`ldap-lookup()` function](/reference/functions/ldap-lookup.md) +- The `cacheHit` property in the [upstream response information (`$upstream`)](/reference/variables.md#usdupstream) ## Fixed From d8863f62d27970bcdb35a2186d9fb2f3b1057c3e Mon Sep 17 00:00:00 2001 From: Johannes Koch Date: Fri, 28 Aug 2020 09:56:11 +0200 Subject: [PATCH 47/47] Release 20200828 --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 35729d9..96cb1e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,10 @@ # Changelog -## Unreleased +## [20200828](https://hub.docker.com/r/sevenvaltechnologies/flatrunner/tags) ### Added -- [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#request-debugging) to enable debugging using the `Debug` request header, defaults to `false` +- [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#per-request-debugging) to enable debugging using the `Debug` request header, defaults to `false` - The [request option](/reference/actions/request.md#options) `force-cache-refresh` - The [`ldap-lookup()` function](/reference/functions/ldap-lookup.md) - The `cacheHit` property in the [upstream response information (`$upstream`)](/reference/variables.md#usdupstream)