diff --git a/CHANGELOG.md b/CHANGELOG.md index f622d91..96cb1e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,23 @@ # Changelog -## Unreleased +## [20200828](https://hub.docker.com/r/sevenvaltechnologies/flatrunner/tags) + +### Added + +- [`FLAT_DEBUG_ALLOW_HEADER`](/reference/debugging.md#per-request-debugging) to enable debugging using the `Debug` request header, defaults to `false` +- The [request option](/reference/actions/request.md#options) `force-cache-refresh` +- The [`ldap-lookup()` function](/reference/functions/ldap-lookup.md) +- The `cacheHit` property in the [upstream response information (`$upstream`)](/reference/variables.md#usdupstream) + +## Fixed + +- Empty objects are no longer [logged](/cookbook/custom-logging.md#adding-a-log-field) as empty arrays. +- The [`json-to-csv()` function](/reference/functions/json-to-csv.md) allows `null` values in array entry objects. + +## Changed + +The [`log` action](/reference/actions/log.md) can no longer override [system log fields](/administration/logging.md#fields). + ## [20200519](https://hub.docker.com/r/sevenvaltechnologies/flatrunner/tags) diff --git a/administration/configuration.md b/administration/configuration.md index ed909c7..49245af 100644 --- a/administration/configuration.md +++ b/administration/configuration.md @@ -1,7 +1,7 @@ # Configuration -Certain settings of the FLAT runner can be configured using environment variables. +Certain settings of the FLAT server can be configured using environment variables. The chapter [Defining Env Vars](/cookbook/envvars.md#defining-env-vars) in the Cookbook provides information on how to set environment variables. @@ -11,7 +11,8 @@ The chapter [Defining Env Vars](/cookbook/envvars.md#defining-env-vars) in the C * `FLAT_SERVER_ROLE`: This setting is typically used to distinguish production systems from staging or development servers. Its value is accessible in the [`$server` variable](/reference/variables.md#predefined-variables) as `$server/role`. * `FLAT_STATUS_AUTH`: Username and password, separated by a `:` for access to the `php-fpm` and `httpd` status pages. The pages are disabled entirely if `FLAT_STATUS_AUTH` is not set. If enabled, the `httpd` status can be requested via HTTP at `/ServerStatus`, and the php-fpm status at `/FPMStatus?full`. - +* `FLAT_DEBUG_ALLOW_HEADER`: enable [per request debugging](/reference/debugging.md#request-debugging), defaults to `false` unless `FLAT_DEBUG_AUTH` is set. +* `FLAT_DEBUG_AUTH`: sets a password to protect [per request debugging](/reference/debugging.md#request-debugging). ### Request Timeouts diff --git a/cookbook/README.md b/cookbook/README.md index a3da9ac..86d55de 100644 --- a/cookbook/README.md +++ b/cookbook/README.md @@ -1,37 +1,39 @@ # Cookbook -## [How can I inspect the client request?](see-client-request.md) - -## [How can I pass an arbitrary header field to an upstream system?](pass-header-field-upstream.md) +## [Protecting Access using JWT Tokens](x-flat-jwt.md) ## [Proxying Requests to Upstream APIs (Swagger)](proxy-requests.md) -## [Forwarding a Request to an Upstream API (Flow)](forward-request-upstream.md) +## [Testing Templates](test-templates.md) -## [How can I pass response headers to the client?](pass-header-field-downstream.md) +## [Testing API Requests](test-api-request.md) -## [How can I increase the request timeout to deal with a slow upstream system?](request-timeout.md) +## [Testing with Backend Requests](test-backend.md) -## [Sending POST requests](upstream-post-requests.md) +## [File Serving](file-serving.md) -## [Working with JWT](jwt.md) +## [Error Handling](error-flow.md) -## [Auto Docs with Swagger UI](swagger-docs.md) +## [Extracting Common Initialization Flow Tasks](init-flow.md) -## [Using the Built-in Mocking](builtin-mocking.md) +## [Using Environment Variables](envvars.md) -## [Extracting Common Initialization Flow Tasks](init-flow.md) +## [Logging Custom Fields](custom-logging.md) -## [Testing Templates](test-templates.md) +## [Forwarding a Request to an Upstream API (Flow)](forward-request-upstream.md) -## [Testing API Requests](test-api-request.md) +## [How can I pass an arbitrary header field to an upstream system?](pass-header-field-upstream.md) -## [Testing with Backend Requests](test-backend.md) +## [How can I pass response headers to the client?](pass-header-field-downstream.md) -## [Using Environment Variables](envvars.md) +## [How can I increase the request timeout to deal with a slow upstream system?](request-timeout.md) -## [File Serving](file-serving.md) +## [Sending POST requests](upstream-post-requests.md) -## [Error Handling](error-flow.md) +## [Signing JWT](jwt.md) -## [Logging Custom Fields](custom-logging.md) +## [How can I inspect the client request?](see-client-request.md) + +## [Auto Docs with Swagger UI](swagger-docs.md) + +## [Using the Built-in Mocking](builtin-mocking.md) diff --git a/cookbook/custom-logging.md b/cookbook/custom-logging.md index bd762a7..cabbdc7 100644 --- a/cookbook/custom-logging.md +++ b/cookbook/custom-logging.md @@ -17,7 +17,7 @@ We want all of that! To play with logging, you need a FLAT project. If you don't have on, yet, have a look into the [tutorial](/tutorial/README.md) to [get started](/tutorial/README.md#getting-started). -## Reading logs +## Reading Logs Before start customizing our logs, we need to setup our workplace in order to see what we are doing. Where can we inspect FLAT logs? @@ -34,7 +34,7 @@ However, once you call your API with `curl` you will see JSON lines in your term $ curl localhost:8080/api/… ``` ```json -{"timestamp":"2019-10-15T15:49:13+00:00","type":"flat_access","requestID":"XaXqeccky00IowY@OUgG8AAAAAA","path":"/api/…","status":200,"method":"GET","agent":"curl/7.54.0","referrer":"","mime":"application/json"} +{"timestamp":"2019-10-15T15:49:13+00:00","type":"flat_access",…} ``` You can have the JSON colored and pretty printed by piping the output to the [`jq` command](https://stedolan.github.io/jq/): @@ -51,17 +51,23 @@ Now, the access log will be nicely readable on your terminal: "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", + "url": "http://localhost:8080/api/…", "path": "/api/…", "status": 200, "method": "GET", "agent": "curl/7.54.0", "referrer": "", - "mime": "application/json" + "mime": "application/json", + "realtime": 0.08935, + "bytes": 1387, + "requestbytes": 0, + "flow": "flow.xml", + "requestmime": "", + "location": "" } ``` - -## Adding a log field +## Adding a Log Field Now that we know where our logs go, we're finally ready to add fields! @@ -90,20 +96,14 @@ $ curl localhost:8080/api/… { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject" } ``` Our first custom log field has hit the terminal! -## Adding dynamic log fields +## Adding Dynamic Log Fields With [JSON templates](/reference/templating/README.md) you can use expressions to dynamically set field names and values. @@ -127,13 +127,7 @@ Notice, how we can provide default values for missing data. If _both_ of the var { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject", "stage": "unknown" } @@ -141,7 +135,7 @@ Notice, how we can provide default values for missing data. If _both_ of the var `stage` is always defined. But `location` is missing, because its expression has evaluated to `null`. Nulled fields don't show up in the log. -## Structured log fields +## Structured Log Fields Your custom log fields are not restricted to the top-level field-list. If you have more fields, that belong together, you can group them in an object. @@ -159,19 +153,11 @@ In our case, we might want to gather information about the environment: ``` -Notice, how we can provide default values for missing data. If _both_ of the variables are not set, a log may look like this: - ```json { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject", "env": { "stage": "production", @@ -180,9 +166,9 @@ Notice, how we can provide default values for missing data. If _both_ of the var } ``` -## Using request data +## Using Request Data -Client's provide a lot of useful information in the HTTP request headers. We can easily use them to augment our log: +Clients provide a lot of useful information in the HTTP request headers. We can easily use them to augment our log: ```xml @@ -209,13 +195,7 @@ $ curl -H "X-Forwarded-Proto: https" localhost:8080/api/… { "timestamp": "2019-10-15T15:49:13+00:00", "type": "flat_access", - "requestID": "XaXqeccky00IowY@OUgG8AAAAAA", - "path": "/api/…", - "status": 200, - "method": "GET", - "agent": "curl/7.54.0", - "referrer": "", - "mime": "application/json", + … "project": "myAPIProject", "env": { "stage": "production", @@ -233,7 +213,7 @@ For logging alone, the JSON armor prevents format breakouts and log attacks. HTT ## Testing -Log augmentation with the [`log` action](/reference/actions/log.md) belongs to our project code. Therefore, we would like to test that! This can be done by using the [`get-log()` function](/reference/functions/get-log.md) in a [FLAT test](/reference/testing/README.md). +Log augmentation with the [`log` action](/reference/actions/log.md) belongs to our project code. Therefore, we would like to test that! This can be done by using the [`get-log()` function](/reference/functions/get-log.md) in a [FLAT test](/reference/testing/README.md). Put this into `tests/loggging.xml`: ```xml @@ -260,7 +240,7 @@ $ flat test tests/logging.xml ok 1 tests/logging.xml: 1 assertions ``` -In a real project, you would rather send a request to your FLAT API and check if the log was written as expected +In a real project, you would rather send a request to your FLAT API and check if the log was written as expected: ```xml diff --git a/cookbook/jwt.md b/cookbook/jwt.md index 1a9a24c..e79b5d5 100644 --- a/cookbook/jwt.md +++ b/cookbook/jwt.md @@ -101,5 +101,6 @@ Note that, with `RSASSA` based algorithms, you have to specify the algorithm in ## See also -* [`jwt-encode()`](/reference/functions/jwt-encode.md) -* [`jwt-decode()`](/reference/functions/jwt-decode.md) +* [`jwt-encode()`](/reference/functions/jwt-encode.md) (reference) +* [`jwt-decode()`](/reference/functions/jwt-decode.md) (reference) +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) diff --git a/cookbook/x-flat-jwt.md b/cookbook/x-flat-jwt.md new file mode 100644 index 0000000..7bd4d5a --- /dev/null +++ b/cookbook/x-flat-jwt.md @@ -0,0 +1,354 @@ +# Protecting Access using JWT Tokens + +Imagine you had a [proxy](/reference/OpenAPI/routing.md#assigning-flat-proxies) to an API, e.g. httpbin.org. + +swagger.yml: +```yaml +swagger: "2.0" +basePath: / +paths: + /httpbin/**: + x-flat-proxy: + origin: https://httpbin.org + stripEndpoint: true +``` + +Sending a request to FLAT running on localhost port 8080 results in: + +```sh +$ curl -i http://localhost:8080/httpbin/anything +HTTP/1.1 200 OK +… +Content-Type: application/json + +{ + "args": {}, + "data": "", + "files": {}, + "form": {}, + "headers": { + "Accept": "*/*", + "Accept-Encoding": "deflate, gzip", + "Host": "httpbin.org", + "User-Agent": "curl/7.29.0", + "X-Amzn-Trace-Id": "…" + }, + "json": null, + "method": "GET", + "origin": "…", + "url": "https://httpbin.org/anything" +} +``` + +That's what you would expect from httpbin.org, right? + +## Restricting access: Swagger Security and `x-flat-jwt` + +Now, you don't want anyone except authorized users to use this proxy. This is typically achieved with access tokens. Some tokens are JSON Web Tokens ([JWT](https://tools.ietf.org/html/rfc7519)), while others are opaque. + +Swagger has a two-part feature to describe protected access to routes: `securityDefinitions` (what sort of protection is applied …) and `security` (… to which routes), e.g.: + +```yaml +swagger: "2.0" +basePath: / +securityDefinitions: + JWTCookie: + type: apiKey + in: header + name: Cookie +paths: + /httpbin/**: + security: + - JWTCookie: [] +``` + +This defines a security scheme object (named `JWTCookie`), meaning that some sort of cookie is needed to access certain routes. This is applied to the [wildcard path](/reference/OpenAPI/differences.md#wildcard-paths) `/httpbin/**`. + +This documentation feature, with some extensions, is used to make FLAT actually permit access to the route only if a valid JWT token is presented. + +First, we define the name of the cookie expected to accompany the API request: + +```yaml +… +securityDefinitions: + JWTCookie: + type: apiKey + in: header + name: Cookie + x-flat-cookiename: authtoken # ⬅ specify the cookie name +… +``` + +Then we specify the [configuration](/reference/OpenAPI/security.md#the-x-flat-jwt-field) for decoding the JWT token: + +```yaml +… + JWTCookie: + type: apiKey + in: header + name: Cookie + x-flat-cookiename: authtoken + x-flat-jwt: # ⬅ our JWT configuration: + key: # ⬅ the key to decode the JWT … + file: pubkey.pem # ⬅ … is read from the file pubkey.pem + alg: RS256 # ⬅ the signing algorithm is RS256 +… +``` + +The specified key is a public key, read from the file pubkey.pem, e.g.: +``` +-----BEGIN PUBLIC KEY----- +MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGSd+sSTss2uOuVJKpumpFAaml +t1CWLMTAZNAabF71Ur0P6u833RhAIjXDSA/QeVitzvqvCZpNtbOJVegaREqLMJqv +FOUkFdLNRP3f9XjYFFvubo09tcjX6oGEREKDqLG2MfZ2Z8LVzuJc6SwZMgVFk/63 +rdAOci3W9u3zOSGj4QIDAQAB +-----END PUBLIC KEY----- +``` + +That's all. Now FLAT will only permit requests if they supply a token that bear an RS256 signature that was created with the private key that matches the given public key. + +Usually, you would get the key and algorithm from your identity provider (e.g. an OAuth2 authorization server). That service would be responsible for issuing JWT tokens for your users. + +For this tutorial we have prepared a couple of JWT tokens for you to try out different situations: + +``` +eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4 +``` + +``` +eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU +``` + +Let's give it a try: + +```sh +$ curl -i http://localhost:8080/httpbin/anything +HTTP/1.1 403 Forbidden +… +Content-Type: application/json + +{ + "error": { + "message": "Security violation", + "status": 403, + "requestID": "Xsz@QAv8CEBZfWbmQhKLIQAAAII", + "info": [ + "JWT Security (JWTCookie): No Cookie header sent" + ], + "code": 3206 + } +} +``` + +Ah, yes, we forgot to present a token in the `authtoken` cookie. But we see, that the protection works. + +Let's try again with the first token: + +```sh +$ curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIiwiZXhwIjoxNTkwNDkxNTI4fQ.lJnUpBzMx84_5yigeHeLw4f8sbdSdu_7fWr1--t7EAp8v8K-kSmVYUGnR0Jx4o_ZE84N2M72Kn1pKssrzgTHsFi7txcZHHz_JqgnPgKqsZwjrmWDC-XVvdrSXjAsPO6wn0qy3KEMT1y6Z8YQA4ZyzA1dDsRRIUFiNrgF6_b5pC4" http://localhost:8080/httpbin/anything +HTTP/1.1 403 Forbidden +… +Content-Type: application/json + +{ + "error": { + "message": "Security violation", + "status": 403, + "requestID": "Xsz80Av8CEBZfWbmQhKLIAAAAIE", + "info": [ + "JWT Security (JWTCookie): Invalid JWT: Token has expired." + ], + "code": 3206 + } +} +``` + +Hmm, expired. So this one is too old. (Access tokens typically have a restricted period of use.) + +OK, let's use the other token: + +```sh +$ curl -i -H "Cookie: authtoken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +HTTP/1.1 200 OK +… +Content-Type: application/json + +{ + "args": {}, + "data": "", + "files": {}, + "form": {}, + "headers": { + "Accept": "*/*", + "Accept-Encoding": "deflate, gzip", + "Host": "httpbin.org", + "User-Agent": "curl/7.29.0", + "X-Amzn-Trace-Id": "…" + }, + "json": null, + "method": "GET", + "origin": "…", + "url": "https://httpbin.org/anything" +} +``` + +Tada! + +By the way, apart from cookies, this also works similarly with the `Authorization: Bearer …` header: + +``` +… +securityDefinitions: + JWTBearer: + type: apiKey + in: header + name: Authorization # ⬅ + x-flat-jwt: + key: + file: pubkey.pem + alg: RS256 +paths: + /httpbin/**: + security: + - JWTBearer: [] +``` + +You can try that with + +```sh +$ curl -i -H "Authorization: Bearer eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +``` + +But there are two additional features that can be quite handy: `out-header` and `out-var`. + +## Sending JWT claims upstream: `out-header` + +With [`out-header`](/reference/OpenAPI/security.md#forwarding-jwt-upstream) you can send the whole set of claims from the JWT upstream: + +```yaml: +… + x-flat-jwt: + key: + file: pubkey.pem + alg: RS256 + out-header: JWT # ⬅ the name of the request header with the JWT claims +… +``` + +```sh +$ curl -i -H "Cookie: authtoken=eybGciOiJSUzI1NiJ9.eyJzdWIiOiJzb21lX3VzZXIiLCJpc3MiOiJzb21lX3Byb3ZpZGVyIn0.bNXv28XmnFBjirPbCzBqyfpqHKo6PpoFORHsQ-80IJLi3IhBh1y0pFR0wm-2hiz_F7PkGQLTsnFiSXxCt1DZvMstbQeklZIh7O3tQGJyCAi-HRVASHKKYqZ_-eqQQhNr8Ex00qqJWD9BsWVJr7Q526Gua7ghcttmVgTYrfSNDzU" http://localhost:8080/httpbin/anything +HTTP/1.1 200 OK +… +Content-Type: application/json + +{ +… + "headers": { +… + "Jwt": "{\"sub\":\"some_user\",\"iss\":\"some_provider\"}", +… + }, +… +} +``` + +## Accessing JWT claims: `out-var` + +With `out-var` you can specify the name of a [variable](/reference/variables.md) where FLAT will store the JSON claims encoded in the JWT, in order to make them available for further processing. E.g. + + +```yaml +… + x-flat-jwt: + key: + file: pubkey.pem + alg: RS256 + out-header: JWT + out-var: $the_claims # ⬅ +… +``` + +We can log the claims by adding a [`log` action](/reference/actions/log.md) to an [init flow](/reference/OpenAPI/routing.md#init-flow): + +```yaml +swagger: "2.0" +… +x-flat-init: init.xml +paths: +… +``` + +with init.xml: + +```xml + + + { + "JWT-Claims": {{ $the_claims }} + } + + +``` + +If you look at the [FLAT logs](/administration/logging.md) and try again with a valid token, you'll notice: + +``` +{…,"type":"flat_access",…,"JWT-Claims":{"sub":"some_user","iss":"some_provider"}} +``` + +Here you see the two claims from the JWT token. + +## All files together + +swagger.yaml: +```yaml +swagger: "2.0" +basePath: / +securityDefinitions: + JWTCookie: + type: apiKey + in: header + name: Cookie + x-flat-cookiename: authtoken + x-flat-jwt: + key: + file: pubkey.pem + alg: RS256 + out-var: $the_claims + out-header: JWT +x-flat-init: init.xml +paths: + /httpbin/**: + security: + - JWTCookie: [] + x-flat-proxy: + origin: https://httpbin.org + stripEndpoint: true +``` + +pubkey.pem: +``` +-----BEGIN PUBLIC KEY----- +MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGSd+sSTss2uOuVJKpumpFAaml +t1CWLMTAZNAabF71Ur0P6u833RhAIjXDSA/QeVitzvqvCZpNtbOJVegaREqLMJqv +FOUkFdLNRP3f9XjYFFvubo09tcjX6oGEREKDqLG2MfZ2Z8LVzuJc6SwZMgVFk/63 +rdAOci3W9u3zOSGj4QIDAQAB +-----END PUBLIC KEY----- +``` + +init.xml: +```xml + + + { + "JWT-Claims": {{ $the_claims }} + } + + +``` + +## See also + +* [FLAT Security](/reference/OpenAPI/security.md) (reference) +* [Routing: Init Flow and FLAT Proxies](/reference/OpenAPI/routing.md) (reference) +* [Encoding and Decoding JWT](jwt.md) (cookbook) diff --git a/reference/OpenAPI/security.md b/reference/OpenAPI/security.md index 58aa7a7..c3f8e1e 100644 --- a/reference/OpenAPI/security.md +++ b/reference/OpenAPI/security.md @@ -12,7 +12,7 @@ The `x-flat-jwt` field references an object with fields describing the expected * `key` - REQUIRED. The key to decode the JSON Web Signature (JWS). This can either be specified with a value, or by referencing a file (`file`) or an environment variable (`env`). * `alg` - The signing algorithm the JWS is expected to be created with. This can either be specified with a value, or by referencing a file (`file`) or an environment variable (`env`). See the [`algorithm` parameter for `jwt-decode()`](/reference/functions/jwt-decode.md) for more information. * `out-var` - The name of the variable in which the JWT is stored (must be a proper variable name, starting with `$`; default: `"$jwt"`). -* `out-header` - The name of the HTTP header that shall carry the JWT +* `out-header` - The name of an HTTP request header that shall carry the JWT in upstream requests. * `claims` - An object with claims the JWT payload is expected to contain. The field names are the claim names, the expected claim value is specified either with a value, or by referencing a file (`file`) or an environment variable (`env`). The token is considered valid if all of the following are true: @@ -22,6 +22,8 @@ The token is considered valid if all of the following are true: * the JWT contains the expected claims, if any are configured, * the JWT can be stored in a variable. +`$jwt` or the alternative variable specified in `out-var` and the header specified in `out-header` will be unset if the token is not valid. + ## JWT in `Authorization` Header Use a [Security Scheme Object](https://swagger.io/specification/v2/#securitySchemeObject) with `type: apiKey`, `in: header` and `name: Authorization` if the JWT is expected to be sent as a bearer token in an `Authorization` header; e.g.: @@ -158,7 +160,7 @@ paths: ## Forwarding JWT Upstream -The claims of an incoming JWT are stored in `$jwt` – or in any other global +The claims of an incoming JWT are stored in `$jwt` – or in the global [variable](/reference/variables.md) that you specify in the `out-var` property of `x-flat-jwt`. In a [`request`](/reference/actions/request.md) or @@ -226,3 +228,7 @@ paths: security: - JWTHeaderAuth: [] ``` + +## See also + +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) diff --git a/reference/actions/log.md b/reference/actions/log.md index 5ade2b8..8d558dd 100644 --- a/reference/actions/log.md +++ b/reference/actions/log.md @@ -19,6 +19,8 @@ The action takes a JSON object as its argument. The object may contain nested fi All name/value pairs of the object are registered for logging. When the system writes the `flat_access` event the registered fields are included in that log line. +[System log fields](/administration/logging.md#fields) like `timestamp` cannot be overriden. + You can call the action multiple times. Fields of the same name are overwritten. However, nested fields are merged into the previously registered log fields. ```xml diff --git a/reference/actions/request.md b/reference/actions/request.md index 6d312f2..d710ce3 100644 --- a/reference/actions/request.md +++ b/reference/actions/request.md @@ -306,6 +306,7 @@ The `options` property sets the request options. Its value must be a JSON object * `proxy-credentials` - Authentication data for the proxy server (type: `string`) * `use-http-cache` - Whether to enable the HTTP cache (type: `boolean`, default: `false`). This option is mutually exclusive with `force-cache-ttl`. * `force-cache-ttl` - All resources are cached with a fixed time-to-live, ignoring all cache response headers and other obstacles for caching (type: `integer`). A number > 0 enables this kind of caching, specifying a lifetime in seconds. This option is mutually exclusive with `use-http-cache`. +* `force-cache-refresh` - if `true`, fetch a fresh response from upstream, even if a valid cached response is available (type: `boolean`, default: `false`). This works with both `use-http-cache` or `force-cache-ttl`. * `respect-client-cache-headers` - Whether to respect incoming cache headers (type: `boolean`, default: `false`) * `follow-redirects` - Whether to automatically follow redirects (type: `boolean`, default: `false`) * `max-redirects` - Maximum number of consecutive redirects to follow (type: `integer`) @@ -334,11 +335,6 @@ Example: } ``` -> 📎 -> Any request options set in a `conf/sources.xml` file that match the requested domain and path -> will also be applied to your request. Options directly set in the action have precedence, though. -> We do **not** recommend using `conf/sources.xml`. - ## See also diff --git a/reference/debugging.md b/reference/debugging.md index 79ccd52..80c1502 100644 --- a/reference/debugging.md +++ b/reference/debugging.md @@ -32,8 +32,11 @@ sending the desired parameters in the HTTP `Debug` header field, for example: $ curl -H "Debug: request:info:append" localhost:8080 ``` -Per request debugging may be password-protected via the `$FLAT_DEBUG_AUTH` environment variable. -If such authorization is required, the password has to be appended as `;auth=…` in the `Debug` header, for example: +To use per request debugging, you must enable it by setting the +`$FLAT_DEBUG_ALLOW_HEADER` environment variable to `true` or by turning on +password protection via the `$FLAT_DEBUG_AUTH` environment variable. +If authorization is required, the password has to be appended as `;auth=…` +in the `Debug` header, for example: ```bash $ curl -H "Debug: flow::append; auth=Pas5W0Rd" localhost:8080 diff --git a/reference/functions/README.md b/reference/functions/README.md index 94a32d5..3d4c572 100644 --- a/reference/functions/README.md +++ b/reference/functions/README.md @@ -85,6 +85,7 @@ * [`has-class()`](has-class.md) * [`html-parse()`](html-parse.md) * [`id()` ↗](https://developer.mozilla.org/en/XPath/Functions/id) +* [`ldap-lookup()`](ldap-lookup.md) * [`lang()` ↗](https://developer.mozilla.org/en/XPath/Functions/lang) * [`last()` ↗](https://developer.mozilla.org/en/XPath/Functions/last) * [`local-name()` ↗](https://developer.mozilla.org/en/XPath/Functions/local-name) diff --git a/reference/functions/json-to-csv.md b/reference/functions/json-to-csv.md index 279d2f7..78570a4 100644 --- a/reference/functions/json-to-csv.md +++ b/reference/functions/json-to-csv.md @@ -6,7 +6,7 @@ string json-to-csv(OXN array) The `json-to-csv` function translates the given [OXN `array`](/reference/templating/oxn.md) into CSV as described in [RFC 4180](https://tools.ietf.org/html/rfc4180). -The array entries must either be arrays or "flat" objects with `number`, `boolean` or `string` values. +The array entries must either be arrays or "flat" objects with `number`, `boolean`, `string`, or `null` values. If any errors occur, an empty `string` is returned. @@ -19,7 +19,8 @@ Example: array of arrays [ [ 1, " foo ", true ], [ 2, "ba, r", false ], - [ 3.21, "q\"u\"x", true ] + [ 3.21, "q\"u\"x", true ], + [ '', null, '' ] ] json-to-csv($arr) @@ -33,6 +34,7 @@ creates the following output: 1, foo ,true 2,"ba, r",false 3.21,"q""u""x",true +,, ``` @@ -45,7 +47,8 @@ Example: array of "flat" objects [ { "A": 1, "B": " foo ", "C": true }, { "A": 2, "B": "ba, r", "C": false }, - { "A": 3.21, "B": "q\"u\"x", "C": true } + { "A": 3.21, "B": "q\"u\"x", "C": true }, + { "A": "", "B": null, "C": "" } ] json-to-csv($arr) diff --git a/reference/functions/jwt-decode.md b/reference/functions/jwt-decode.md index c0ce4fc..b70c9f4 100644 --- a/reference/functions/jwt-decode.md +++ b/reference/functions/jwt-decode.md @@ -48,5 +48,6 @@ The unpacked web token is stored in `$jwt`, which provides easy access to its co ## See also -* [`jwt-encode()`](jwt-encode.md) -* [Encoding and Decoding JWT](/cookbook/jwt.md) +* [`jwt-encode()`](jwt-encode.md) (reference) +* [Encoding and Decoding JWT](/cookbook/jwt.md) (cookbook) +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) diff --git a/reference/functions/jwt-encode.md b/reference/functions/jwt-encode.md index 9218616..e236a2c 100644 --- a/reference/functions/jwt-encode.md +++ b/reference/functions/jwt-encode.md @@ -46,5 +46,6 @@ After 600 seconds the token becomes invalid: ## See also -* [`jwt-decode()`](jwt-decode.md) -* [Encoding and Decoding JWT](/cookbook/jwt.md) +* [`jwt-decode()`](jwt-decode.md) (reference) +* [Encoding and Decoding JWT](/cookbook/jwt.md) (cookbook) +* [Protecting Access using JWT Tokens](/cookbook/x-flat-jwt.md) (cookbook) diff --git a/reference/functions/ldap-lookup.md b/reference/functions/ldap-lookup.md new file mode 100644 index 0000000..01f94d5 --- /dev/null +++ b/reference/functions/ldap-lookup.md @@ -0,0 +1,61 @@ +# `ldap-lookup()` + +``` +OXN-node-set ldap-lookup(string url, string rdn, string rdnPassword, string base_dn, string userSearch, string userPassword, string attributes) +``` + +The `ldap-lookup()` function connects to an LDAP server with the given `url`, `rdn` and `rdnPassword`. +It then searches for a user by the given `userSearch`. +If a user is found, it connects with the user's DN and the given `userPassword`. +If the password is correct, an [OXN](/reference/templating/oxn.md) JSON document is returned with at least the user's `dn` and additional attributes given by `attributes`. +Otherwise an empty node-set is returned. + +## Parameters + +* `url` The ldap URL (string) +* `rdn` The (relative) distinguished name of the (system) user (string) +* `rdnPassword` The password of the (system) user (string) +* `base_dn` The base distinguished name for the directory, used for the search (string) +* `userSearch` The filter for searching a user (string) +* `userPassword` The user's password (string) +* `attributes` A comma-separated list of attributes to return (string) + + +## Example + +In the following example, the LDAP server is connected with the DN given in `$ldap_settings/bind_dn` and the password from `$env/FLAT_SYSTEM_PASSWORD`. +The given filter is used to search for an entry of a person which is a member of a group `Users` and has the email address `john.doe@example.com`. +In addition to the (default) `dn`, the `sAMAccountName` and `mail` from the entry are added to the result. + +```xml + + concat("(&(objectClass=person)(memberOf=CN=Users,ou=People,dc=example,dc=com)(mail=john.doe@example.com))") + "sAMAccountName,mail" + + ldap-lookup($ldap_settings/url, $ldap_settings/bind_dn, $env/FLAT_SYSTEM_PASSWORD, "dc=example,dc=com", $userSearch, "myP4s5w0rD", $attributes) + + { + "status": 403, + "message": "ldap-lookup() failed" + } + + +``` + +The result in the case of success, is +```json +{ + "dn": "cn=John Doe,ou=People,dc=example,dc=com", + "sAMAccountName": "john.doe", + "mail": "john.doe@example.com" +} +``` + +In a real setup you would read the user (here `john.doe@example.com`) and password parameters from user input, such as the JSON request body (e.g. `$body/json/username` and `$body/json/password`). + +The attributes returned from the function can then be used to set claims in a JWT token with [`jwt-encode()`](/reference/functions/jwt-encode.md). + +## See also + +* [`jwt-encode()`](jwt-encode.md) +* [Encoding and Decoding JWT](/cookbook/jwt.md) diff --git a/reference/variables.md b/reference/variables.md index c1b10ee..02a0d61 100644 --- a/reference/variables.md +++ b/reference/variables.md @@ -8,12 +8,12 @@ A valid variable name starts with `$` followed by a letter `a`…`z` or `A`…`Z The following predefined variables exist: -* `$body`: client request body +* [`$body`](#usdbody): client request body * `$env`: [environment variables](/cookbook/envvars.md) -* `$request`: client request information +* [`$request`](#usdrequest): client request information * `$server`: server information -* `$upstream`: upstream response information -* `$error`: Contains information regarding the most recent error, but is initially empty. +* [`$upstream`](#usdupstream): upstream response information +* [`$error`](#usderror): Contains information regarding the most recent error, but is initially empty. Try the following flow with @@ -47,7 +47,7 @@ The actions [`request`](actions/request.md) and [`requests`](actions/requests.md { - "ok": { + "ok": { "url": "https://httpbin.org/status/200" }, "failure": { @@ -73,6 +73,7 @@ The actions [`request`](actions/request.md) and [`requests`](actions/requests.md "ok": { "url": "https://httpbin.org/status/200", "status": 200, + "cacheHit": false, "headers": { … } }, "failure": { @@ -159,6 +160,25 @@ conditions and produces the string `null` in placeholders: ``` +## `$body` + +The `$body` variable contains the request body. + +If the request body is JSON (`Content-Type: application/json`) `$body` contains the parsed JSON. You can access its properties with XPath expressions with a `json` segment before the top-level properties. E.g. + +```json +{ + "foo": 1, + "bar": { + "baz": true + } +} +``` +The value for `foo` can be accessed by `$body/json/foo`, the value for `baz` by `$body/json/bar/baz`. + +In other cases the content is stored in `$body` as a string and cannot be accessed by XPath. + + ## `$request` The `$request` variable contains information about the incoming client request, such as the URL, the request header fields and possibly the query component or cookies, if any were sent. @@ -228,6 +248,54 @@ paths: | https://example.com/api/bar | /** | /api/bar | /api | +## `$upstream` + +The `$upstream` variable contains information about upstream responses. The properties for each upstream response are stored with the request ID ([`id` property](/reference/actions/request.md#id) or [`content` attribute](/reference/actions/request.md#syntax)). + +* `url` - The request URL (string) +* `status` - The response status code (integer) +* `cacheHit` - `true` if the response was served from a cache (see [`use-http-cache` or `force-cache-ttl` request options](/reference/actions/request.md#options)), `false` otherwise +* `headers` - The response headers, each with a lower-cased field name + +Example: +```xml + + + https://httpbin.org/status/200 + 200 + false + + Thu, 27 Aug 2020 14:12:33 GMT + text/html; charset=utf-8 + keep-alive + gunicorn/19.9.0 + * + true + + + + https://httpbin.org/status/500 + 500 + false + + Thu, 27 Aug 2020 14:12:33 GMT + text/html; charset=utf-8 + keep-alive + gunicorn/19.9.0 + * + true + + + +``` + +To check, for example, if the status code of the response with the ID `myRequest` is successful you can use the following XPath expression: + +``` +$upstream/myRequest/status = 200 +``` + + ## `$error` Client request and response validation, upstream connection and request and response validation errors, and those triggered by the [`error` action](/reference/actions/error.md) will store information about the error in `$error`. While initially empty, `$error` will have the following properties containing information about the most recent error, unless it is triggered by the `error` action: diff --git a/tutorial/README.md b/tutorial/README.md index 8f5f6b2..aadd160 100644 --- a/tutorial/README.md +++ b/tutorial/README.md @@ -1045,10 +1045,13 @@ If we also set the debug sink to `inline` or `append`, the output will be included in the HTTP response rather than the log file, for example: ```bash -$ curl --header Debug:time:info:inline localhost:8080/html +$ curl --header Debug:time:debug:inline localhost:8080/html ``` The latter is the only reasonable way to debug on a production system where we usually can't access the log file. +While the [`flat` command line tool](/reference/flat-cli.md) enables header debugging by default, +the appropriate environment variables must be set to enable it in other environments, typically +[`$FLAT_DEBUG_AUTH`](/administration/configuration.md). > 📎 > If the `$FLAT_DEBUG_AUTH` environment variable is set (this is a **must** on production systems!), FLAT requires a password to enable debugging by means of the `Debug` header, for example `--header 'Debug: *:warn:append; auth=Pa5sw0rd'`.