Skip to content

Commit

Permalink
FEATURE: Log access logs directly to NATS
Browse files Browse the repository at this point in the history
  • Loading branch information
skurfuerst committed Mar 7, 2024
1 parent 94669d2 commit cbb5282
Show file tree
Hide file tree
Showing 9 changed files with 561 additions and 31 deletions.
165 changes: 134 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,41 +8,47 @@
> tl;dr: Pick whichever works for you - Open Source rocks :)
`caddy-nats-bridge` is a caddy module that allows the caddy server to interact with a
[NATS](https://nats.io/) server. This extension supports multiple patterns:
publish/subscribe, fan in/out, and request reply.
[NATS](https://nats.io/) server in meaningful ways.

**Main Features**:

- Connect NATS and HTTP in all possible directions (we call this **Bridging**)
- experimental: offload large HTTP bodies to JetStream
- NEW in 0.7: publish Caddy Log messages to NATS

The purpose of this project is to better bridge HTTP based services with NATS
The initial of this project was to better bridge HTTP based services with NATS
in a pragmatic and straightforward way. If you've been wanting to use NATS, but
have some use cases that still need to use HTTP, this may be a really good
option for you.

<!-- TOC -->
- [Caddy-NATS-Bridge](#caddy-nats-bridge)
- [Concept Overview](#concept-overview)
- [Installation](#installation)
- [Getting Started](#getting-started)
- [Connecting to NATS](#connecting-to-nats)
- [NATS -\> HTTP via `subscribe`](#nats---http-via-subscribe)
- [Placeholders for `subscribe`](#placeholders-for-subscribe)
- [Queue Groups](#queue-groups)
- [HTTP -\> NATS via `nats_request` (interested about response)](#http---nats-via-nats_request-interested-about-response)
- [Placeholders for `nats_request`](#placeholders-for-nats_request)
- [Extra headers for `nats_request`](#extra-headers-for-nats_request)
- [HTTP -\> NATS via `nats_publish` (fire-and-forget)](#http---nats-via-nats_publish-fire-and-forget)
- [Placeholders for `nats_publish`](#placeholders-for-nats_publish)
- [Extra headers for `nats_publish`](#extra-headers-for-nats_publish)
- [large HTTP payloads with store\_body\_to\_jetstream](#large-http-payloads-with-store_body_to_jetstream)
- [Development](#development)
<!-- TOC -->
option for you.

## Concept Overview
This extension supports multiple patterns:
publish/subscribe, fan in/out, and request reply.

![](./connectivity-modes.drawio.png)
Additionally, this extension supports using NATS as Log output.

The module works if you want to bridge *HTTP -> NATS*, and also *NATS -> HTTP* - both in unidirectional, and in
bidirectional mode.
<!-- TOC -->
* [Caddy-NATS-Bridge](#caddy-nats-bridge)
* [Installation](#installation)
* [Getting Started - NATS as Log Output](#getting-started---nats-as-log-output)
* [Getting Started - Bridging HTTP <-> NATS](#getting-started---bridging-http---nats)
* [Connecting to NATS](#connecting-to-nats)
* [Logging to NATS](#logging-to-nats)
* [Bridging HTTP <-> NATS](#bridging-http---nats)
* [NATS -> HTTP via `subscribe`](#nats---http-via-subscribe)
* [Placeholders for `subscribe`](#placeholders-for-subscribe)
* [Queue Groups](#queue-groups)
* [FAQ: HTTP URL Parameters](#faq-http-url-parameters)
* [HTTP -> NATS via `nats_request` (interested about response)](#http---nats-via-nats_request-interested-about-response)
* [Placeholders for `nats_request`](#placeholders-for-nats_request)
* [Extra headers for `nats_request`](#extra-headers-for-nats_request)
* [HTTP -> NATS via `nats_publish` (fire-and-forget)](#http---nats-via-nats_publish-fire-and-forget)
* [Placeholders for `nats_publish`](#placeholders-for-nats_publish)
* [Extra headers for `nats_publish`](#extra-headers-for-nats_publish)
* [large HTTP payloads with store_body_to_jetstream](#large-http-payloads-with-store_body_to_jetstream)
* [Development](#development)
<!-- TOC -->

## Installation
# Installation

To use `caddy-nats-bridge`, simply run the [xcaddy](https://github.com/caddyserver/xcaddy) build tool to create a
`caddy-nats-bridge` compatible caddy server.
Expand All @@ -56,7 +62,9 @@ go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
~/go/bin/xcaddy build --with github.com/sandstorm/caddy-nats-bridge
```

## Getting Started
# Getting Started - NATS as Log Output

(supported since version 0.7.0 of this extension).

Getting up and running with `caddy-nats-bridge` is pretty simple:

Expand All @@ -66,9 +74,59 @@ First [install NATS](https://docs.nats.io/running-a-nats-service/introduction/in
nats-server
```

> :bulb: To try this example, `cd examples/logs; ./build-run.sh`
>
> - [Caddyfile](./examples/getting-started/Caddyfile)
> - [build-run.sh](./examples/getting-started/build-run.sh)
Then create and run your Caddyfile:

```nginx
# run with: ./caddy run --config Caddyfile
{
nats {
url 127.0.0.1:4222
clientName "My Caddy Server"
}
}
http://127.0.0.1:8888 {
log {
output nats my.log.subject
}
respond "Hello World"
}
```

Then, you can can listen to `my.log` with the `nats` CLI, and do a HTTP Request to see it in the Logs:

```bash
nats subscribe 'my.log.>'

# in another console
curl http://127.0.0.1:8888
```

What has happened here?

1. You sent a HTTP request to Caddy
2. the access log is routed to NATS.


# Getting Started - Bridging HTTP <-> NATS

This is a more advanced scenario, but still getting up and running with `caddy-nats-bridge` is pretty simple:

First [install NATS](https://docs.nats.io/running-a-nats-service/introduction/installation) and make sure the NATS server is running:

```sh
nats-server
```


> :bulb: To try this example, `cd examples/getting-started; ./build-run.sh`
>
>
> - [Caddyfile](./examples/getting-started/Caddyfile)
> - [build-run.sh](./examples/getting-started/build-run.sh)
Expand Down Expand Up @@ -137,7 +195,7 @@ Because of the `nats_request` rule in the `Caddyfile` above, you can also reques
2. This has been forwarded to the NATS topic `cli.weather.Dresden`, which is responded by the `nats reply` tool above.
3. The NATS response is converted to a HTTP response.

## Connecting to NATS
# Connecting to NATS

To connect to `nats`, use the `nats` global option in your Caddyfile with the URL of the NATS server:

Expand Down Expand Up @@ -187,6 +245,51 @@ Configuration with all configuration options is specified below:
}
```

# Logging to NATS

Simple usage:

```
{
nats {
url nats://127.0.0.1:4222
}
}
my-domain.com {
# will use the "default" NATS server defined above
log nats my.log.subject
}
```

You can also specify the NATS server alias to use for logging; in the example below "myNatsServer":

```
{
nats {
url nats://127.0.0.1:4222
}
nats myNatsServer {
url nats://127.0.0.1:5444
}
}
my-domain.com {
# will use the "myNatsServer" NATS server defined above
log nats myNatsServer my.log.subject
}
```

This concept is fully pluggable; you can configure the log output any way you like in Caddy.

# Bridging HTTP <-> NATS

![](./connectivity-modes.drawio.png)

The module works if you want to bridge *HTTP -> NATS*, and also *NATS -> HTTP* - both in unidirectional, and in
bidirectional mode.


## NATS -> HTTP via `subscribe`

`caddy-nats-bridge` supports subscribing to NATS subjects in a few different flavors, depending on your needs:
Expand Down
4 changes: 4 additions & 0 deletions app.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"github.com/caddyserver/caddy/v2"
"github.com/caddyserver/caddy/v2/caddyconfig/httpcaddyfile"
"github.com/sandstorm/caddy-nats-bridge/body_jetstream"
"github.com/sandstorm/caddy-nats-bridge/logoutput"
"github.com/sandstorm/caddy-nats-bridge/natsbridge"
"github.com/sandstorm/caddy-nats-bridge/publish"
"github.com/sandstorm/caddy-nats-bridge/request"
Expand All @@ -24,4 +25,7 @@ func init() {
// store request body to Jetstream
caddy.RegisterModule(body_jetstream.StoreBodyToJetStream{})
httpcaddyfile.RegisterHandlerDirective("store_body_to_jetstream", body_jetstream.ParseStoreBodyToJetstream)

// logging output to NATS
caddy.RegisterModule(logoutput.LogOutput{})
}
15 changes: 15 additions & 0 deletions examples/logs/Caddyfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# run with: ./caddy run --config Caddyfile

{
nats {
url 127.0.0.1:4222
clientName "My Caddy Server"
}
}

http://127.0.0.1:8888 {
log {
output nats my.log.subject
}
respond "Hello World"
}
20 changes: 20 additions & 0 deletions examples/logs/build-run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env bash

# detect location of XCADDY
XCADDY=xcaddy
if [ -f ~/go/bin/xcaddy ]; then
XCADDY=~/go/bin/xcaddy
fi

# build caddy server if needed
if [ ! -f caddy ]; then
go get -u github.com/sandstorm/caddy-nats-bridge
$XCADDY build --with github.com/sandstorm/caddy-nats-bridge
fi

nats-server &
sleep 1
trap 'killall nats-server' EXIT

./caddy run --config Caddyfile

55 changes: 55 additions & 0 deletions integrationtest/caddyfile_adapt/logoutput_explicitServerAlias.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
localhost {
log {
output nats server2 my.topic.foo.bar
}
}
----------
{
"logging": {
"logs": {
"default": {
"exclude": [
"http.log.access.log0"
]
},
"log0": {
"writer": {
"output": "nats",
"serverAlias": "server2",
"subject": "my.topic.foo.bar"
},
"include": [
"http.log.access.log0"
]
}
}
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"localhost"
]
}
],
"terminal": true
}
],
"logs": {
"logger_names": {
"localhost": "log0"
}
}
}
}
}
}
}
55 changes: 55 additions & 0 deletions integrationtest/caddyfile_adapt/logoutput_minimal.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
localhost {
log {
output nats my.topic.foo.bar
}
}
----------
{
"logging": {
"logs": {
"default": {
"exclude": [
"http.log.access.log0"
]
},
"log0": {
"writer": {
"output": "nats",
"serverAlias": "default",
"subject": "my.topic.foo.bar"
},
"include": [
"http.log.access.log0"
]
}
}
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"localhost"
]
}
],
"terminal": true
}
],
"logs": {
"logger_names": {
"localhost": "log0"
}
}
}
}
}
}
}
Loading

0 comments on commit cbb5282

Please sign in to comment.