Kustomize base for fluent-bit

README.md

@@ -1,7 +1,5 @@
 # Kubernetes Logging with Fluent Bit
 
-
-
 [Fluent Bit](http://fluentbit.io) is a lightweight and extensible __Log Processor__ that comes with full support for Kubernetes:
 
 - Read Kubernetes/Docker log files from the file system or through systemd Journal
@@ -16,72 +14,36 @@ This repository contains a set of Yaml files to deploy Fluent Bit which consider
 
 ```
 $ kubectl create namespace logging
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
+$ kubectl apply -k ./base
 ```
 
+(`./base` can be replaced with a URL if [#90](https://github.com/fluent/fluent-bit-kubernetes-logging/pull/90) gets merged)
+
 If you are deploying fluent-bit on openshift, you additionally need to run:
 
 ```
 $ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-openshift-security-context-constraints.yaml
 ```
 
-#### Fluent Bit to Elasticsearch
-
-The next step is to create a ConfigMap that will be used by our Fluent Bit DaemonSet:
-
-```
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
-```
-
-If the cluster uses a CRI runtime, like containerd or CRI-O, change the `Parser` described in `input-kubernetes.conf` from docker to cri.
-
-Fluent Bit DaemonSet ready to be used with Elasticsearch on a normal Kubernetes Cluster:
-
-```
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml
-```
-
-#### Fluent Bit to Elasticsearch on Minikube
-
-If you are using Minikube for testing purposes, use the following alternative DaemonSet manifest:
-
-```
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds-minikube.yaml
-```
-
-#### Fluent Bit to Kafka
-
-Create a ConfigMap that will be used by our Fluent Bit DaemonSet:
-
-```
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/kafka/fluent-bit-configmap.yaml
-```
-
-Fluent Bit DaemonSet ready to be used with Kafka on a normal Kubernetes Cluster:
+The base only configures [Stdout](https://docs.fluentbit.io/manual/pipeline/outputs/standard-output) output.
+Log records are found using `kubectl logs` on fluentbit pods.
+This is a good way to experiment with configuration.
+Switch to [Format](https://docs.fluentbit.io/manual/pipeline/outputs/standard-output#configuration-parameters) `msgpack` to also see the "tag" (helpful when developing your pipeline).
 
-```
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/kafka/fluent-bit-ds.yaml
-```
+## Logs ingested to Loki
 
-#### Fluent Bit to Elasticsearch on Minikube
+This repository serves as example of Kubernetes yaml for a log processing stack,
+not as example of [how](https://docs.fluentbit.io/manual/concepts/data-pipeline) Fluent-bit can process, refine and forward log entries.
 
-If you are using Minikube for testing purposes, use the following alternative DaemonSet manifest:
-
-```
-$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds-minikube.yaml
-```
+Use the [loki](./loki) base to replace Stdout with forwarding to a standalone Loki instance. Loki can do some processing at [query time](https://grafana.com/docs/loki/latest/logql/) which helps keep this repo light on pipeline config.
 
-## Details
+The [loki-plugin](./loki-plugin) base uses an out-of-tree plugin with `DropSingleKey`
+so that the log records forwarded are the actual logged lines from pods,
+but still queryable on labels like `pod` and `container`.
 
-The default configuration of Fluent Bit makes sure of the following:
+See the [lokitest](./loki/test/lokitest-job.yaml) Job for some examples of how to consume logs through Loki.
 
-- Consume all containers logs from the running Node.
-- The [Tail input plugin](http://fluentbit.io/documentation/0.12/input/tail.html) will not append more than __5MB__  into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for [backpressure](http://fluentbit.io/documentation/0.13/configuration/backpressure.html) scenarios.
-- The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically _labels_ and _annotations_. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.
-- The default backend in the configuration is Elasticsearch set by the [Elasticsearch Output Plugin](http://fluentbit.io/documentation/0.13/output/elasticsearch.html). It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
-- There is an option called __Retry_Limit__ set to False that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeeds.
+The loki folders also serve as [example](./loki/kustomization.yaml) of how to use Kustomize to override selected parts of the `base`'s *.conf.
 
 ## Get in touch with us!
 

base/filter-kubernetes.conf

@@ -0,0 +1,11 @@
+
+[FILTER]
+    Name                kubernetes
+    Match               kube.*
+    Kube_Tag_Prefix     kube.
+    Regex_Parser        kube-tag
+    # Merge_Log           On
+    # Merge_Log_Key       log_processed
+    K8S-Logging.Parser  On
+    K8S-Logging.Exclude On
+    Annotations         Off

base/fluent-bit.conf

@@ -0,0 +1,13 @@
+[SERVICE]
+    Flush         1
+    Log_Level     ${LOG_LEVEL}
+    Daemon        off
+    Parsers_File  parser-cri.conf
+    Parsers_File  parsers.conf
+    HTTP_Server   On
+    HTTP_Listen   0.0.0.0
+    HTTP_Port     2020
+
+@INCLUDE input-kubernetes.conf
+@INCLUDE filter-kubernetes.conf
+@INCLUDE outputs.conf

base/fluentbit-daemonset.yaml

@@ -0,0 +1,73 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: fluentbit
+  labels:
+    app.kubernetes.io/name: fluentbit
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: fluentbit
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: fluentbit
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "2020"
+        prometheus.io/path: /api/v1/metrics/prometheus
+    spec:
+      serviceAccountName: fluentbit
+      terminationGracePeriodSeconds: 25
+      tolerations:
+      - key: node-role.kubernetes.io/master
+        operator: Exists
+        effect: NoSchedule
+      containers:
+      - name: fluentbit
+        image: docker.io/fluent/fluent-bit
+        command:
+        - /fluent-bit/bin/fluent-bit
+        - -c
+        - /fluent-bit/etc/fluent-bit.conf
+        env:
+        - name: LOG_LEVEL
+          value: debug
+        - name: MEM_BUF_LIMIT
+          value: 5MB
+        ports:
+        - name: http
+          containerPort: 2020
+        livenessProbe:
+          httpGet:
+            path: /
+            port: http
+        readinessProbe:
+          httpGet:
+            path: /api/v1/health
+            port: http
+        volumeMounts:
+        - name: varlogpods
+          mountPath: /var/log/pods
+          readOnly: true
+        - name: db
+          mountPath: /var/run/fluentbit
+        - name: config
+          mountPath: /fluent-bit/etc
+        # - mountPath: /etc/machine-id
+        #   name: etcmachineid
+        #   readOnly: true
+      volumes:
+      - name: varlogpods
+        hostPath:
+          path: /var/log/pods
+      - name: db
+        hostPath:
+          path: /var/run/fluentbit
+      # - name: etcmachineid
+      #   hostPath:
+      #     path: /etc/machine-id
+      #     type: File
+      - name: config
+        configMap:
+          name: fluentbit

base/input-kubernetes.conf

@@ -0,0 +1,11 @@
+[INPUT]
+    Name              tail
+    DB                /var/run/fluentbit/tail-positions.db
+    Mem_Buf_Limit     ${MEM_BUF_LIMIT}
+    Skip_Long_Lines   On
+    DB.locking        true
+    Path              /var/log/pods/*/*/*.log
+    Exclude_Path      /var/log/pods/*/fluentbit/*.log
+    Parser            cri
+    Tag               kube.<pod_id>.<namespace_name>.<pod_name>.<container_name>
+    Tag_Regex         /var/log/pods/(?<namespace_name>[^_/]+)_(?<pod_name>[^_/]+)_(?<pod_id>[^_/]+)/(?<container_name>[^/]+)/.*

base/kustomization.yaml

@@ -0,0 +1,25 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: logging
+
+images:
+- name: docker.io/fluent/fluent-bit
+  newTag: 1.8.3@sha256:10ea2709cef6e7059d980b4969d5f9d753ef97278a817c214cbe9120b1152082
+
+commonLabels:
+  app.kubernetes.io/name: fluentbit
+
+resources:
+- ../rbac
+- fluentbit-daemonset.yaml
+
+configMapGenerator:
+- name: fluentbit
+  files:
+  - fluent-bit.conf
+  - parser-cri.conf
+  - input-kubernetes.conf
+  - filter-kubernetes.conf
+  - parsers.conf
+  - outputs.conf

base/outputs.conf

@@ -0,0 +1,6 @@
+[OUTPUT]
+    Name             stdout
+    # Switch to format msgpack to see the tags
+    Format           json_lines
+    json_date_key    t
+    json_date_format iso8601

base/parser-cri.conf

@@ -0,0 +1,12 @@
+
+[PARSER]
+    Name        cri
+    Format      regex
+    Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
+    Time_Key    time
+    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
+
+[PARSER]
+    Name        kube-tag
+    Format      regex
+    Regex       ^(?<pod_id>[^.]+)\.(?<namespace_name>[^.]+)\.(?<pod_name>[^.]+)\.(?<container_name>[^.]+)$