Kubernetes filters are not working

[PradeepMurani]

Bug Report

Describe the bug
I have set up a platform to monitor the logs of our cluster(AWS EKS). I'm using the OpenSearch dashboard for monitoring and fluent-bit for log processing. I have deployed fluent bit (Helm) as a daemon set and configured to read the log files from the location /var/log/pods/*/*/*.log

I am also using the Kubernetes filter to extract Kubernetes metadata and append this information to each log.
However, application logs are floating to Open Search but it seems to be like filters are not appending the Kubernetes metadata on logs.

To Reproduce

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On

[INPUT]
    Name              tail
    Tag               kube.*  
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    DB.locking        true
    Path              /var/log/pods/*/*/*.log
    multiline.parser  docker, cri
         

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.var.log.pods.
    Merge_Log           On
    Merge_Log_Key       log_processed
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off


[OUTPUT]
    Name                opensearch
    Match               *
    Host                XXX
    Port                443
    TLS                 On
    AWS_Auth            On
    AWS_Region          eu-central-1
    Retry_Limit         6
    Retry_Limit         False
    Suppress_Type_Name  On
    Index               XXX
    Trace_Error         On
    Trace_Output        Off

Application Logs:

23-10-04 08:10:33 21         DEBUG                          Router route(): A cached Route with the cache identifier "91bdb3979cf77593d41f9b123bdcb33e" matched the request
23-10-04 08:10:33 21         DEBUG                          CSRF: No token required, safe request
23-10-04 08:10:33 633        DEBUG                          Session: Resumed session with id rSiY5V7SfDsu2yKCak4WIqxU241ALI93 which was inactive for 0 seconds. (0s)

Expected behavior

It should attach Kubernetes metadata to each log process by Kubernetes filter.

Open Search Logs (Screenshots)

Open Search filter options

Your Environment

• Environment name: Kubernetes
• Kubernetes Version: v1.27.4
• Fluent bit version: 2.1.9

I also checked with debug mode but did not find anything on the logs. Any help and guidance is highly appreciated.

[patrick-stephens]

What do the Fluent Bit logs show?
I would run with log_level debug and it'll show you the http requests and responses then.

Also use stdout output to check the actual tags are having the prefix you have defined, it must remove the prefix and just leave the filename (no further directories like your nested slashes seem to indicate).

Does it work with the standard /var/log/containers path?

I bet it'll be an issue either with kube_tag_prefix not matching the actual tag prefix you are getting or RBAC on the cluster not allowing FB to get any details.

[PradeepMurani]

Hey @patrick-stephens,

I raised it to debug and found some warning in Kubernetes filers as follow
[ warn] [filter:kubernetes:kubernetes.0] invalid pattern for given tag kube.var.log.pods.

It seems that the prefix is the same as defined in Kube_Tag_Prefix.

I'm not sure what exactly the cause of it.

Since I'm running with the latest version of Kubernetes, all the logs are now generated on /var/log/pod/* instead of /var/log/container

[patrick-stephens]

/var/log/containers is usually still there for legacy reasons as a symlink - is that entirely removed now? If it's not, just use that.

What is your actual tag? My guess it being a problem with kube_tag_prefix is correct then :) However, it would be good to know the general structure - I think it uses directories now instead of part of the filename for namespaces? My bet is that it cannot strip off the prefix and be left with just a filename made up of namespace+pod+container which is what it wants to query the K8S API with.

[PradeepMurani]

I gave the suggested solution from another thread #1535 a try, but it didn't work out for me. So, I ended by going with the PARSER implementation recommended in #4365 and everything is working as expected now.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.