fluent-bit kafka data pipeline is not working.

[bharathreddy901]

Hi Team,

We have configured the fluent-bit->Kafka data pipeline with the below mentioned link.

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/master/output/kafka

When we have deployed this pipeline, pods are coming up, Kafka topic was created, but logs are not flowing to Kafka cluster.
We could not see any error in the pod logs, we are seeing below info.
#########################################################

[root@centos]# kubectl logs fluent-bit-nb886 -n ec2-kafka
Fluent Bit v1.5.7

• Copyright (C) 2019-2020 The Fluent Bit Authors
• Copyright (C) 2015-2018 Treasure Data
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

[2022/09/12 07:11:40] [ info] [engine] started (pid=1)
[2022/09/12 07:11:40] [ info] [storage] version=1.0.5, initializing...
[2022/09/12 07:11:40] [ info] [storage] in-memory
[2022/09/12 07:11:40] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] API server connectivity OK
[2022/09/12 07:11:40] [ info] [output:kafka:kafka.0] brokers='x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092' topics='kafka-bit'
[2022/09/12 07:11:40] [ info] [sp] stream processor started
##########################################################################

Please let us know how to solve this issue.

Thanks in advance..!!

[bharathreddy901]

Hi Team - Please let me know if anyone resolved above problem.

[bharathreddy901]

Hi @edsiper and @hobti01,

Logs are not flowing from Fluent-bit to Kafka, But Kafka topic is getting created.
We are using below link to configure the pipeline. We have not changed anything, only thig we have updated Kafka broker name & topic name.

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/master/output/kafka

Thanks in advance.