Fluent bit

[shiv630]

Hi everyone,

I have a few questions about Fluent Bit's capabilities and configurations:

1. Can Fluent Bit perform file integrity monitoring and system inventory monitoring (e.g., running processes, installed programs, open ports, and connected network interfaces)?

2. What types of log schemas does Fluent Bit support? Can we define a customized log schema, and if so, how can this be achieved?

3. Can Fluent Bit compress logs before sending them to the destination? If so, how does this compression work, and how much system resources does it require?

4. If we have a log file containing thousands of different types and formats of logs, how can Fluent Bit separate and process them?

5. How much system resources does Fluent Bit require if we use it solely for log forwarding? Additionally, how does resource usage change if we also perform log parsing and enrichment, especially considering a scenario where each Fluent Bit node processes around 50,000 logs?

6. Is it feasible to use Fluent Bit as a single point of exit from the production environment? In this model, Fluent Bit would be deployed on all endpoints and send data to a centralized Fluent Bit instance, which would then forward logs to Apache NiFi. We are currently considering FluentD for this centralized log forwarding role.

7. Fluent Bit supports both in-memory and on-disk buffering. What is the maximum buffer size it supports, and can the buffer size be dynamic, especially for on-disk buffers?

8. How much log data can Fluent Bit handle in its buffer if it gets disconnected from the network? Under what conditions will it start dropping data?

9. What happens if the buffer storage becomes full?

10. Can we use both in-memory and on-disk buffer storage simultaneously?

Thanks in advance for your help!