This repository has been archived by the owner on Dec 29, 2020. It is now read-only.
Added capability for TCP output #52
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This update allows fw1-loggrabber to output CheckPoint logs over TCP.
I implemented this due to the number of issues I was having trying to process the file output, too many edge cases lead to eventual loss of log data, especially when using builtin log rotation, logrotated, or anything else. Even logrotated with pre/post scripts completely stopping and starting the service was becoming an issue. Due to the 32bit process limitation in only being able to write up to 2GB files I've had to resort to restarting services every 10 minutes* to prevent fw1 from crashing at my log ingest rate. With TCP output I no longer have to deal with any of these issues, also with this option there's much less IO involved which makes the SSD's happy.
I didn't implement UDP because when payloads are too large the data is sent with multiple packets (as expected). The issue is that so far every SIEM or log platform I've used can't seem to deal with single log events sent over multiple UDP packets and I always see those systems trying to parse truncated log events - so there's no point in using UDP for Checkpoint logs, which I know will always be very large.
The combination of this change and wrapping fw1 with a systemd service file to auto-recover from the occasional unexpected failure turns this into a decently resilient service and I hope others will find this useful. I'm also working on having the Makefile install a systemd service for fw1 automatically, I'll also create a pull request when I have that working.