Slips pcap analysis needs hours for bigger test files

[maldwg]

Describe the bug
When analyzing a small pcap file using Slips (e.g. one that is already shipped in the image or repository) everything works fine though it might take a few minutes to finish. However, when using larger pcap files (in my tests ~800 MB ), slips is working terribly slow. Given that such ammount of data are not unusual in external datasets, Slips should be able to finish them in a few minutes and not over 12 hours. The container running has sufficient ressources and the system has even spare ones when running the analysis. From what I can tell, the Profiler Process is taking a big ammount of the time.

To Reproduce
Steps to reproduce the behavior:

1. use a current version of Slips
2. run a static analysis using a pcap that is ~800 MB or bigger
3. look at the slips.log

slips.log

Here are the logs for analyzing a 800 MB file:

2024/09/12 06:36:12.643701 [Main] Using redis server on port: 6379
2024/09/12 06:36:12.643929 [Main] Started Main process [PID 299]
2024/09/12 06:36:12.647809 [Main] Started PBar process [PID 309]
2024/09/12 06:36:12.852498 [Main] Starting modules
2024/09/12 06:36:15.637477 [Main]               Starting the module ARP (Detect ARP attacks) [PID 327]
2024/09/12 06:36:15.976603 [Main]               Starting the module Flow Alerts (Alerts about flows: long connection, successful ssh, password guessing, self-signed certificate, data exfiltration, etc.) [PID 329]
2024/09/12 06:36:16.025707 [Main]               Starting the module HTTP Analyzer (Analyze HTTP flows) [PID 331]
2024/09/12 06:36:16.033399 [Main]               Starting the module Network Discovery (Detect Horizonal, Vertical Port scans, ICMP, and DHCP scans) [PID 332]
2024/09/12 06:36:16.080859 [Main]               Starting the module RNN C&C Detection (Detect C&C channels based on behavioral letters) [PID 333]
2024/09/12 06:36:16.132170 [Main]               Starting the module Threat Intelligence (Check if the source IP or destination IP are in a malicious list of IPs) [PID 351]
2024/09/12 06:36:16.632745 [Main] Disabled Modules: ['template', 'ensembling', 'updatemanager', 'ipinfo', 'flowmldetection', 'virustotal', 'cesnet', 'timeline', 'riskiq', 'exporting_alerts', 'p2ptrust', 'cesnet', 'blocking', 'leak_detector', 'cyst']
2024/09/12 06:36:16.718750 [Evidence] Storing Slips logs in /opt/logs
2024/09/12 06:36:16.723661 [Main] Started Evidence Process [PID 353]
2024/09/12 06:36:16.808540 [Main] Started Profiler Process [PID 354]
2024/09/12 06:36:16.812206 [Main] Metadata added to /opt/logs/metadata
2024/09/12 06:36:16.857231 [Main] Started Input Process [PID 355]
2024/09/12 06:36:16.862085 [Main] Warning: Slips may generate a large amount of traffic by querying TI sites.
2024/09/12 06:36:16.862547 [Input] Storing zeek log files in /opt/logs/zeek_files
2024/09/12 06:40:08.120583 [Input] Zeek error. return code: 0 error:b'warning in /opt/zeek/share/zeek/policy/tuning/defaults/__load__.zeek, line 1: deprecated script loaded from /StratosphereLinuxIPS/zeek-scripts/__load__.zeek:25 "Remove in v7.1 The policy/tuning/defaults package is deprecated. The options set here are now the defaults for Zeek in general.";'
2024/09/12 06:53:47.662258 [Input] We read everything. No more input. Stopping input process. Sent 1009554 lines
2024/09/12 06:53:47.662958 [Input] Telling Profiler to stop because no more input is arriving.
2024/09/12 06:53:47.663094 [Input] Waiting for Profiler to stop.
2024/09/12 19:53:41.135456 [Main] 
---------------------------
2024/09/12 19:53:41.135750 [Main] Stopping Slips
2024/09/12 19:53:41.175734 [Main] Analysis of /tmp/dataset.pcap finished in 797.48 minutes
2024/09/12 19:53:41.189171 [Main] Total flows read (without altflows): 333592
2024/09/12 19:53:41.215257 [Profiler] Stopping. Total lines read: 880876
2024/09/12 19:53:44.194729 [Main]       ARP                     Stopped. 9 left.
2024/09/12 19:53:44.194944 [Main]       RNN C&C Detection       Stopped. 8 left.
2024/09/12 19:53:47.198404 [Main]       Threat Intelligence     Stopped. 7 left.
2024/09/12 19:53:47.198723 [Main]       HTTP Analyzer           Stopped. 6 left.
2024/09/12 19:53:50.202190 [Main]       Progress Bar            Stopped. 5 left.
2024/09/12 19:53:50.202459 [Main]       Profiler                Stopped. 4 left.
2024/09/12 19:53:50.202572 [Main] The following modules are busy working on your data.

['Flow Alerts', 'Input', 'Network Discovery', 'Evidence']

You can wait for them to finish, or you can press CTRL-C again to force-kill.

As you can see, I stopped slips after 13 hours into the analysis, as the Profiler Process either hangs up or needs even more time. I already set the wait_for_modules_to_finish value in the slips.yaml to 1200 mins.

Expected behavior
Slips should be able to analyze such files within minutes not days.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: Ubuntu 22.04
• Version [e.g. 22]
• Python version: 3.8.10 (Docker container)
• Are you running slips in docker or locally? Docker
• Docker version (if running slips in docker) Docker version 24.0.5
• Slips docker image used (if running slips in docker) slips:latest (v 1.1)

[AlyaGomaa]

hey @maldwg thanks for reporting!
is the pcap private or can you share it with us to better understand what's going on?

[maldwg]

Hi @AlyaGomaa , thank you for the quick reply. The pcap is not private, it is a file from the CIC-IDS-2017 dataset https://www.unb.ca/cic/datasets/ids-2017.html . In fact it is the pcap for the thursday (the dataset consists of 5 pcaps for each day from monday to friday). Originally this file is ~8GB and it will produce the same error, but I reduced it to 800 MB by filtering out packets. At the bottom of the linked page you can download the dataset if you want. I tryed uploading a zipped version of the 800MB file here but it is still too big... So I uploaded it also here, for you to download:
https://www.file-upload.net/download-15384477/pcap_file.zip.html

Thank you for having a look into it :)