Merge pull request #1055 from stratosphereips/develop

Slips v1.1.3
stratosphereips · Oct 31, 2024 · 6e6bc6f · 6e6bc6f
2 parents ab474d6 + e80c38a
commit 6e6bc6f
Show file tree

Hide file tree

Showing 83 changed files with 4,639 additions and 2,502 deletions.
diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml
@@ -45,7 +45,6 @@ jobs:
           - tests/test_asn_info.py
           - tests/test_urlhaus.py
           - tests/test_markov_chain.py
-          - tests/test_progress_bar.py
           - tests/test_daemon.py
           - tests/test_go_director.py
           - tests/test_notify.py
@@ -56,6 +55,15 @@ jobs:
           - tests/test_cesnet.py
           - tests/test_output.py
           - tests/test_riskiq.py
+          - tests/test_spamhaus.py
+          - tests/test_circllu.py
+          - tests/test_evidence_handler.py
+          - tests/test_alert_handler.py
+          - tests/test_redis_manager.py
+          - tests/test_ioc_handler.py
+          - tests/test_timeline.py
+          - tests/test_database.py
+          - tests/test_symbols_handler.py
 
     steps:
     - uses: actions/checkout@v4
@@ -89,10 +97,6 @@ jobs:
     - name: Start redis server
       run: redis-server --daemonize yes
 
-    - name: Run Database Unit Tests
-      run: |
-        python3 -m pytest tests/test_database.py -p no:warnings -vv
-
     - name: Run Unit Tests for ${{ matrix.test_file }}
       run: |
         python3 -m pytest ${{ matrix.test_file }} -p no:warnings -vv -s -n 5

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,17 @@
+- 1.1.3 (October 30th, 2024)
+- Enhanced Slips shutdown process for smoother operations.
+- Optimized resource management in Slips, resolving issues with lingering threads in memory.
+- Remove the progress bar; Slips now provides regular statistical updates.
+- Improve unit testing—special thanks to @Sekhar-Kumar-Dash.
+- Drop support for macOS, P2P, and platform-specific Docker images. A unified Docker image is now available for all platforms.
+- Correct the number of evidence reported in statistics.
+- Fix incorrect end date reported in metadata/info.txt upon analysis completion.
+- Print more information to CLI on Slips startup, including network details, client IP, thresholds used, and more.
+- Reduce false positives from Spamhaus by looking up inbound traffic only.
+- Speed up horizontal port scan detections.
+- Enhance logging of IDMEF errors.
+- Resolve issues with the accumulated threat level reported in alerts.json.
+
 - 1.1.2 (September 30th, 2024)
 - Add a relation between related evidence in alerts.json
 - Better unit tests. Thanks to @Sekhar-Kumar-Dash

diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 <h1 align="center">
-Slips v1.1.2
+Slips v1.1.3
 </h1>
 
 
@@ -33,7 +33,6 @@ Slips v1.1.2
 - [GUI](#graphical-user-interface)
 - [Requirements](#requirements)
 - [Installation](#installation)
-- [Extended Usage](#extended-usage)
 - [Configuration](#configuration)
 - [Features](#features)
 - [Contributing](#contributing)
@@ -60,7 +59,7 @@ Slips is a powerful endpoint behavioral intrusion prevention and detection syste
 Slips is the first free software behavioral machine learning-based IDS/IPS for endpoints. It was created in 2012 by Sebastian Garcia at the Stratosphere Laboratory, AIC, FEE, Czech Technical University in Prague. The goal was to offer a local IDS/IPS that leverages machine learning to detect network attacks using behavioral analysis.
 
 
-Slips is supported on Linux and MacOS only. The blocking features of Slips are only supported on Linux
+Slips is supported on Linux, MacOS, and windows dockers only. The blocking features of Slips are only supported on Linux
 
 Slips is Python-based and relies on [Zeek network analysis framework](https://zeek.org/get-zeek/) for capturing live traffic and analyzing PCAPs. and relies on
 Redis >= 7.0.4 for interprocess communication.
@@ -70,7 +69,7 @@ Redis >= 7.0.4 for interprocess communication.
 
 The recommended way to use Slips is on Docker.
 
-#### Linux
+#### Linux and Windows hosts
 ```
 docker run --rm -it -p 55000:55000  --cpu-shares "700" --memory="8g" --memory-swap="8g" --net=host --cap-add=NET_ADMIN --name slips stratosphereips/slips:latest
 ```
@@ -83,26 +82,11 @@ docker run --rm -it -p 55000:55000  --cpu-shares "700" --memory="8g" --memory-sw
 cat output_dir/alerts.log
 ```
 
-#### Macos M1
-In macos do not use --net=host if you want to access the internal container's ports from the host.
+#### Macos
+In MacOS, do not use --net=host if you want to access the internal container's ports from the host.
 
 ```
-docker run --rm -it -p 55000:55000  --cpu-shares "700" --memory="8g" --memory-swap="8g" --cap-add=NET_ADMIN --name slips stratosphereips/slips_macos_m1:latest
-```
-
-```
-./slips.py -f dataset/test7-malicious.pcap -o output_dir
-```
-
-```
-cat output_dir/alerts.log
-```
-
-
-#### Macos Intel
-
-```
-docker run --rm -it -p 55000:55000  --cpu-shares "700" --memory="8g" --memory-swap="8g" --net=host --cap-add=NET_ADMIN --name slips stratosphereips/slips:latest
+docker run --rm -it -p 55000:55000 --platform linux/amd64 --cpu-shares "700" --memory="8g" --memory-swap="8g" --cap-add=NET_ADMIN --name slips stratosphereips/slips_macos_m1:latest
 ```
 
 ```
@@ -160,7 +144,8 @@ Slips can be run on different platforms, the easiest and most recommended way if
 
 * [Docker](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#slips-in-docker)
   * Dockerhub (recommended)
-    * [Linux, MacOS and windows hosts](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#Running-Slips-from-DockerHub)
+    * [Linux and windows hosts](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#linux-and-windows-hosts)
+    * [MacOS hosts](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#macos-hosts)
   * [Docker-compose](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#running-slips-using-docker-compose)
   * [Dockerfile](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#building-slips-from-the-dockerfile)
 * Native
@@ -169,40 +154,8 @@ Slips can be run on different platforms, the easiest and most recommended way if
 * [on RPI (Beta)](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#installing-slips-on-a-raspberry-pi)
 
 
-
 ---
 
-# Extended Usage
-
-### Linux
-
-##### [Analyse your own traffic without P2P](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#analyse-your-own-traffic)
-
-
-##### [Analyse your own traffic with P2P ](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#for-p2p-support-on-linux)
-
-
-##### [Analyse a pcap without using P2P](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#analyze-your-pcap-file)
-
-
-
-### Macos M1
-
-#### [Analyse your own traffic without using P2P](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#id1)
-
-
-### MacOS Intel processors
-
-
-##### [Analyse your own traffic without using P2P](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#id2)
-
-
-##### [Analyse your own traffic with using P2P](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#for-p2p-support-on-macos-intel)
-
-
-##### [Analyse a PCAP without using P2P](https://stratospherelinuxips.readthedocs.io/en/develop/installation.html#id2)
-
-
 
 # Configuration
 Slips has a [config/slips.yaml](https://github.com/stratosphereips/StratosphereLinuxIPS/blob/develop/config/slips.yaml) that contains user configurations for different modules and general execution.

diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-1.1.2
+1.1.3
diff --git a/config/slips.yaml b/config/slips.yaml
@@ -54,8 +54,8 @@ parameters:
    # analysis_direction : all
    analysis_direction : out
 
-
    # Delete zeek log files after stopping slips.
+   # this parameter deletes arp.log every 1h. useful for saving disk space
    delete_zeek_files : False
 
    # Store a copy of zeek files in the output dir after the analysis is done.
@@ -140,7 +140,7 @@ parameters:
    # your own IP and is used to improve detections
    # it would be useful to specify it when analyzing pcaps or zeek logs
    # client_ips : [10.0.0.1, 172.16.0.9, 172.217.171.238]
-   client_ips : "[]"
+   client_ips : []
 
 #############################
 detection:
@@ -172,9 +172,9 @@ detection:
 modules:
    # List of modules to ignore. By default we always ignore the template! do not remove it from the list
    # Names of other modules that you can disable (they all should be lowercase with no special characters):
-   # threatintelligence, blocking,
-   # networkdiscovery, timeline, virustotal, rnnccdetection, flowmldetection, updatemanager
-   disable: "[template]"
+   # threatintelligence, blocking, networkdiscovery, timeline, virustotal,
+   # rnnccdetection, flowmldetection, updatemanager
+   disable: [template]
 
    # For each line in timeline file there is a timestamp.
    # By default the timestamp is seconds in unix time. However

diff --git a/docs/architecture.md b/docs/architecture.md
@@ -4,19 +4,19 @@ The architecture of Slips is basically:
     - To receive some data as input
     - To process it to a common format
     - To enrich it (gather all possible info about the IPs/MAC/User-Agents etc.)
-    - To apply detection modules 
+    - To apply detection modules
     - To output results
 
 Slips is heavily based on the Zeek monitoring tool as input tool for packets from the interface and pcap file, due to its excelent recognition of protocols and easiness to identify the content of the traffic.
 
-Figure 1 shows how the data is analyzed by Slips. 
+Figure 1 shows how the data is analyzed by Slips.
 As we can see, Slips internally uses <a href="https://zeek.org/">Zeek</a>, an
 open source network security monitoring tool. Slips divides flows into profiles and
 each profile into a timewindows.
-Slips runs detection modules on each flow and stores all evidence, 
-alerts and features in an appropriate profile structure. 
+Slips runs detection modules on each flow and stores all evidence,
+alerts and features in an appropriate profile structure.
 All profile info, performed detections, profiles and timewindows' data,
-is stored inside a <a href="https://redis.io/">Redis</a> database. 
+is stored inside a <a href="https://redis.io/">Redis</a> database.
 All flows are read, interpreted by Slips, labeled, and stored in the SQLite database in the output/ dir of each run
 The output of Slips is a folder with logs (output/ directory) that has alert.json, alerts.log, errors.log.
 Kalipso, a terminal graphical user interface. or the Web interface.
@@ -25,7 +25,7 @@ Kalipso, a terminal graphical user interface. or the Web interface.
 .zoom {
   transition: transform .2s; /* Animation */
   margin: 0;
-  position: relative; 
+  position: relative;
   z-index:999;
 }
 
@@ -42,7 +42,7 @@ Kalipso, a terminal graphical user interface. or the Web interface.
 
 
 Below is more explanation on internal representation of data, usage of Zeek and usage of Redis inside Slips.
-### Internal representation of data. 
+### Internal representation of data.
 
 Slips works at a flow level, instead of a packet level, gaining a high level view of behaviors. Slips creates traffic profiles for each IP that appears in the traffic. A profile contains the complete behavior of an IP address. Each profile is divided into time windows. Each time window is 1 hour long by default and contains dozens of features computed for all connections that start in that time window. Detections are done in each time window, allowing the profile to be marked as uninfected in the next time window.
 
@@ -56,20 +56,20 @@ This is what slips stores for each IP/Profile it creates:
 * Used software - list of software used by this profile, for example SSH, Browser, etc.
 * MAC and MAC Vendor - Ether MAC of the IP and the name of the vendor
 * Host-name - the name of the IP
-* first User-agent - First UA seen use dby this profile. 
+* first User-agent - First UA seen use dby this profile.
 * OS Type - Type of OS used by this profile as extracted from the user agent
 * OS Name - Name of OS used by this profile as extracted from the user agent
 * Browser - Name of the browser used by this profile as extracted from the user agent
-* User-agents history -  history of the all user agents used by this profile 
+* User-agents history -  history of the all user agents used by this profile
 * DHCP - if the IP is a dhcp or not
-* Starttime - epoch formatted timestamp of when the profile first appeared 
+* Starttime - epoch formatted timestamp of when the profile first appeared
 * Duration -  the standard duration of every TW in this profile
 * Modules labels - the labels assigned to this profile by each module
 * Gateway - if the IP is the gateway (router) of the network
-* Timewindow count -  Amount of timewindows in this profile 
+* Timewindow count -  Amount of timewindows in this profile
 * ASN - autonomous service number of the IP
 * Asnorg - name of the org that own the ASN of this IP
-* ASN Number 
+* ASN Number
 * SNI - Server name indicator
 * Reverse DNS - name of the IP in reverse dns
 * Threat Intelligence - If the IP appeared in any of Slips blacklist
@@ -85,32 +85,32 @@ This is what slips stores for each IP/Profile it creates:
   * Url ratio: The higher the score the more malicious this IP is
 
 
-### Alerts vs Evidence 
+### Alerts vs Evidence
 
 When running Slips, the alerts you see in red in the CLI or at the very bottom in kalispo, are a bunch of evidence. Evidence in slips are detections caused by a specific IP in a specific timeframe. Slips doesn't alert on every evidence/detection. it accumulates evidence and only generates and alert when the amount of gathered evidence crosses a threshold. After this threshold Slips generates an alert, marks the timewindow as malicious(displays it in red in kalipso) and blocks the IP causing the alert.
- 
-### Usage of Zeek. 
+
+### Usage of Zeek.
 
 Slips uses Zeek to generate files for most input types, and this data is used to create the profiles. For example, Slips uses this data to create a visual timeline of activities for each time window. This timeline consists of Zeek generated flows and additional interpretation from other logs like dns log and http log.
 
 
-### Usage of Redis database. 
+### Usage of Redis database.
 
 All the data inside Slips is stored in Redis, an in-memory data structure.
 Redis allows all the modules in Slips to access the data in parallel.
 Apart from read and write operations, Slips takes advantage of the Redis messaging system called Redis PUB/SUB.
-Processes may publish data into the channels, while others subscribe to these channels and process the new data when it is published. 
+Processes may publish data into the channels, while others subscribe to these channels and process the new data when it is published.
 
-### Usage of SQLite database. 
+### Usage of SQLite database.
 
 Slips uses SQLite database to store all flows in Slips interpreted format.
 The SQLite database is stored in the output/ dir and each flow is labeled to either 'malicious' or 'benign' based on slips detections.
 all the labeled flows in the SQLite database can be exported to tsv or json format.
 
 
-### Threat Levels  
+### Threat Levels
 
-Slips has 4 threat levels.
+Slips has 5 threat levels.
 
 <style type="text/css">
 .tg  {border-collapse:collapse;border-spacing:0;}
@@ -157,6 +157,17 @@ Slips has 4 threat levels.
   </tr>
 
 
+### How Slips Stops
+
+- When slips is running on an interface or a growing zeek directory, slips keeps running forever until the user presses ctrl+c
+- When Slips is analyzing a PCAP or a zeek directory or any other supported file, It keeps running until no more flows are received.
+- After the modules receive that signal that says "no more new flows are coming", all modules keep processing the existing flows normally until they run out of msgs and stop.
+- Modules stop only if no more msgs are received in their Redis channels, and if they receive the signal that slips is no longer receiving new flows.
+- Slips knows that no more flows are arriving when it reaches the end of the given zeek/suricata/nfdump logs.
+- If some processes are hanging in memory, slips wait by default 1 week before killing them. This can be modified in the config.yaml.
+
+For more techincal details about this check https://stratospherelinuxips.readthedocs.io/en/develop/contributing.html#faq
+
 
 </tbody>
-</table>
+</table>
diff --git a/docs/code_documentation.md b/docs/code_documentation.md
@@ -1,41 +1,3 @@
 # Code documentation
 
-### How Slips Works
-
-<img src="https://raw.githubusercontent.com/stratosphereips/StratosphereLinuxIPS/develop/docs/images/slips_workflow.png" title="Slips Workflow">
-
-1. slips.py is the entry point, it's responsible for starting all modules, and keeping slips up until the analysis is finished.
-2. slips.py starts the input process, which is the one responsible for reading the flows from the files given to slips using -f
-it detects the type of file, reads it and passes the flows to the profiler process. if slips was given a PCAP or is running on an interface
-, the input process starts a zeek thread that analyzes the pcap/interface using slips' own zeek configuration and sends the generated zeek
-flows to the profiler process.
-3. slips.py also starts the update manager, it updates slips local TI files, like the ones stored in slips_files/organizations_info and slips_files/ports_info.
-later, when slips is starting all the modules, slips also starts the update manager but to update remote TI files in the background in this case.
-4. Once the profiler process receives the flows read by the input process, it starts to convert them to a structure that slips can deal with.
-it creates profiles and time windows for each IP it encounters.
-5. Profiler process gives each flow to the appropriate module to deal with it. for example flows from http.log will be sent to http_analyzer.py
-to analyze them.
-6. Profiler process stores the flows, profiles, etc. in slips databases for later processing. the info stored in the dbs will be used by all modules later.
-Slips has 2 databases, Redis and SQLite. it uses the sqlite db to store all the flows read and labeled. and uses redis for all other operations. the sqlite db is
-created in the output directory, meanwhite the redis database is in-memory.
-7-8. using the flows stored in the db in step 6 and with the help of the timeline module, slips puts the given flows in a human-readable form which is
-then used by the web UI and kalipso UI.
-9. when a module finds a detection, it sends the detection to the evidence process to deal with it (step 10) but first, this evidence is checked by the whitelist to see if it's
-whitelisted in our config/whitelist.conf or not. if the evidence is whitelisted, it will be discarded and won't go through the next steps
-10. now that we're sure that the evidence isn't whitelisted, the evidence process logs it to slips log files and gives the evidence to all modules responsible for exporting
-evidence. so, if CEST, Exporting modules, or CYST is enabled, the evidence process notifies them
-through redis channels that it found an evidence and it's time to share the evidence.
-11. if the blocking module is enabled using -p, the evidence process shares all detected alerts to the blocking module. and the blocking module handles
-the blocking of the attacker IP through the linux firewall (supported in linux only)
-12. if p2p is enabled in config/slips.yaml, the p2p module shares the IP of the attacker, its' score and blocking requests sent by the evidence process
-with other peers in the network so they can block the attackers before they reach them.
-13. The output process is slips custom logging framework. all alerts, warnings and info printed are sent here first for proper formatting and printing.
-
-This is a brief explanation of how slips works for new contributors.
-
-All modules described above are talked about in more detail in the rest of the documentation.
-
-
-### Code Docs
-
-[Slips auto-generated code documentation here](https://stratosphereips.github.io/StratosphereLinuxIPS/files.html)
+[Slips auto-generated code documentation](https://stratosphereips.github.io/StratosphereLinuxIPS/files.html)