This adds a proper documentation on how and when slips queries the APIS of urlhaus, spamhaus and cirl, in the threat_intelligence module.

docs/detection_modules.md

@@ -314,17 +314,48 @@ IncompatibleUserAgent, ICMP-Timestamp-Scan, ICMP-AddressScan, ICMP-AddressMaskSc
 ## Threat Intelligence Module
 
 Slips has a complex system to deal with Threat Intelligence feeds. 
-
 Slips supports different kinds of IoCs from TI feeds (IPs, IP ranges, domains, JA3 hashes, SSL hashes)
-
 File hashes and URLs aren't supported in TI feeds.
-
 Besides the searching 40+ TI files for every IP/domain Slips encounters, It also uses the following websites for threat intelligence:
 
-URLhaus: for each url seen in http.log and downloaded file seen in files.log
-Spamhaus: for IP lookups
-Circl.lu: for hash lookups (for each downloaded file)
+CIRCL.LU
+
+Slips looks up for (MD5) files hashes for downloaded files found in the files.log using the ```CIRCL.LU API.log``` obtained from Zeek. This lookup is handled by the ThreatIntel class's ```circl_lu function```.
+
+Slips creates the following URL for every file that is downloaded:```https://hashlookup.circl.lu/lookup/md5/<md5_hash>```. This URL is used to query the CIRCL.LU API with the file's MD5 hash.
+
+It parses the result after sending a GET request to this URL.
+
+Slips collects pertinent data, including confidence level, threat level, and blacklist sources, if the answer indicates that the file is known to be malicious.
+
+After that, it creates an evidence object and stores it in the database, indicating that a malicious file was downloaded, by calling the set_evidence_malicious_hash method.
+
+URLhaus
+
+Slips looks up file hashes (MD5) and URLs for malicious content using the URLhaus API. These lookups are handled by the URLhaus class.
+
+Slips constructs a URL to query the URLhaus API for URLs encountered in http.log or downloaded files found in files.log. It can do this by using the URL itself ```(https://urlhaus-api.abuse.ch/v1)``` or the MD5 hash.
+
+It sends the URL or MD5 hash as the payload of a POST request to the relevant URL.
+
+If the response indicates that the URL or hash is known to be malicious, Slips parses the response to extract pertinent information such as threat level, description, tags, and file details (if applicable).
+
+For malicious URLs, it calls the set_evidence_malicious_url function to create an evidence object and store it in the database, indicating that a malicious URL was accessed.
+
+For malicious file hashes, it calls the set_evidence_malicious_hash function to create an evidence object and store it in the database, indicating that a malicious file was downloaded.
+
+Spamhaus
 
+Slips checks if an IP address is listed as a known source of spam or malicious behavior using the Spamhaus DNS-based Blacklist (DNSBL).
+
+This lookup is handled by the spamhaus function of the ThreatIntel class. Slips creates a DNS query for every IP 
+address it encounters by reversing the address and appending .zen.spamhaus.org. For example, the query for IP 1.2.3.4 would be ```4.3.2.1.zen.spamhaus.org```. 
+
+Using the dns.resolver.resolve function from the dns Python library, it resolves the DNS for this query. A non-empty result from the resolution indicates that the IP address is listed on one or more Spamhaus blacklists. 
+
+Slips parses the response to determine which specific Spamhaus blacklists the IP is listed in and retrieves the corresponding descriptions and threat levels. 
+
+It then calls the set_evidence_malicious_ip function to create an evidence object and store it in the database, indicating that a malicious IP was encountered.
 
 ### Matching of IPs
 

modules/threat_intelligence/threat_intelligence.py

@@ -748,7 +748,20 @@ def is_outgoing_icmp_packet(self, protocol: str, ip_state: str) -> bool:
 
     def spamhaus(self, ip):
         """
-        Supports IP lookups only
+
+        Check if the given IP address is listed on the Spamhaus DNS-based Blacklist (DNSBL).
+
+        This function constructs a DNS query for the given IP address using the Spamhaus
+        DNSBL format, resolves the query using the `dns` library, and parses the response
+        to determine if the IP is listed on one or more Spamhaus blacklists.
+
+        Args:
+             ip (str): The IP address to check against the Spamhaus blacklists.
+
+        Returns:
+        [dict]: A dictionary containing the source dataset (list of blacklists),
+                description, threat level, and tags if the IP is listed on a Spamhaus
+                blacklist. If the IP is not listed or an error occurs, returns None.
         """
         # these are spamhaus datasets
         lists_names = {
@@ -902,7 +915,21 @@ def set_evidence_malicious_hash(self, file_info: Dict[str, any]):
 
     def circl_lu(self, flow_info: dict):
         """
-        Supports lookup of MD5 hashes on Circl.lu
+
+        Look up the MD5 hash of a downloaded file on the CIRCL.LU API.
+
+        This function constructs the URL for the CIRCL.LU API endpoint
+        based on the provided MD5 hash, sends a GET request to the API,
+        and processes the response to determine if the file is malicious.
+
+        Args:
+        flow_info (dict): A dictionary containing information about the file,
+        including the MD5 hash.
+
+        Returns:
+        [dict]: A dictionary containing the threat level, confidence,
+                and blacklist information if the file is found to be malicious.
+                If the file is not malicious or an error occurs, returns None.
         """
         def calculate_threat_level(circl_trust: str):
             """