From 019aaa9f6705c788f0f577ba1f8af0ccbb1df70a Mon Sep 17 00:00:00 2001 From: Sekhar-Kumar-Dash <119131588+Sekhar-Kumar-Dash@users.noreply.github.com> Date: Tue, 26 Mar 2024 21:25:13 +0530 Subject: [PATCH 1/5] Updated detection_modules.md --- docs/detection_modules.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/detection_modules.md b/docs/detection_modules.md index a24aaff6e..2979922fa 100644 --- a/docs/detection_modules.md +++ b/docs/detection_modules.md @@ -314,16 +314,17 @@ IncompatibleUserAgent, ICMP-Timestamp-Scan, ICMP-AddressScan, ICMP-AddressMaskSc ## Threat Intelligence Module Slips has a complex system to deal with Threat Intelligence feeds. - Slips supports different kinds of IoCs from TI feeds (IPs, IP ranges, domains, JA3 hashes, SSL hashes) - File hashes and URLs aren't supported in TI feeds. - Besides the searching 40+ TI files for every IP/domain Slips encounters, It also uses the following websites for threat intelligence: -URLhaus: for each url seen in http.log and downloaded file seen in files.log -Spamhaus: for IP lookups -Circl.lu: for hash lookups (for each downloaded file) +CIRCL.LU +Slips looks up file hashes (MD5) for downloaded files found in the files using the CIRCL.LU API.log obtained from Zeek. This lookup is handled by the ThreatIntel class's circl_lu function. + +Slips creates the following URL for every file that is downloaded: https://hashlookup.circl.lu/lookup/md5/. This URL is used to query the CIRCL.LU API with the file's MD5 hash. +It parses the result after sending a GET request to this URL. +Slips collects pertinent data, including confidence level, threat level, and blacklist sources, if the answer indicates that the file is known to be malicious. +After that, it creates an evidence object and stores it in the database, indicating that a malicious file was downloaded, by calling the set_evidence_malicious_hash method. ### Matching of IPs From ccbc13ffa8d490e8beb87abf72bdddb29d3b11c9 Mon Sep 17 00:00:00 2001 From: Sekhar-Kumar-Dash <119131588+Sekhar-Kumar-Dash@users.noreply.github.com> Date: Wed, 27 Mar 2024 00:35:50 +0530 Subject: [PATCH 2/5] Update detection_modules.md --- docs/detection_modules.md | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/docs/detection_modules.md b/docs/detection_modules.md index 2979922fa..8484195d4 100644 --- a/docs/detection_modules.md +++ b/docs/detection_modules.md @@ -319,13 +319,43 @@ File hashes and URLs aren't supported in TI feeds. Besides the searching 40+ TI files for every IP/domain Slips encounters, It also uses the following websites for threat intelligence: CIRCL.LU -Slips looks up file hashes (MD5) for downloaded files found in the files using the CIRCL.LU API.log obtained from Zeek. This lookup is handled by the ThreatIntel class's circl_lu function. -Slips creates the following URL for every file that is downloaded: https://hashlookup.circl.lu/lookup/md5/. This URL is used to query the CIRCL.LU API with the file's MD5 hash. +Slips looks up for (MD5) files hashes for downloaded files found in the files.log using the ```CIRCL.LU API.log``` obtained from Zeek. This lookup is handled by the ThreatIntel class's ```circl_lu function```. + +Slips creates the following URL for every file that is downloaded:```https://hashlookup.circl.lu/lookup/md5/```. This URL is used to query the CIRCL.LU API with the file's MD5 hash. + It parses the result after sending a GET request to this URL. + Slips collects pertinent data, including confidence level, threat level, and blacklist sources, if the answer indicates that the file is known to be malicious. + After that, it creates an evidence object and stores it in the database, indicating that a malicious file was downloaded, by calling the set_evidence_malicious_hash method. +URLhaus + +Slips looks up file hashes (MD5) and URLs for malicious content using the URLhaus API. These lookups are handled by the URLhaus class. + +Slips constructs a URL to query the URLhaus API for URLs encountered in http.log or downloaded files found in files.log. It can do this by using the URL itself ```(https://urlhaus-api.abuse.ch/v1)``` or the MD5 hash. + +It sends the URL or MD5 hash as the payload of a POST request to the relevant URL. + +If the response indicates that the URL or hash is known to be malicious, Slips parses the response to extract pertinent information such as threat level, description, tags, and file details (if applicable). + +For malicious URLs, it calls the set_evidence_malicious_url function to create an evidence object and store it in the database, indicating that a malicious URL was accessed. + +For malicious file hashes, it calls the set_evidence_malicious_hash function to create an evidence object and store it in the database, indicating that a malicious file was downloaded. + +Spamhaus + +Slips checks if an IP address is listed as a known source of spam or malicious behavior using the Spamhaus DNS-based Blacklist (DNSBL). + +This lookup is handled by the spamhaus function of the ThreatIntel class. Slips creates a DNS query for every IP +address it encounters by reversing the address and appending .zen.spamhaus.org. For example, the query for IP 1.2.3.4 would be ```4.3.2.1.zen.spamhaus.org```. + +Using the dns.resolver.resolve function from the dns Python library, it resolves the DNS for this query. A non-empty result from the resolution indicates that the IP address is listed on one or more Spamhaus blacklists. + +Slips parses the response to determine which specific Spamhaus blacklists the IP is listed in and retrieves the corresponding descriptions and threat levels. + +It then calls the set_evidence_malicious_ip function to create an evidence object and store it in the database, indicating that a malicious IP was encountered. ### Matching of IPs From 73166cece4e628a23e783609ced21166baf0db7e Mon Sep 17 00:00:00 2001 From: Sekhar-Kumar-Dash <119131588+Sekhar-Kumar-Dash@users.noreply.github.com> Date: Wed, 27 Mar 2024 01:05:58 +0530 Subject: [PATCH 3/5] added detailed description to spamhaus and circl_lu function --- .../threat_intelligence.py | 31 +++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/modules/threat_intelligence/threat_intelligence.py b/modules/threat_intelligence/threat_intelligence.py index e53921275..f1b02583b 100644 --- a/modules/threat_intelligence/threat_intelligence.py +++ b/modules/threat_intelligence/threat_intelligence.py @@ -748,7 +748,20 @@ def is_outgoing_icmp_packet(self, protocol: str, ip_state: str) -> bool: def spamhaus(self, ip): """ - Supports IP lookups only + + Check if the given IP address is listed on the Spamhaus DNS-based Blacklist (DNSBL). + + This function constructs a DNS query for the given IP address using the Spamhaus + DNSBL format, resolves the query using the `dns` library, and parses the response + to determine if the IP is listed on one or more Spamhaus blacklists. + + Args: + ip (str): The IP address to check against the Spamhaus blacklists. + + Returns: + [dict]: A dictionary containing the source dataset (list of blacklists), + description, threat level, and tags if the IP is listed on a Spamhaus + blacklist. If the IP is not listed or an error occurs, returns None. """ # these are spamhaus datasets lists_names = { @@ -902,7 +915,21 @@ def set_evidence_malicious_hash(self, file_info: Dict[str, any]): def circl_lu(self, flow_info: dict): """ - Supports lookup of MD5 hashes on Circl.lu + + Look up the MD5 hash of a downloaded file on the CIRCL.LU API. + + This function constructs the URL for the CIRCL.LU API endpoint + based on the provided MD5 hash, sends a GET request to the API, + and processes the response to determine if the file is malicious. + + Args: + flow_info (dict): A dictionary containing information about the file, + including the MD5 hash. + + Returns: + [dict]: A dictionary containing the threat level, confidence, + and blacklist information if the file is found to be malicious. + If the file is not malicious or an error occurs, returns None. """ def calculate_threat_level(circl_trust: str): """ From 080b1931a2238f4bd53661b62ff8cb85298bdc67 Mon Sep 17 00:00:00 2001 From: Sekhar-Kumar-Dash <119131588+Sekhar-Kumar-Dash@users.noreply.github.com> Date: Thu, 28 Mar 2024 21:24:09 +0530 Subject: [PATCH 4/5] Update detection_modules.md --- docs/detection_modules.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/detection_modules.md b/docs/detection_modules.md index 8484195d4..7ca833c73 100644 --- a/docs/detection_modules.md +++ b/docs/detection_modules.md @@ -318,9 +318,9 @@ Slips supports different kinds of IoCs from TI feeds (IPs, IP ranges, domains, J File hashes and URLs aren't supported in TI feeds. Besides the searching 40+ TI files for every IP/domain Slips encounters, It also uses the following websites for threat intelligence: -CIRCL.LU +## CIRCL.LU -Slips looks up for (MD5) files hashes for downloaded files found in the files.log using the ```CIRCL.LU API.log``` obtained from Zeek. This lookup is handled by the ThreatIntel class's ```circl_lu function```. +Slips looks up for (MD5) files hashes for downloaded files found in the files.log ```CIRCL.LU API``` . This lookup is handled by the ThreatIntel class's ```circl_lu function```. Slips creates the following URL for every file that is downloaded:```https://hashlookup.circl.lu/lookup/md5/```. This URL is used to query the CIRCL.LU API with the file's MD5 hash. @@ -330,7 +330,7 @@ Slips collects pertinent data, including confidence level, threat level, and bla After that, it creates an evidence object and stores it in the database, indicating that a malicious file was downloaded, by calling the set_evidence_malicious_hash method. -URLhaus +##URLhaus Slips looks up file hashes (MD5) and URLs for malicious content using the URLhaus API. These lookups are handled by the URLhaus class. @@ -344,7 +344,7 @@ For malicious URLs, it calls the set_evidence_malicious_url function to create a For malicious file hashes, it calls the set_evidence_malicious_hash function to create an evidence object and store it in the database, indicating that a malicious file was downloaded. -Spamhaus +## Spamhaus Slips checks if an IP address is listed as a known source of spam or malicious behavior using the Spamhaus DNS-based Blacklist (DNSBL). From 18c5d6117827481d8ca6aa4a10f3ad47efff6c5a Mon Sep 17 00:00:00 2001 From: Sekhar-Kumar-Dash <119131588+Sekhar-Kumar-Dash@users.noreply.github.com> Date: Thu, 28 Mar 2024 21:28:22 +0530 Subject: [PATCH 5/5] Update detection_modules.md --- docs/detection_modules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detection_modules.md b/docs/detection_modules.md index 7ca833c73..26a4cfe9d 100644 --- a/docs/detection_modules.md +++ b/docs/detection_modules.md @@ -330,7 +330,7 @@ Slips collects pertinent data, including confidence level, threat level, and bla After that, it creates an evidence object and stores it in the database, indicating that a malicious file was downloaded, by calling the set_evidence_malicious_hash method. -##URLhaus +## URLhaus Slips looks up file hashes (MD5) and URLs for malicious content using the URLhaus API. These lookups are handled by the URLhaus class.