Elgato_dark/bunkerweb
1
0
Fork 0
mirror of https://github.com/bunkerity/bunkerweb synced 2026-05-24 09:28:37 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: Add nightly build of the OWASP coreruleset that are automatically downloaded and updated

Browse source
This commit is contained in:
Théophile Diot 2024-05-15 18:03:34 +02:00
parent 7216c770c8
commit 3d76e10e8a
No known key found for this signature in database
GPG key ID: 248FEA4BAE400D06
8 changed files with 202 additions and 28 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
CHANGELOG.md
View file

@ -1,5 +1,9 @@
# Changelog
## v1.5.8 - ????/??/??
- [FEATURE] Add nightly build of the OWASP coreruleset that are automatically downloaded and updated
## v1.5.7 - 2024/05/14
- [LINUX] Support Ubuntu 24.04 (Noble)

6
docs/security-tuning.md
View file

@ -241,7 +241,7 @@ ModSecurity is integrated and enabled by default alongside the OWASP Core Rule S
| :-----------------------: | :-----: | :---------------------------------------------------------------------------------------------------- |
| `USE_MODSECURITY` | `yes` | When set to `yes`, ModSecurity will be enabled. |
| `USE_MODSECURITY_CRS` | `yes` | When set to `yes` and `USE_MODSECURITY` is also set to `yes`, the OWASP Core Rule Set will be loaded. |
| `MODSECURITY_CRS_VERSION` | `3` | Version of the OWASP Core Rule Set to use. |
| `MODSECURITY_CRS_VERSION` | `3` | Version of the OWASP Core Rule Set to use with ModSecurity (3, 4 or nightly). |
!!! warning "ModSecurity and the OWASP Core Rule Set"
**We strongly recommend keeping both ModSecurity and the OWASP Core Rule Set enabled**. The only downsides are the false positives that may occur. But they can be fixed with some efforts and the CRS team maintains a list of exclusions for common applications (e.g., WordPress, Nextcloud, Drupal, Cpanel, ...).
@ -250,6 +250,10 @@ You can choose between the following versions of the OWASP Core Rule Set :
- **3** : The version [v3.3.5](https://github.com/coreruleset/coreruleset/releases/tag/v3.3.5) of the OWASP Core Rule Set (***default***)
- **4** : The version [v4.2.0](https://github.com/coreruleset/coreruleset/releases/tag/v4.2.0) of the OWASP Core Rule Set
- **nightly** : The latest [nightly](https://github.com/coreruleset/coreruleset/releases/tag/nightly) build of the OWASP Core Rule Set which is updated every day
!!! example "OWASP Core Rule Set's nightly build"
The nightly build of the OWASP Core Rule Set is updated every day and contains the latest rules. It is recommended to use it in a staging environment before using it in production.
### Custom configurations

36
docs/settings.md
View file

@ -368,16 +368,16 @@ STREAM support :white_check_mark:
Automatic creation, renewal and configuration of Let's Encrypt certificates using DNS challenges.
| Setting | Default | Context |Multiple| Description |
|----------------------------------|---------|---------|--------|---------------------------------------------------------------------------------------|
|`AUTO_LETS_ENCRYPT_DNS` |`no` |multisite|no |Activate automatic Let's Encrypt DNS. |
|`LETS_ENCRYPT_DNS_EMAIL` | |multisite|no |The email address to use for Let's Encrypt notifications. |
|`USE_LETS_ENCRYPT_DNS_STAGING` |`no` |multisite|no |Use the Let's Encrypt staging environment. |
|`LETS_ENCRYPT_DNS_PROVIDER` | |multisite|no |The DNS provider to use for DNS challenges. |
|`USE_LETS_ENCRYPT_DNS_WILDCARD` |`yes` |multisite|no |Create wildcard certificates for all domains using DNS challenges. |
|`LETS_ENCRYPT_DNS_PROPAGATION` |`default`|multisite|no |The time to wait for DNS propagation in seconds. |
|`LETS_ENCRYPT_DNS_CREDENTIAL_ITEM`| |multisite|yes |Configuration item that will be added to the credentials.ini file for the DNS provider.|
|`LETS_ENCRYPT_DNS_CLEAR_OLD_CERTS`|`no` |global |no |Clear old certificates when renewing. |
| Setting | Default | Context |Multiple| Description |
|----------------------------------|---------|---------|--------|----------------------------------------------------------------------------------------------------------------------------|
|`AUTO_LETS_ENCRYPT_DNS` |`no` |multisite|no |Activate automatic Let's Encrypt DNS. |
|`LETS_ENCRYPT_DNS_EMAIL` | |multisite|no |The email address to use for Let's Encrypt notifications. |
|`USE_LETS_ENCRYPT_DNS_STAGING` |`no` |multisite|no |Use the Let's Encrypt staging environment. |
|`LETS_ENCRYPT_DNS_PROVIDER` | |multisite|no |The DNS provider to use for DNS challenges. |
|`USE_LETS_ENCRYPT_DNS_WILDCARD` |`yes` |multisite|no |Create wildcard certificates for all domains using DNS challenges. |
|`LETS_ENCRYPT_DNS_PROPAGATION` |`default`|multisite|no |The time to wait for DNS propagation in seconds. |
|`LETS_ENCRYPT_DNS_CREDENTIAL_ITEM`| |multisite|yes |Configuration item that will be added to the credentials.ini file for the DNS provider (e.g. 'cloudflare_api_token 123456').|
|`LETS_ENCRYPT_DNS_CLEAR_OLD_CERTS`|`no` |global |no |Clear old certificates when renewing. |
## Limit
@ -448,14 +448,14 @@ STREAM support :x:
Management of the ModSecurity WAF.
| Setting | Default | Context |Multiple| Description |
|---------------------------------|--------------|---------|--------|------------------------------------------|
|`USE_MODSECURITY` |`yes` |multisite|no |Enable ModSecurity WAF. |
|`USE_MODSECURITY_CRS` |`yes` |multisite|no |Enable OWASP Core Rule Set. |
|`MODSECURITY_CRS_VERSION` |`3` |multisite|no |Version of the OWASP Core Rule Set to use.|
|`MODSECURITY_SEC_AUDIT_ENGINE` |`RelevantOnly`|multisite|no |SecAuditEngine directive of ModSecurity. |
|`MODSECURITY_SEC_RULE_ENGINE` |`On` |multisite|no |SecRuleEngine directive of ModSecurity. |
|`MODSECURITY_SEC_AUDIT_LOG_PARTS`|`ABCFHZ` |multisite|no |SecAuditLogParts directive of ModSecurity.|
| Setting | Default | Context |Multiple| Description |
|---------------------------------|--------------|---------|--------|-----------------------------------------------------------------------------|
|`USE_MODSECURITY` |`yes` |multisite|no |Enable ModSecurity WAF. |
|`USE_MODSECURITY_CRS` |`yes` |multisite|no |Enable OWASP Core Rule Set. |
|`MODSECURITY_CRS_VERSION` |`3` |multisite|no |Version of the OWASP Core Rule Set to use with ModSecurity (3, 4 or nightly).|
|`MODSECURITY_SEC_AUDIT_ENGINE` |`RelevantOnly`|multisite|no |SecAuditEngine directive of ModSecurity. |
|`MODSECURITY_SEC_RULE_ENGINE` |`On` |multisite|no |SecRuleEngine directive of ModSecurity. |
|`MODSECURITY_SEC_AUDIT_LOG_PARTS`|`ABCFHZ` |multisite|no |SecAuditLogParts directive of ModSecurity. |
## Monitoring <img src='../assets/img/pro-icon.svg' alt='crow pro icon' height='24px' width='24px' style='transform : translateY(3px);'> (PRO)

19
src/common/core/modsecurity/confs/server-http/modsecurity-rules.conf.modsec
View file

@ -1,3 +1,4 @@
{% set os_path = import("os.path") %}
# process rules with disruptive actions
SecRuleEngine {{ MODSECURITY_SEC_RULE_ENGINE }}
@ -66,7 +67,16 @@ SecAuditLog /var/log/bunkerweb/modsec_audit.log
# include OWASP CRS configurations
{% if USE_MODSECURITY_CRS == "yes" %}
{% if MODSECURITY_CRS_VERSION == "nightly" %}
{% if os_path.isfile("/var/cache/bunkerweb/modsecurity/crs/crs-setup-nightly.conf") %}
include /var/cache/bunkerweb/modsecurity/crs/crs-setup-nightly.conf
{% else %}
# fallback to the default CRS setup as the nightly one is not available
include /usr/share/bunkerweb/core/modsecurity/files/crs-setup-v3.conf
{% endif %}
{% else %}
include /usr/share/bunkerweb/core/modsecurity/files/crs-setup-v{{ MODSECURITY_CRS_VERSION }}.conf
{% endif %}
# custom CRS configurations before loading rules (e.g. exclusions)
{% if is_custom_conf("/etc/bunkerweb/configs/modsec-crs") %}
@ -100,7 +110,16 @@ SecRule ENV:is_whitelisted "yes" "id:1000,phase:1,allow,nolog,ctl:ruleEngine=Off
{% endif +%}
# include OWASP CRS rules
{% if MODSECURITY_CRS_VERSION == "nightly" %}
{% if os_path.exists("/var/cache/bunkerweb/modsecurity/crs/crs-nightly/rules") %}
include /var/cache/bunkerweb/modsecurity/crs/crs-nightly/rules/*.conf
{% else %}
# fallback to the default CRS setup as the nightly one is not available
include /usr/share/bunkerweb/core/modsecurity/files/coreruleset-v3/rules/*.conf
{% endif %}
{% else %}
include /usr/share/bunkerweb/core/modsecurity/files/coreruleset-v{{ MODSECURITY_CRS_VERSION }}/rules/*.conf
{% endif %}
{% endif +%}
# custom rules after loading the CRS

127
src/common/core/modsecurity/jobs/coreruleset-nightly.py Normal file
View file

@ -0,0 +1,127 @@
#!/usr/bin/env python3
from io import BytesIO
from os import getenv, sep
from os.path import join
from pathlib import Path
from re import MULTILINE, search
from shutil import rmtree
from sys import exit as sys_exit, path as sys_path
from tarfile import open as tar_open
from threading import Lock
from traceback import format_exc
for deps_path in [join(sep, "usr", "share", "bunkerweb", *paths) for paths in (("deps", "python"), ("utils",), ("db",))]:
if deps_path not in sys_path:
sys_path.append(deps_path)
from requests import RequestException, get
from logger import setup_logger # type: ignore
from jobs import Job # type: ignore
LOGGER = setup_logger("MODSECURITY.coreruleset-nightly", getenv("LOG_LEVEL", "INFO"))
status = 0
LOCK = Lock()
CRS_PATH = Path(sep, "var", "cache", "bunkerweb", "modsecurity", "crs")
try:
# * Check if we're using the nightly version of the Core Rule Set (CRS)
use_nightly_crs = False
if getenv("MODSECURITY_CRS_VERSION", "3") == "nightly":
use_nightly_crs = True
elif getenv("MULTISITE", "no") == "yes":
for first_server in getenv("SERVER_NAME", "").split(" "):
if first_server and getenv(f"{first_server}_MODSECURITY_CRS_VERSION", getenv("MODSECURITY_CRS_VERSION", "3")) == "nightly":
use_nightly_crs = True
break
if not use_nightly_crs:
LOGGER.info("Core Rule Set (CRS) nightly is not being used, skipping download...")
sys_exit(0)
JOB = Job(LOGGER)
LOGGER.info("Checking if Core Rule Set (CRS) nightly needs to be downloaded...")
commit_hash = JOB.get_cache("commit_hash")
resp = get("https://github.com/coreruleset/coreruleset/releases/tag/nightly", timeout=5)
resp.raise_for_status()
content = resp.text
page_commit_hash = search(r"/coreruleset/coreruleset/commit/(?P<hash>[0-9a-f]{40})", content, MULTILINE)
if page_commit_hash is None:
LOGGER.error("Failed to find commit hash on Core Rule Set (CRS) nightly page.")
sys_exit(2)
page_commit_hash = page_commit_hash.group("hash")
LOGGER.debug(f"Page commit hash: {page_commit_hash}")
if commit_hash:
LOGGER.debug(f"Current commit hash: {commit_hash.decode()}")
if commit_hash.decode() == page_commit_hash:
LOGGER.info("Core Rule Set (CRS) nightly is up to date.")
sys_exit(0)
LOGGER.info("Core Rule Set (CRS) nightly is outdated.")
cached, err = JOB.cache_file("commit_hash", page_commit_hash.encode())
if not cached:
LOGGER.error(f"Failed to cache the Core Rule Set (CRS) nightly commit hash: {err}")
status = 2
LOGGER.info("Downloading Core Rule Set (CRS) nightly tarball...")
file_content = BytesIO()
try:
with get("https://github.com/coreruleset/coreruleset/archive/refs/tags/nightly.tar.gz", stream=True, timeout=5) as resp:
resp.raise_for_status()
for chunk in resp.iter_content(chunk_size=4 * 1024):
if chunk:
file_content.write(chunk)
except RequestException:
LOGGER.exception("Failed to download Core Rule Set (CRS) nightly tarball.")
sys_exit(2)
file_content.seek(0)
rmtree(CRS_PATH, ignore_errors=True)
CRS_PATH.mkdir(parents=True, exist_ok=True)
LOGGER.info("Extracting Core Rule Set (CRS) nightly tarball...")
with tar_open(fileobj=file_content, mode="r:gz") as tar_file:
try:
tar_file.extractall(CRS_PATH, filter="data")
except TypeError:
tar_file.extractall(CRS_PATH)
# * Rename the extracted folder to "crs-nightly"
extracted_folder = next(CRS_PATH.iterdir())
extracted_folder.rename(CRS_PATH.joinpath("crs-nightly"))
# * Move and rename the example configuration file to "crs-setup-nightly.conf"
example_conf = CRS_PATH.joinpath("crs-nightly", "crs-setup.conf.example")
example_conf.rename(CRS_PATH.joinpath("crs-setup-nightly.conf"))
cached, err = JOB.cache_dir(CRS_PATH)
if not cached:
LOGGER.error(f"Error while saving Core Rule Set (CRS) nightly data to db cache: {err}")
else:
LOGGER.info("Successfully saved Core Rule Set (CRS) nightly data to db cache.")
status = 1
except SystemExit as e:
status = e.code
except:
status = 2
LOGGER.error(f"Exception while running coreruleset-nightly.py :\n{format_exc()}")
sys_exit(status)

16
src/common/core/modsecurity/plugin.json
View file

@ -26,12 +26,12 @@
"MODSECURITY_CRS_VERSION": {
"context": "multisite",
"default": "3",
"help": "Version of the OWASP Core Rule Set to use.",
"help": "Version of the OWASP Core Rule Set to use with ModSecurity (3, 4 or nightly).",
"id": "modsecurity-crs-version",
"label": "Core Rule Set Version",
"regex": "^(3|4)$",
"regex": "^(3|4|nightly)$",
"type": "select",
"select": ["3", "4"]
"select": ["3", "4", "nightly"]
},
"MODSECURITY_SEC_AUDIT_ENGINE": {
"context": "multisite",
@ -62,5 +62,13 @@
"regex": "^A(([B-K])(?!.*\\2))+Z$",
"type": "text"
}
}
},
"jobs": [
{
"name": "coreruleset-nightly",
"file": "coreruleset-nightly.py",
"every": "day",
"reload": true
}
]
}

8
tests/core/modsecurity/main.py
View file

@ -79,13 +79,17 @@ try:
bw_instance = bw_instances[0]
for log in bw_instance.logs(since=current_time).split(b"\n"):
if f'[ver "OWASP_CRS/{modsecurity_crs_version}'.encode() in log:
if (
modsecurity_crs_version == "nightly" and b'[file "/var/cache/bunkerweb/modsecurity/crs/crs-nightly' in log
) or f'[ver "OWASP_CRS/{modsecurity_crs_version}'.encode() in log:
found = True
break
else:
with open("/var/log/bunkerweb/error.log", "r") as f:
for line in f.readlines():
if search(r'\[ver "OWASP_CRS/' + modsecurity_crs_version, line):
if (modsecurity_crs_version == "nightly" and search(r'\[file "/var/cache/bunkerweb/modsecurity/crs/crs-nightly', line)) or search(
r'\[ver "OWASP_CRS/' + modsecurity_crs_version, line
):
found = True
break

14
tests/core/modsecurity/test.sh
View file

@ -47,7 +47,7 @@ cleanup_stack () {
if [ "$integration" == "docker" ] ; then
find . -type f -name 'docker-compose.*' -exec sed -i 's@USE_MODSECURITY: "no"@USE_MODSECURITY: "yes"@' {} \;
find . -type f -name 'docker-compose.*' -exec sed -i 's@USE_MODSECURITY_CRS: "no"@USE_MODSECURITY_CRS: "yes"@' {} \;
find . -type f -name 'docker-compose.*' -exec sed -i 's@MODSECURITY_CRS_VERSION: "4"@MODSECURITY_CRS_VERSION: "3"@' {} \;
find . -type f -name 'docker-compose.*' -exec sed -i 's@MODSECURITY_CRS_VERSION: ".*"@MODSECURITY_CRS_VERSION: "3"@' {} \;
else
sudo sed -i 's@USE_MODSECURITY=.*$@USE_MODSECURITY=yes@' /etc/bunkerweb/variables.env
sudo sed -i 's@USE_MODSECURITY_CRS=.*$@USE_MODSECURITY_CRS=yes@' /etc/bunkerweb/variables.env
@ -80,7 +80,7 @@ cleanup_stack () {
# Cleanup stack on exit
trap cleanup_stack EXIT
for test in "activated" "crs_deactivated" "crs_v4" "deactivated"
for test in "activated" "crs_deactivated" "crs_v4" "crs_nightly" "deactivated"
do
if [ "$test" = "activated" ] ; then
echo "👮 Running tests with modsecurity activated ..."
@ -95,11 +95,19 @@ do
elif [ "$test" = "crs_v4" ] ; then
echo "👮 Running tests with the CRS v4 ..."
if [ "$integration" == "docker" ] ; then
find . -type f -name 'docker-compose.*' -exec sed -i 's@MODSECURITY_CRS_VERSION: "3"@MODSECURITY_CRS_VERSION: "4"@' {} \;
find . -type f -name 'docker-compose.*' -exec sed -i 's@MODSECURITY_CRS_VERSION: ".*"@MODSECURITY_CRS_VERSION: "4"@' {} \;
else
sudo sed -i 's@MODSECURITY_CRS_VERSION=.*$@MODSECURITY_CRS_VERSION=4@' /etc/bunkerweb/variables.env
export MODSECURITY_CRS_VERSION="4"
fi
elif [ "$test" = "crs_nightly" ] ; then
echo "👮 Running tests with the CRS nightly ..."
if [ "$integration" == "docker" ] ; then
find . -type f -name 'docker-compose.*' -exec sed -i 's@MODSECURITY_CRS_VERSION: ".*"@MODSECURITY_CRS_VERSION: "nightly"@' {} \;
else
sudo sed -i 's@MODSECURITY_CRS_VERSION=.*$@MODSECURITY_CRS_VERSION=nightly@' /etc/bunkerweb/variables.env
export MODSECURITY_CRS_VERSION="nightly"
fi
elif [ "$test" = "deactivated" ] ; then
echo "👮 Running tests without modsecurity ..."
if [ "$integration" == "docker" ] ; then