Merge pull request #996 from bunkerity/dev

Merge branch "dev" into branch "staging"

docs/about.md

@@ -4,16 +4,24 @@
 
 BunkerWeb is maintained by [Bunkerity](https://www.bunkerity.com/?utm_campaign=self&utm_source=doc), a French 🇫🇷 company specialized in Cybersecurity 🛡️.
 
+## Do you have a professional version ?
+
+Yes, we do offer a professional version of BunkerWeb called "BunkerWeb PRO".
+
+Here are the main benefits of BunkerWeb PRO :
+
+- Unlock all features of BunkerWeb
+- Pay per protected services
+- Respond to professional needs
+- Free trial with no credit card
+- Best effort support included
+
+You can go the [BunkerWeb panel](https://panel.bunkerweb.io/?utm_campaign=self&utm_source=doc) to get more information and claim your free trial.
+
 ## Do you offer professional services ?
 
 Yes, we offer professional services related to BunkerWeb.
 
-**We have a [dedicated panel](https://panel.bunkerweb.io/?utm_campaign=self&utm_source=doc) to centralize all professional requests.**
-
-You can also contact use at [contact@bunkerity.com](mailto:contact@bunkerity.com) if you are interested.
-
-### Support
-
 You can get in touch with us about any of the following :
 
 - Consulting
@@ -21,13 +29,9 @@ You can get in touch with us about any of the following :
 - Custom development
 - Partnership
 
-### Pro version
+We have a [dedicated panel](https://panel.bunkerweb.io/?utm_campaign=self&utm_source=doc) to centralize all professional requests.
 
-We have a (pro version of BunkerWeb)[https://panel.bunkerweb.io/?utm_campaign=self&utm_source=doc#pro], which includes :
-
-- additional features and plugins
-- regular updates to improve your experience
-- we take your feedback into account to develop the plugins or offer you custom plugins.
+You can also contact use at [contact@bunkerity.com](mailto:contact@bunkerity.com) if you are interested.
 
 ## Where to get community support ?
 

docs/web-ui.md

@@ -733,7 +733,22 @@ You can access the account management page by clicking on `manage account` insid
 !!! info "What is BunkerWeb PRO ?"
     BunkerWeb PRO is an enhanced version of BunkerWeb open-source. Whether it's enhanced security, an enriched user experience, or technical supervision, the BunkerWeb PRO version will allow you to fully benefit from BunkerWeb and respond to your professional needs. Do not hesitate to visit the [BunkerWeb panel](https://panel.bunkerweb.io/knowledgebase?utm_campaign=self&utm_source=doc) or [contact us](https://panel.bunkerweb.io/contact.php?utm_campaign=self&utm_source=doc) if you have any question regarding the PRO version.
 
-TODO : screenshots
+Once you have your PRO license key from the [BunkerWeb panel](https://panel.bunkerweb.io/?utm_campaign=self&utm_source=doc), you can paste it into the PRO section of the account management page.
+
+<figure markdown>
+  ![PRO upgrade](assets/img/pro-ui-upgrade.webp){ align=center, width="550" }
+  <figcaption>Upgrade to PRO from the web UI</figcaption>
+</figure>
+
+!!! warning "Upgrade time"
+    The PRO version is downloaded in the background by the scheduler, it may take some time to upgrade.
+
+When your BunkerWeb instance has upgraded to the PRO version, you will see your license expiration date and the maximum number of services you can protect.
+
+<figure markdown>
+  ![PRO upgrade](assets/img/ui-pro.webp){ align=center, width="550" }
+  <figcaption>PRO license information</figcaption>
+</figure>
 
 ### Username / Password
 
@@ -1701,25 +1716,3 @@ After a successful login/password combination, you will be prompted to enter you
     ```shell
     systemctl restart bunkerweb
     ```
-
-## Upgrade to PRO
-
-In case you have buy a (pro version)[https://panel.bunkerweb.io/?utm_campaign=self&utm_source=doc#pro] and you already have a BunkerWeb setup with an UI, you can upgrade doing this :
-
-- access the **global config page**.
-- click on the **pro plugin**
-- fill the **setting Pro License Key**
-- **save** your changes
-
-!!! warning "Download"
-
-    The pro version is downloaded in the background by the scheduler. It may take some time before you see the changes to the UI.
-
-If your license key is valid, the upgrade to the pro version will take place automatically in the background.
-
-You can check the status of your version by going to the **home page**, you will see this in case of success :
-
-<figure markdown>
-  ![Overview](assets/img/pro-home-card.png){ align=center, width="450" }
-  <figcaption>PRO version card</figcaption>
-</figure>

src/common/utils/jobs.py

@@ -69,16 +69,17 @@ class Job:
                     extract_path = cache_path.parent
                     if job_cache_file["file_name"].startswith("folder:"):
                         extract_path = Path(job_cache_file["file_name"].split("folder:", 1)[1].rsplit(".tgz", 1)[0])
-                    ignored_dirs.add(extract_path)
+                    ignored_dirs.add(extract_path.as_posix())
                     if job_cache_file["job_name"] != job_name:
                         continue
-                    rmtree(extract_path, ignore_errors=True)
-                    extract_path.mkdir(parents=True, exist_ok=True)
-                    with tar_open(fileobj=BytesIO(job_cache_file["data"]), mode="r:gz") as tar:
-                        try:
-                            tar.extractall(extract_path, filter="fully_trusted")
-                        except TypeError:
-                            tar.extractall(extract_path)
+                    with LOCK:
+                        rmtree(extract_path, ignore_errors=True)
+                        extract_path.mkdir(parents=True, exist_ok=True)
+                        with tar_open(fileobj=BytesIO(job_cache_file["data"]), mode="r:gz") as tar:
+                            try:
+                                tar.extractall(extract_path, filter="fully_trusted")
+                            except TypeError:
+                                tar.extractall(extract_path)
                     continue
                 elif job_cache_file["job_name"] != job_name:
                     continue
@@ -88,27 +89,29 @@ class Job:
                 self.logger.error(f"Exception while restoring cache file {job_cache_file['file_name']} :\n{e}")
                 ret = False
 
-        if not manual and self.job_path.is_dir():
-            for file in self.job_path.glob("**/*"):
-                skipped = False
-                for ignored_dir in ignored_dirs:
-                    if file.as_posix().startswith(ignored_dir.as_posix()):
+        with LOCK:
+            if not manual and self.job_path.is_dir():
+                for file in self.job_path.rglob("*"):
+                    skipped = False
+                    if file.as_posix().startswith(tuple(ignored_dirs)):
                         skipped = True
                         break
 
-                if skipped:
-                    continue
+                    if skipped:
+                        continue
 
-                self.logger.debug(f"Checking if {file} should be removed")
-                if file not in plugin_cache_files and file.is_file():
-                    self.logger.debug(f"Removing non-cached file {file}")
-                    file.unlink(missing_ok=True)
-                    if file.parent.is_dir() and not list(file.parent.iterdir()):
-                        self.logger.debug(f"Removing empty directory {file.parent}")
-                        rmtree(file.parent, ignore_errors=True)
-                elif file.is_dir() and not list(file.iterdir()):
-                    self.logger.debug(f"Removing empty directory {file}")
-                    rmtree(file, ignore_errors=True)
+                    self.logger.debug(f"Checking if {file} should be removed")
+                    if file not in plugin_cache_files and file.is_file():
+                        self.logger.debug(f"Removing non-cached file {file}")
+                        file.unlink(missing_ok=True)
+                        if file.parent.is_dir() and not list(file.parent.iterdir()):
+                            self.logger.debug(f"Removing empty directory {file.parent}")
+                            rmtree(file.parent, ignore_errors=True)
+                            if file.parent == self.job_path:
+                                break
+                    elif file.is_dir() and not list(file.iterdir()):
+                        self.logger.debug(f"Removing empty directory {file}")
+                        rmtree(file, ignore_errors=True)
 
         return ret
 

src/linux/scripts/postinstall.sh

@@ -66,10 +66,10 @@ fi
 
 # Create wizard config
 if [ "$UI_WIZARD" != "" ] ; then
-    echo -ne 'DNS_RESOLVERS=8.8.8.8 8.8.4.4\nHTTP_PORT=80\nHTTPS_PORT=443\nAPI_LISTEN_IP=127.0.0.1\nMULTISITE=yes\nUI_HOST=http://127.0.0.1:7000\nSERVER_NAME=\n' > /etc/bunkerweb/variables.env
+    echo -ne 'DNS_RESOLVERS=9.9.9.9 8.8.8.8 8.8.4.4\nHTTP_PORT=80\nHTTPS_PORT=443\nAPI_LISTEN_IP=127.0.0.1\nMULTISITE=yes\nUI_HOST=http://127.0.0.1:7000\nSERVER_NAME=\n' > /etc/bunkerweb/variables.env
     do_and_check_cmd chown nginx:nginx /etc/bunkerweb/variables.env
     do_and_check_cmd chmod 660 /etc/bunkerweb/variables.env
-    echo "" > /etc/bunkerweb/ui.env
+    touch /etc/bunkerweb/ui.env
     do_and_check_cmd chown nginx:nginx /etc/bunkerweb/ui.env
     do_and_check_cmd chmod 660 /etc/bunkerweb/ui.env
     do_and_check_cmd systemctl enable bunkerweb-ui

src/ui/main.py

@@ -52,6 +52,7 @@ from src.User import AnonymousUser, User
 
 from utils import check_settings, get_b64encoded_qr_image, path_to_dict, get_remain
 from Database import Database  # type: ignore
+from logger import setup_logger  # type: ignore
 from logging import getLogger
 
 
@@ -87,9 +88,14 @@ app.config["SECRET_KEY"] = getenv("FLASK_SECRET", urandom(32))
 
 PROXY_NUMBERS = int(getenv("PROXY_NUMBERS", "1"))
 app.wsgi_app = ReverseProxied(app.wsgi_app, x_for=PROXY_NUMBERS, x_proto=PROXY_NUMBERS, x_host=PROXY_NUMBERS, x_prefix=PROXY_NUMBERS)
+app.logger = setup_logger("UI")
+
+gunicorn_access_logger = getLogger("gunicorn.access")
+gunicorn_access_logger.setLevel(app.logger.level)
 gunicorn_logger = getLogger("gunicorn.error")
-app.logger = gunicorn_logger
-app.logger.setLevel(gunicorn_logger.level)
+gunicorn_logger.setLevel(app.logger.level)
+werkzeug_logger = getLogger("werkzeug")
+werkzeug_logger.setLevel(app.logger.level)
 
 login_manager = LoginManager()
 login_manager.init_app(app)
@@ -2052,34 +2058,20 @@ def jobs():
 @app.route("/jobs/download", methods=["GET"])
 @login_required
 def jobs_download():
+    plugin_id = request.args.get("plugin_id", "")
     job_name = request.args.get("job_name", None)
     file_name = request.args.get("file_name", None)
+    service_id = request.args.get("service_id", "")
 
-    if not job_name or not file_name:
-        return (
-            jsonify(
-                {
-                    "status": "ko",
-                    "message": "job_name and file_name are required",
-                }
-            ),
-            422,
-        )
+    if not plugin_id or not job_name or not file_name:
+        return jsonify({"status": "ko", "message": "plugin_id, job_name and file_name are required"}), 422
 
-    cache_file = db.get_job_cache_file(job_name, file_name)
+    cache_file = db.get_job_cache_file(job_name, file_name, service_id=service_id, plugin_id=plugin_id)
 
     if not cache_file:
-        return (
-            jsonify(
-                {
-                    "status": "ko",
-                    "message": "file not found",
-                }
-            ),
-            404,
-        )
+        return jsonify({"status": "ko", "message": "file not found"}), 404
 
-    file = BytesIO(cache_file.data)
+    file = BytesIO(cache_file)
     return send_file(file, as_attachment=True, download_name=file_name)
 
 

src/ui/static/js/cache.js

@@ -20,21 +20,59 @@ class Download {
             .closest("button")
             .hasAttribute(`data-${this.prefix}-download`)
         ) {
+          const dataValue = e.target
+            .closest('div:has(span[data-cache-content=""])')
+            .firstElementChild.getAttribute("data-value");
           const btnEl = e.target.closest("button");
-          const jobName = btnEl.getAttribute("data-cache-download");
-          const fileName = btnEl.getAttribute("data-cache-file");
-          this.sendFileToDL(jobName, fileName);
+          const jobName = btnEl.getAttribute(`data-${this.prefix}-job`);
+          const fileName = btnEl.getAttribute(`data-${this.prefix}-download`);
+          const pluginId = document
+            .querySelector('[data-level="1"]')
+            .getAttribute("data-name");
+          var serviceId = null;
+          if (
+            document.querySelector(
+              '[data-level="2"][data-cache-breadcrumb-item=""]:not(.hidden)',
+            )
+          ) {
+            serviceId = document
+              .querySelector('[data-level="2"]')
+              .getAttribute("data-name");
+          }
+
+          if (dataValue !== "Download file to view content") {
+            this.download(fileName, dataValue);
+          } else {
+            this.sendFileToDL(pluginId, jobName, fileName, serviceId);
+          }
         }
       } catch (err) {}
     });
   }
 
-  async sendFileToDL(jobName, fileName) {
+  download(filename, text) {
+    var element = document.createElement("a");
+    element.setAttribute(
+      "href",
+      "data:text/plain;charset=utf-8," + encodeURIComponent(text),
+    );
+    element.setAttribute("download", filename);
+
+    element.style.display = "none";
+    document.body.appendChild(element);
+
+    element.click();
+
+    document.body.removeChild(element);
+  }
+
+  async sendFileToDL(pluginId, jobName, fileName, serviceId) {
     window.open(
       `${location.href.replace(
         "cache",
         "jobs",
-      )}/download?job_name=${jobName}&file_name=${fileName}`,
+      )}/download?plugin_id=${pluginId}&job_name=${jobName}&file_name=${fileName}` +
+        (serviceId ? `&service_id=${serviceId}` : ""),
     );
   }
 }

src/ui/static/js/utils/settings.js

@@ -88,6 +88,14 @@ class TabsSelect {
           //close dropdown and change btn textcontent on mobile
           this.setDropBtnText(tabAtt, text);
           this.closeDropdown();
+          // Change URL fragment
+          if (window.location.pathname.endsWith("global_config")) {
+            window.history.replaceState(
+              null,
+              "",
+              `${window.location.pathname}#${tabAtt}`,
+            );
+          }
         }
       } catch (e) {}
 
@@ -101,6 +109,24 @@ class TabsSelect {
         }
       } catch (err) {}
     });
+
+    // If fragment exists, click on the corresponding tab
+    if (
+      window.location.hash &&
+      window.location.pathname.endsWith("global_config")
+    ) {
+      const fragment = window.location.hash.substring(1);
+      if (fragment) {
+        const tab = this.tabContainer.querySelector(
+          `button[data-tab-select-handler='${fragment}']`,
+        );
+        tab.click();
+        // Scroll to the top of the page (with a delay to ensure the tab is clicked first)
+        setTimeout(() => {
+          window.scrollTo(0, 0);
+        }, 100);
+      }
+    }
   }
 
   resetTabsStyle() {

src/ui/templates/bans_modal.html

@@ -125,7 +125,7 @@
             <input data-ban-add-inp type="hidden" name="data" value="" />
             <!-- action button -->
             <div class="w-full justify-center flex mt-6 mb-4">
-                <button data-bans-modal-close type="button" class="close-btn mr-3 text-base">Close</button>
+                <button data-bans-modal-close type="button" class="dark:bg-slate-800 close-btn mr-3 text-base">Close</button>
                 <button disabled data-bans-modal-submit type="submit" class="valid-btn">Add</button>
             </div>
             <!-- end action button-->

src/ui/templates/file_manager.html

@@ -57,7 +57,7 @@
                             {% endif %}
                             <!-- end service root-->
                             <!-- services folder -->
-                            {% if child['type'] == "folder" and current_endpoint == "configs" and loop.depth != 1 or  child['type'] == "folder" and current_endpoint == "cache" and loop.depth > 2 %}
+                            {% if child['type'] == "folder" and loop.depth != 1 %}
                                 <svg class="col-span-2 h-[2.5rem] w-[2.5rem] fill-primary stroke-gray-100 dark:stroke-gray-400"
                                      xmlns="http://www.w3.org/2000/svg"
                                      fill="none"
@@ -161,7 +161,7 @@
                             <!-- download button -->
                             {% if child['type'] == "file" and child['can_download'] == True %}
                                 {% if current_endpoint == "cache" %}
-                                    <button role="tab"  value="download" data-{{ current_endpoint }}-download="{{ child['name'].split("/")[0] }}" data-{{ current_endpoint }}-file="{{ child['name'].split("/")[1] }}" data-{{ current_endpoint }}-setting-select-dropdown-btn="{{ child['name'].split("/")[0] }}" class="duration-300 border-gray-300 hover:brightness-90 bg-white text-white my-0 relative px-6 py-2 text-center align-middle transition-all rounded-none cursor-pointer leading-normal text-sm ease-in tracking-tight-rem dark:border-slate-600 dark:bg-slate-700 dark:text-gray-300 w-full border-b border-l border-r hover:bg-gray-100">
+                                    <button role="tab"  value="download" data-{{ current_endpoint }}-download="{{ child['name'] }}" data-{{ current_endpoint }}-job="{{ child['job_name'] }}" class="duration-300 border-gray-300 hover:brightness-90 bg-white text-white my-0 relative px-6 py-2 text-center align-middle transition-all rounded-none cursor-pointer leading-normal text-sm ease-in tracking-tight-rem dark:border-slate-600 dark:bg-slate-700 dark:text-gray-300 w-full border-b border-l border-r hover:bg-gray-100">
                                         <span class="flex justify-start items-center">
                                             <svg class="h-5.5 w-5.5 stroke-sky-500"
                                                  xmlns="http://www.w3.org/2000/svg"
@@ -242,7 +242,7 @@
 <!-- end main container -->
 <!-- modal -->
 <div data-{{ current_endpoint }}-modal class="hidden w-full h-screen fixed bg-gray-600/50 z-[1001] top-0 left-0 justify-center items-center">
-    <div class="mx-1 px-4 py-3 w-full max-w-180 flex flex-col break-words bg-white shadow-xl dark:bg-slate-850 dark:shadow-dark-xl rounded-2xl bg-clip-border">
+    <div class="mx-1 px-4 py-3 w-full max-w-screen-lg flex flex-col break-words bg-white shadow-xl dark:bg-slate-850 dark:shadow-dark-xl rounded-2xl bg-clip-border">
         <div class="w-full flex justify-between">
             <p data-{{ current_endpoint }}-modal-title class="dark:text-white mb-0 font-sans font-semibold leading-normal uppercase text-sm">
                 TITLE
@@ -269,11 +269,11 @@
             <input type="hidden" id="_type" value="file" name="type" />
             <textarea class="hidden" id="content" name="content"></textarea>
             <!-- editor-->
-            <div data-{{ current_endpoint }}-modal-editor id="editor" class="text-base w-full h-48 overflow-hidden overflow-y-auto my-2 border border-gray-300 dark:border-slate-800">
+            <div data-{{ current_endpoint }}-modal-editor id="editor" class="text-base w-full h-96 overflow-hidden overflow-y-auto my-2 border border-gray-300 dark:border-slate-800">
             </div>
             <!-- editor-->
             <div class="mt-4 w-full justify-end flex">
-                <button type="button" data-{{ current_endpoint }}-modal-close  class="close-btn text-xs mr-2">
+                <button type="button" data-{{ current_endpoint }}-modal-close  class="dark:bg-slate-800 close-btn text-xs mr-2">
                     Close
                 </button>
                 <button data-{{ current_endpoint }}-modal-submit type="submit" class="valid-btn text-xs">

src/ui/templates/flashs.html

@@ -9,14 +9,14 @@
                 aria-expanded="false"
                 aria-label="Open flash action sidebar"
                 data-flash-sidebar-open
-                class="transition scale-90 sm:scale-100 dark:brightness-95 p-3 text-xl bg-white shadow-sm cursor-pointer rounded-circle text-slate-700">
+                class="transition scale-90 sm:scale-100 dark:bg-slate-750 dark:brightness-95 dark:hover:brightness-105 hover:brightness-75 p-3 text-xl bg-white shadow-sm cursor-pointer rounded-circle text-slate-700">
             <svg class="fill-yellow-500 -translate-y-0.4 h-6 w-6"
                  xmlns="http://www.w3.org/2000/svg"
                  viewBox="0 0 448 512">
                 <path d="M224 0c-17.7 0-32 14.3-32 32V51.2C119 66 64 130.6 64 208v18.8c0 47-17.3 92.4-48.5 127.6l-7.4 8.3c-8.4 9.4-10.4 22.9-5.3 34.4S19.4 416 32 416H416c12.6 0 24-7.4 29.2-18.9s3.1-25-5.3-34.4l-7.4-8.3C401.3 319.2 384 273.9 384 226.8V208c0-77.4-55-142-128-156.8V32c0-17.7-14.3-32-32-32zm45.3 493.3c12-12 18.7-28.3 18.7-45.3H224 160c0 17 6.7 33.3 18.7 45.3s28.3 18.7 45.3 18.7s33.3-6.7 45.3-18.7z" />
             </svg>
         </button>
-        <div class="dark:brightness-95 px-2 translate-x-2 bottom-0 right-0 absolute rounded-full bg-white">
+        <div class="dark:bg-slate-700 dark:brightness-95 px-2 translate-x-2 bottom-0 right-0 absolute rounded-full bg-white">
             <p data-flash-count class="mb-0 text-sm text-bold text-red-500">
                 {% if messages %}
                     {{ messages|length }}

src/ui/templates/loading.html

@@ -43,7 +43,7 @@
         res = await response.json();
         if (res.reloading === false) {
           clearInterval(reloading);
-          window.location.replace("{{ next }}");
+          window.location.replace("{{ next }}" + (window.location.hash ? window.location.hash : ""));
         }
       }
     }

src/ui/templates/news.html

@@ -3,7 +3,7 @@
         aria-controls="sidebar-news"
         aria-expanded="false"
         aria-label="Open news sidebar"
-        class="transition-all scale-90 sm:scale-100 dark:brightness-95 dark:hover:brightness-105 hover:brightness-75 fixed p-3 text-xl bg-white shadow-sm cursor-pointer top-[4.5rem] right-5 sm:right-40 xl:right-6 z-990 rounded-circle text-slate-700">
+        class="transition-all scale-90 sm:scale-100 dark:bg-slate-750 dark:brightness-95 dark:hover:brightness-105 hover:brightness-75 fixed p-3 text-xl bg-white shadow-sm cursor-pointer top-[4.5rem] right-5 sm:right-40 xl:right-6 z-990 rounded-circle text-slate-700">
     <svg class="fill-sky-500 h-6 w-6"
          xmlns="http://www.w3.org/2000/svg"
          viewBox="0 0 512 512">

src/ui/templates/services_modal.html

@@ -91,7 +91,7 @@
                 <div class="flex justify-center">
                     <button data-services-modal-close
                             type="button"
-                            class="close-btn mb-4 mr-3 text-base">Close</button>
+                            class="dark:bg-slate-800 close-btn mb-4 mr-3 text-base">Close</button>
                     <button data-services-modal-submit type="submit" class="mb-4 valid-btn">Save</button>
                 </div>
                 <!-- end action button-->
@@ -118,7 +118,7 @@
             <div class="w-full justify-center flex mt-10">
                 <button data-services-modal-close
                         type="button"
-                        class="close-btn mb-4 mr-3 text-base">Close</button>
+                        class="dark:bg-slate-800 close-btn mb-4 mr-3 text-base">Close</button>
                 <button type="submit" class="delete-btn mb-4 mr-3 text-base">Delete</button>
             </div>
             <!-- end action button-->

src/ui/utils.py

@@ -159,6 +159,7 @@ def path_to_dict(
 
             file_info = {
                 "name": conf["file_name"],
+                "job_name": conf["job_name"],
                 "type": "file",
                 "path": join(
                     path,