Explicitly log migration process

Signed-off-by: DL6ER <dl6er@dl6er.de>
Merge branch 'fix/install' into fix/migration_test
2024-12-23 08:52:56 +01:00 · 2024-12-23 08:44:24 +01:00
47 changed files with 1011 additions and 981 deletions
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -1,5 +0,0 @@
-# see https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners#codeowners-syntax
-
-# These owners will be the default owners for everything in
-# the repo. Unless a later match takes precedence,
-*       @pi-hole/core-maintainers
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -8,6 +8,8 @@ updates:
    time: "10:00"
  open-pull-requests-limit: 10
  target-branch: development
+  reviewers:
+    - "pi-hole/core-maintainers"
 - package-ecosystem: pip
  directory: "/test"
  schedule:
@@ -16,3 +18,5 @@ updates:
    time: "10:00"
  open-pull-requests-limit: 10
  target-branch: development
+  reviewers:
+    - "pi-hole/core-maintainers"
--- a/.github/workflows/merge-conflict.yml
+++ b/.github/workflows/merge-conflict.yml
@@ -13,7 +13,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - name: Check if PRs are have merge conflicts
-        uses: eps1lon/actions-label-merge-conflict@v3.0.3
+        uses: eps1lon/actions-label-merge-conflict@v3.0.2
        with:
          dirtyLabel: "PR: Merge Conflict"
          repoToken: "${{ secrets.GITHUB_TOKEN }}"
--- a/.github/workflows/stale.yml
+++ b/.github/workflows/stale.yml
@@ -17,7 +17,7 @@ jobs:
      issues: write

    steps:
-      - uses: actions/stale@v9.1.0
+      - uses: actions/stale@v9.0.0
        with:
          repo-token: ${{ secrets.GITHUB_TOKEN }}
          days-before-stale: 30
--- a/.github/workflows/stale_pr.yml
+++ b/.github/workflows/stale_pr.yml
@@ -17,7 +17,7 @@ jobs:
      pull-requests: write

    steps:
-      - uses: actions/stale@v9.1.0
+      - uses: actions/stale@v9.0.0
        with:
          repo-token: ${{ secrets.GITHUB_TOKEN }}
          # Do not automatically mark PR/issue as stale
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -19,8 +19,6 @@ jobs:
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4.2.2
-        with:
-          fetch-depth: 0 # Differential ShellCheck requires full git history

      - name: Check scripts in repository are executable
        run: |
@@ -30,12 +28,12 @@ jobs:
          # If FAIL is 1 then we fail.
          [[ $FAIL == 1 ]] && exit 1 || echo "Scripts are executable!"

-      - name: Differential ShellCheck
-        uses: redhat-plumbers-in-action/differential-shellcheck@v5
+      - name: Run shellcheck
+        uses: ludeeus/action-shellcheck@master
        with:
-          severity: warning
-          display-engine: sarif-fmt
-
+          check_together: 'yes'
+          format: tty
+          severity: error

      - name: Spell-Checking
        uses: codespell-project/actions-codespell@master
@@ -69,10 +67,8 @@ jobs:
            ubuntu_22,
            ubuntu_24,
            centos_9,
-            centos_10,
            fedora_40,
            fedora_41,
-            fedora_42,
          ]
    env:
      DISTRO: ${{matrix.distro}}
@@ -81,7 +77,7 @@ jobs:
        uses: actions/checkout@v4.2.2

      - name: Set up Python 3.10
-        uses: actions/setup-python@v5.6.0
+        uses: actions/setup-python@v5.3.0
        with:
          python-version: "3.10"

--- a/.gitignore
+++ b/.gitignore
@@ -10,6 +10,3 @@ __pycache__
 .idea/
 *.iml
 .vscode/
-.venv/
-.fleet/
-.cache/
--- a/.shellcheckrc
+++ b/.shellcheckrc
@@ -1,2 +0,0 @@
-external-sources=true # allow shellcheck to read external sources
-disable=SC3043 #disable SC3043: In POSIX sh, local is undefined.
--- a/README.md
+++ b/README.md
@@ -3,9 +3,13 @@
 #

 <p align="center">
-  <img src="https://raw.githubusercontent.com/pi-hole/graphics/refs/heads/master/Vortex/vortex_with_text.svg" alt="Pi-hole website" width="168" height="270">
-  <br>
-  <strong>Network-wide ad blocking via your own Linux hardware</strong>
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://pi-hole.github.io/graphics/Vortex/Vortex_Vertical_wordmark_darkmode.png">
+    <source media="(prefers-color-scheme: light)" srcset="https://pi-hole.github.io/graphics/Vortex/Vortex_Vertical_wordmark_lightmode.png">
+    <img src="https://pi-hole.github.io/graphics/Vortex/Vortex_Vertical_wordmark_lightmode.png" width="168" height="270" alt="Pi-hole website">
+  </picture>
+    <br>
+    <strong>Network-wide ad blocking via your own Linux hardware</strong>
 </p>

 <!-- markdownlint-enable MD033 -->
@@ -128,10 +132,7 @@ Some of the statistics you can integrate include:
 - Queries cached
 - Unique clients

-Access the API using:
- your browser: http://pi.hole/api/docs
- `curl`: `curl --connect-timeout 2 -ks "https://pi.hole/api/stats/summary" -H "Accept: application/json"`;
- the command line - examples: `pihole api config/webserver/port` or `pihole api stats/summary`.
+Access the API via [`telnet`](https://github.com/pi-hole/FTL), the Web (`admin/api.php`) and Command Line (`pihole -c -j`). You can find out [more details over here](https://discourse.pi-hole.net/t/pi-hole-api/1863).

 ### The Command-Line Interface

@@ -139,7 +140,7 @@ The [pihole](https://docs.pi-hole.net/core/pihole-command/) command has all the

 Some notable features include:

- [Allowlisting, Denylisting (fka Whitelisting, Blacklisting), and Regex](https://docs.pi-hole.net/core/pihole-command/#allowlisting-denylisting-and-regex)
+- [Whitelisting, Blacklisting, and Regex](https://docs.pi-hole.net/core/pihole-command/#whitelisting-blacklisting-and-regex)
 - [Debugging utility](https://docs.pi-hole.net/core/pihole-command/#debugger)
 - [Viewing the live log file](https://docs.pi-hole.net/core/pihole-command/#tail)
 - [Updating Ad Lists](https://docs.pi-hole.net/core/pihole-command/#gravity)
--- a/advanced/Scripts/COL_TABLE
+++ b/advanced/Scripts/COL_TABLE
@@ -1,6 +1,5 @@
-#!/usr/bin/env sh
 # Determine if terminal is capable of showing colors
-if [ -t 1 ] && [ "$(tput colors)" -ge 8 ]; then
+if ([ -t 1 ] && [ $(tput colors) -ge 8 ]) || [ "${WEBCALL}" ]; then
  # Bold and underline may not show up on all clients
  # If something MUST be emphasized, use both
  COL_BOLD='[1m'
--- a/advanced/Scripts/api.sh
+++ b/advanced/Scripts/api.sh
@@ -1,4 +1,5 @@
 #!/usr/bin/env sh
+# shellcheck disable=SC3043 #https://github.com/koalaman/shellcheck/wiki/SC3043#exceptions

 # Pi-hole: A black hole for Internet advertisements
 # (c) 2017 Pi-hole, LLC (https://pi-hole.net)
@@ -19,19 +20,13 @@

 TestAPIAvailability() {

-    local chaos_api_list authResponse authStatus authData apiAvailable DNSport
-
    # as we are running locally, we can get the port value from FTL directly
-    readonly utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-    # shellcheck source=./advanced/Scripts/utils.sh
-    . "${utilsfile}"
-
-    DNSport=$(getFTLConfigValue dns.port)
+    local chaos_api_list availabilityResponse

    # Query the API URLs from FTL using CHAOS TXT local.api.ftl
    # The result is a space-separated enumeration of full URLs
    # e.g., "http://localhost:80/api/" "https://localhost:443/api/"
-    chaos_api_list="$(dig +short -p "${DNSport}" chaos txt local.api.ftl @127.0.0.1)"
+    chaos_api_list="$(dig +short chaos txt local.api.ftl @127.0.0.1)"

    # If the query was not successful, the variable is empty
    if [ -z "${chaos_api_list}" ]; then
@@ -39,12 +34,6 @@ TestAPIAvailability() {
        exit 1
    fi

-    # If an error occurred, the variable starts with ;;
-    if [ "${chaos_api_list#;;}" != "${chaos_api_list}" ]; then
-        echo "Communication error. Is FTL running?"
-        exit 1
-    fi
-
    # Iterate over space-separated list of URLs
    while [ -n "${chaos_api_list}" ]; do
        # Get the first URL
@@ -53,50 +42,39 @@ TestAPIAvailability() {
        API_URL="${API_URL%\"}"
        API_URL="${API_URL#\"}"

-        # Test if the API is available at this URL, include delimiter for ease in splitting payload
-        authResponse=$(curl --connect-timeout 2 -skS -w ">>%{http_code}" "${API_URL}auth")
-
-        # authStatus is the response http_code, eg. 200, 401.
-        # Shell parameter expansion, remove everything up to and including the >> delim
-        authStatus=${authResponse#*>>}
-        # data is everything from response
-        # Shell parameter expansion, remove the >> delim and everything after
-        authData=${authResponse%>>*}
+        # Test if the API is available at this URL
+        availabilityResponse=$(curl -skS -o /dev/null -w "%{http_code}" "${API_URL}auth")

        # Test if http status code was 200 (OK) or 401 (authentication required)
-        if [ "${authStatus}" = 200 ]; then
-            # API is available without authentication
-            apiAvailable=true
-            needAuth=false
-            break
-
-        elif [ "${authStatus}" = 401 ]; then
-            # API is available with authentication
-            apiAvailable=true
-            needAuth=true
-            # Check if 2FA is required
-            needTOTP=$(echo "${authData}"| jq --raw-output .session.totp 2>/dev/null)
-            break
-
-        else
+        if [ ! "${availabilityResponse}" = 200 ] && [ ! "${availabilityResponse}" = 401 ]; then
            # API is not available at this port/protocol combination
-            apiAvailable=false
-            # Remove the first URL from the list
-            local last_api_list
-            last_api_list="${chaos_api_list}"
-            chaos_api_list="${chaos_api_list#* }"
+            API_PORT=""
+        else
+            # API is available at this URL combination

-            # If the list did not change, we are at the last element
-            if [ "${last_api_list}" = "${chaos_api_list}" ]; then
-                # Remove the last element
-                chaos_api_list=""
+            if [ "${availabilityResponse}" = 200 ]; then
+                # API is available without authentication
+                needAuth=false
            fi
+
+            break
+        fi
+
+        # Remove the first URL from the list
+        local last_api_list
+        last_api_list="${chaos_api_list}"
+        chaos_api_list="${chaos_api_list#* }"
+
+        # If the list did not change, we are at the last element
+        if [ "${last_api_list}" = "${chaos_api_list}" ]; then
+            # Remove the last element
+            chaos_api_list=""
        fi
    done

-    # if apiAvailable is false, no working API was found
-    if [ "${apiAvailable}" = false ]; then
-        echo "API not available. Please check FTL.log"
+    # if API_PORT is empty, no working API port was found
+    if [ -n "${API_PORT}" ]; then
+        echo "API not available at: ${API_URL}"
        echo "Exiting."
        exit 1
    fi
@@ -124,51 +102,22 @@ LoginAPI() {
            echo "API Authentication: Trying to use CLI password"
        fi

-        # If we can read the CLI password, we can skip 2FA even when it's required otherwise
-        needTOTP=false
+        # Try to authenticate using the CLI password
+        Authentication "${1}"
+
    elif [ "${1}" = "verbose" ]; then
        echo "API Authentication: CLI password not available"
    fi

-    if [ -z "${password}" ]; then
-        # no password read from CLI file
-        echo "Please enter your password:"
-        # secretly read the password
-        secretRead; printf '\n'
-    fi

-    if [ "${needTOTP}" = true ]; then
-        # 2FA required
-        echo "Please enter the correct second factor."
-        echo "(Can be any number if you used the app password)"
-        read -r totp
-    fi

-    # Try to authenticate using the supplied password (CLI file or user input) and TOTP
-    Authentication "${1}"
-
-    # Try to login again until the session is valid
-    while [ ! "${validSession}" = true ]  ; do
+    # If this did not work, ask the user for the password
+    while [ "${validSession}" = false ] || [ -z "${validSession}" ] ; do
        echo "Authentication failed. Please enter your Pi-hole password"

-        # Print the error message if there is one
-        if  [ ! "${sessionError}" = "null"  ] && [ "${1}" = "verbose" ]; then
-            echo "Error: ${sessionError}"
-        fi
-        # Print the session message if there is one
-        if  [ ! "${sessionMessage}" = "null" ] && [ "${1}" = "verbose" ]; then
-            echo "Error: ${sessionMessage}"
-        fi
-
        # secretly read the password
        secretRead; printf '\n'

-        if [ "${needTOTP}" = true ]; then
-            echo "Please enter the correct second factor:"
-            echo "(Can be any number if you used the app password)"
-            read -r totp
-        fi
-
        # Try to authenticate again
        Authentication "${1}"
    done
@@ -176,27 +125,23 @@ LoginAPI() {
 }

 Authentication() {
-    sessionResponse="$(curl --connect-timeout 2 -skS -X POST "${API_URL}auth" --user-agent "Pi-hole cli" --data "{\"password\":\"${password}\", \"totp\":${totp:-null}}" )"
+  sessionResponse="$(curl -skS -X POST "${API_URL}auth" --user-agent "Pi-hole cli " --data "{\"password\":\"${password}\"}" )"

-    if [ -z "${sessionResponse}" ]; then
-        echo "No response from FTL server. Please check connectivity"
-        exit 1
-    fi
-    # obtain validity, session ID and sessionMessage from session response
-    validSession=$(echo "${sessionResponse}"| jq .session.valid 2>/dev/null)
-    SID=$(echo "${sessionResponse}"| jq --raw-output .session.sid 2>/dev/null)
-    sessionMessage=$(echo "${sessionResponse}"| jq --raw-output .session.message 2>/dev/null)
-
-    # obtain the error message from the session response
-    sessionError=$(echo "${sessionResponse}"| jq --raw-output .error.message 2>/dev/null)
-
-    if [ "${1}" = "verbose" ]; then
-        if [ "${validSession}" = true ]; then
-            echo "API Authentication: ${COL_GREEN}Success${COL_NC}"
-        else
-            echo "API Authentication: ${COL_RED}Failed${COL_NC}"
-        fi
+  if [ -z "${sessionResponse}" ]; then
+    echo "No response from FTL server. Please check connectivity"
+    exit 1
+  fi
+  # obtain validity and session ID from session response
+  validSession=$(echo "${sessionResponse}"| jq .session.valid 2>/dev/null)
+  SID=$(echo "${sessionResponse}"| jq --raw-output .session.sid 2>/dev/null)
+
+  if [ "${1}" = "verbose" ]; then
+    if [ "${validSession}" = true ]; then
+      echo "API Authentication: ${COL_GREEN}Success${COL_NC}"
+    else
+      echo "API Authentication: ${COL_RED}Failed${COL_NC}"
    fi
+  fi
 }

 LogoutAPI() {
@@ -220,21 +165,19 @@ GetFTLData() {
  # get the data from querying the API as well as the http status code
  response=$(curl -skS -w "%{http_code}" -X GET "${API_URL}$1" -H "Accept: application/json" -H "sid: ${SID}" )

+  # status are the last 3 characters
+  status="${response#"${response%???}"}"
+  # data is everything from response without the last 3 characters
+  data="${response%???}"
+
  if [ "${2}" = "raw" ]; then
    # return the raw response
    echo "${response}"
  else
-
-    # status are the last 3 characters
-    # not using ${response#"${response%???}"}" here because it's extremely slow on big responses
-    status=$(printf "%s" "${response}" | tail -c 3)
-    # data is everything from response without the last 3 characters
-    data="${response%???}"
-
    # return only the data
    if [ "${status}" = 200 ]; then
        # response OK
-        printf %s "${data}"
+        echo "${data}"
    else
        # connection lost
        echo "${status}"
@@ -308,30 +251,20 @@ secretRead() {
 }

 apiFunc() {
-  local data response status status_col verbosity
-
-  # Define if the output will be silent (default) or verbose
-  verbosity="silent"
-  if [ "$1" = "verbose" ]; then
-    verbosity="verbose"
-    shift
-  fi
+  local data response status status_col

  # Authenticate with the API
-  LoginAPI "${verbosity}"
+  LoginAPI verbose
+  echo ""

-  if [ "${verbosity}" = "verbose" ]; then
-    echo ""
-    echo "Requesting: ${COL_PURPLE}GET ${COL_CYAN}${API_URL}${COL_YELLOW}$1${COL_NC}"
-    echo ""
-  fi
+  echo "Requesting: ${COL_PURPLE}GET ${COL_CYAN}${API_URL}${COL_YELLOW}$1${COL_NC}"
+  echo ""

  # Get the data from the API
  response=$(GetFTLData "$1" raw)

  # status are the last 3 characters
-  # not using ${response#"${response%???}"}" here because it's extremely slow on big responses
-  status=$(printf "%s" "${response}" | tail -c 3)
+  status="${response#"${response%???}"}"
  # data is everything from response without the last 3 characters
  data="${response%???}"

@@ -341,18 +274,11 @@ apiFunc() {
  else
    status_col="${COL_RED}"
  fi
-
-  # Only print the status in verbose mode or if the status is not 200
-  if [ "${verbosity}" = "verbose" ] || [ "${status}" != 200 ]; then
-    echo "Status: ${status_col}${status}${COL_NC}"
-  fi
+  echo "Status: ${status_col}${status}${COL_NC}"

  # Output the data. Format it with jq if available and data is actually JSON.
  # Otherwise just print it
-  if [ "${verbosity}" = "verbose" ]; then
-    echo "Data:"
-  fi
-
+  echo "Data:"
  if command -v jq >/dev/null && echo "${data}" | jq . >/dev/null 2>&1; then
    echo "${data}" | jq .
  else
@@ -360,5 +286,5 @@ apiFunc() {
  fi

  # Delete the session
-  LogoutAPI "${verbosity}"
+  LogoutAPI verbose
 }
--- a/advanced/Scripts/database_migration/gravity-db.sh
+++ b/advanced/Scripts/database_migration/gravity-db.sh
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-
+# shellcheck disable=SC1090

 # Pi-hole: A black hole for Internet advertisements
 # (c) 2019 Pi-hole, LLC (https://pi-hole.net)
@@ -13,8 +13,9 @@
 readonly scriptPath="/etc/.pihole/advanced/Scripts/database_migration/gravity"

 upgrade_gravityDB(){
-    local database version
+    local database piholeDir version
    database="${1}"
+    piholeDir="${2}"

    # Exit early if the database does not exist (e.g. in CI tests)
    if [[ ! -f "${database}" ]]; then
--- a/advanced/Scripts/list.sh
+++ b/advanced/Scripts/list.sh
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# shellcheck disable=SC1090

 # Pi-hole: A black hole for Internet advertisements
 # (c) 2017 Pi-hole, LLC (https://pi-hole.net)
@@ -11,11 +12,9 @@

 readonly PI_HOLE_SCRIPT_DIR="/opt/pihole"
 readonly utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source="./advanced/Scripts/utils.sh"
 source "${utilsfile}"

 readonly apifile="${PI_HOLE_SCRIPT_DIR}/api.sh"
-# shellcheck source="./advanced/Scripts/api.sh"
 source "${apifile}"

 # Determine database location
@@ -40,7 +39,6 @@ typeId=""
 comment=""

 colfile="/opt/pihole/COL_TABLE"
-# shellcheck source="./advanced/Scripts/COL_TABLE"
 source ${colfile}

 helpFunc() {
--- a/advanced/Scripts/piholeARPTable.sh
+++ b/advanced/Scripts/piholeARPTable.sh
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# shellcheck disable=SC1090

 # Pi-hole: A black hole for Internet advertisements
 # (c) 2019 Pi-hole, LLC (https://pi-hole.net)
@@ -11,13 +12,11 @@

 coltable="/opt/pihole/COL_TABLE"
 if [[ -f ${coltable} ]]; then
-# shellcheck source="./advanced/Scripts/COL_TABLE"
    source ${coltable}
 fi

 readonly PI_HOLE_SCRIPT_DIR="/opt/pihole"
 utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source=./advanced/Scripts/utils.sh
 source "${utilsfile}"

 # Determine database location
--- a/advanced/Scripts/piholeCheckout.sh
+++ b/advanced/Scripts/piholeCheckout.sh
@@ -10,7 +10,6 @@

 readonly PI_HOLE_FILES_DIR="/etc/.pihole"
 SKIP_INSTALL="true"
-# shellcheck source="./automated install/basic-install.sh"
 source "${PI_HOLE_FILES_DIR}/automated install/basic-install.sh"

 # webInterfaceGitUrl set in basic-install.sh
@@ -110,7 +109,7 @@ checkout() {
            echo -e "${OVER}  ${CROSS} $str"
            exit 1
        fi
-        mapfile -t corebranches < <(get_available_branches "${PI_HOLE_FILES_DIR}")
+        corebranches=($(get_available_branches "${PI_HOLE_FILES_DIR}"))

        if [[ "${corebranches[*]}" == *"master"* ]]; then
            echo -e "${OVER}  ${TICK} $str"
@@ -137,7 +136,7 @@ checkout() {
            echo -e "${OVER}  ${CROSS} $str"
            exit 1
        fi
-        mapfile -t webbranches < <(get_available_branches "${webInterfaceDir}")
+        webbranches=($(get_available_branches "${webInterfaceDir}"))

        if [[ "${webbranches[*]}" == *"master"* ]]; then
            echo -e "${OVER}  ${TICK} $str"
@@ -168,7 +167,7 @@ checkout() {

        # Check if requested branch is available
        echo -e "  ${INFO} Checking for availability of branch ${COL_CYAN}${2}${COL_NC} on GitHub"
-        mapfile -t ftlbranches < <(git ls-remote https://github.com/pi-hole/ftl | grep "refs/heads" | cut -d'/' -f3- -)
+        ftlbranches=( $(git ls-remote https://github.com/pi-hole/ftl | grep "refs/heads" | cut -d'/' -f3- -) )
        # If returned array is empty -> connectivity issue
        if [[ ${#ftlbranches[@]} -eq 0 ]]; then
            echo -e "  ${CROSS} Unable to fetch branches from GitHub. Please check your Internet connection and try again later."
@@ -210,15 +209,13 @@ checkout() {
            # Update local and remote versions via updatechecker
            /opt/pihole/updatecheck.sh
        else
-            local status
-            status=$?
-            if [ $status -eq 1 ]; then
+            if [ $? -eq 1 ]; then
                # Binary for requested branch is not available, may still be
                # int he process of being built or CI build job failed
-                printf "  %b Binary for requested branch is not available, please try again later.\\n" "${CROSS}"
+                printf "  %b Binary for requested branch is not available, please try again later.\\n" ${CROSS}
                printf "      If the issue persists, please contact Pi-hole Support and ask them to re-generate the binary.\\n"
                exit 1
-            elif [ $status -eq 2 ]; then
+            elif [ $? -eq 2 ]; then
                printf "  %b Unable to download from ftl.pi-hole.net. Please check your Internet connection and try again later.\\n" "${CROSS}"
                exit 1
            else
--- a/advanced/Scripts/piholeDebug.sh
+++ b/advanced/Scripts/piholeDebug.sh
@@ -8,6 +8,7 @@
 # This file is copyright under the latest version of the EUPL.
 # Please see LICENSE file for your rights under this license.

+# shellcheck source=/dev/null

 # -e option instructs bash to immediately exit if any command [1] has a non-zero exit status
 # -u a reference to any variable you haven't previously defined
@@ -26,7 +27,6 @@ PIHOLE_COLTABLE_FILE="${PIHOLE_SCRIPTS_DIRECTORY}/COL_TABLE"

 # These provide the colors we need for making the log more readable
 if [[ -f ${PIHOLE_COLTABLE_FILE} ]]; then
-# shellcheck source=./advanced/Scripts/COL_TABLE
    source ${PIHOLE_COLTABLE_FILE}
 else
    COL_NC='\e[0m' # No Color
@@ -41,17 +41,9 @@ else
    #OVER="\r\033[K"
 fi

-# shellcheck source=/dev/null
+# shellcheck disable=SC1091
 . /etc/pihole/versions

-# Read the value of an FTL config key. The value is printed to stdout.
-get_ftl_conf_value() {
-    local key=$1
-
-    # Obtain setting from FTL directly
-    pihole-FTL --config "${key}"
-}
-
 # FAQ URLs for use in showing the debug log
 FAQ_HARDWARE_REQUIREMENTS="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/${COL_NC}"
 FAQ_HARDWARE_REQUIREMENTS_PORTS="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/#ports${COL_NC}"
@@ -69,10 +61,10 @@ DNSMASQ_D_DIRECTORY="/etc/dnsmasq.d"
 PIHOLE_DIRECTORY="/etc/pihole"
 PIHOLE_SCRIPTS_DIRECTORY="/opt/pihole"
 BIN_DIRECTORY="/usr/local/bin"
+RUN_DIRECTORY="/run"
 LOG_DIRECTORY="/var/log/pihole"
-HTML_DIRECTORY="$(get_ftl_conf_value "webserver.paths.webroot")"
-WEBHOME_PATH="$(get_ftl_conf_value "webserver.paths.webhome")"
-WEB_GIT_DIRECTORY="${HTML_DIRECTORY}${WEBHOME_PATH}"
+HTML_DIRECTORY="/var/www/html"
+WEB_GIT_DIRECTORY="${HTML_DIRECTORY}/admin"
 SHM_DIRECTORY="/dev/shm"
 ETC="/etc"

@@ -87,6 +79,14 @@ PIHOLE_FTL_CONF_FILE="${PIHOLE_DIRECTORY}/pihole.toml"
 PIHOLE_DNSMASQ_CONF_FILE="${PIHOLE_DIRECTORY}/dnsmasq.conf"
 PIHOLE_VERSIONS_FILE="${PIHOLE_DIRECTORY}/versions"

+# Read the value of an FTL config key. The value is printed to stdout.
+get_ftl_conf_value() {
+    local key=$1
+
+    # Obtain setting from FTL directly
+    pihole-FTL --config "${key}"
+}
+
 PIHOLE_GRAVITY_DB_FILE="$(get_ftl_conf_value "files.gravity")"

 PIHOLE_FTL_DB_FILE="$(get_ftl_conf_value "files.database")"
@@ -94,7 +94,7 @@ PIHOLE_FTL_DB_FILE="$(get_ftl_conf_value "files.database")"
 PIHOLE_COMMAND="${BIN_DIRECTORY}/pihole"
 PIHOLE_COLTABLE_FILE="${BIN_DIRECTORY}/COL_TABLE"

-FTL_PID="$(get_ftl_conf_value "files.pid")"
+FTL_PID="${RUN_DIRECTORY}/pihole-FTL.pid"

 PIHOLE_LOG="${LOG_DIRECTORY}/pihole.log"
 PIHOLE_LOG_GZIPS="${LOG_DIRECTORY}/pihole.log.[0-9].*"
@@ -202,7 +202,7 @@ compare_local_version_to_git_version() {
        if git status &> /dev/null; then
            # The current version the user is on
            local local_version
-            local_version=$(git describe --tags --abbrev=0 2> /dev/null);
+            local_version=$(git describe --tags --abbrev=0);
            # What branch they are on
            local local_branch
            local_branch=$(git rev-parse --abbrev-ref HEAD);
@@ -213,13 +213,7 @@ compare_local_version_to_git_version() {
            local local_status
            local_status=$(git status -s)
            # echo this information out to the user in a nice format
-            if [ "${local_version}" ]; then
-              log_write "${TICK} Version: ${local_version}"
-            elif [ -n "${DOCKER_VERSION}" ]; then
-              log_write "${TICK} Version: Pi-hole Docker Container ${COL_BOLD}${DOCKER_VERSION}${COL_NC}"
-            else
-              log_write "${CROSS} Version: not detected"
-            fi
+            log_write "${TICK} Version: ${local_version}"

            # Print the repo upstreams
            remotes=$(git remote -v)
@@ -296,12 +290,88 @@ check_component_versions() {
    check_ftl_version
 }

+os_check() {
+    # This function gets a list of supported OS versions from a TXT record at versions.pi-hole.net
+    # and determines whether or not the script is running on one of those systems
+    local remote_os_domain valid_os valid_version detected_os detected_version cmdResult digReturnCode response
+    remote_os_domain=${OS_CHECK_DOMAIN_NAME:-"versions.pi-hole.net"}
+
+    detected_os=$(grep "\bID\b" /etc/os-release | cut -d '=' -f2 | tr -d '"')
+    detected_version=$(grep VERSION_ID /etc/os-release | cut -d '=' -f2 | tr -d '"')
+
+    cmdResult="$(dig -4 +short -t txt "${remote_os_domain}" @ns1.pi-hole.net 2>&1; echo $?)"
+    #Get the return code of the previous command (last line)
+    digReturnCode="${cmdResult##*$'\n'}"
+
+    # Extract dig response
+    response="${cmdResult%%$'\n'*}"
+
+    if [ "${digReturnCode}" -ne 0 ]; then
+        log_write "${INFO} Distro: ${detected_os^}"
+        log_write "${INFO} Version: ${detected_version}"
+        log_write "${CROSS} dig IPv4 return code: ${COL_RED}${digReturnCode}${COL_NC}"
+        log_write "${CROSS} dig response: ${response}"
+        log_write "${INFO} Retrying via IPv6"
+
+        cmdResult="$(dig -6 +short -t txt "${remote_os_domain}" @ns1.pi-hole.net 2>&1; echo $?)"
+        #Get the return code of the previous command (last line)
+        digReturnCode="${cmdResult##*$'\n'}"
+
+        # Extract dig response
+        response="${cmdResult%%$'\n'*}"
+    fi
+    # If also no success via IPv6
+    if [ "${digReturnCode}" -ne 0 ]; then
+        log_write "${CROSS} dig IPv6 return code: ${COL_RED}${digReturnCode}${COL_NC}"
+        log_write "${CROSS} dig response: ${response}"
+        log_write "${CROSS} Error: ${COL_RED}dig command failed - Unable to check OS${COL_NC}"
+    else
+        IFS=" " read -r -a supportedOS < <(echo "${response}" | tr -d '"')
+        for distro_and_versions in "${supportedOS[@]}"
+        do
+            distro_part="${distro_and_versions%%=*}"
+            versions_part="${distro_and_versions##*=}"
+
+            if [[ "${detected_os^^}" =~ ${distro_part^^} ]]; then
+                valid_os=true
+                IFS="," read -r -a supportedVer <<<"${versions_part}"
+                for version in "${supportedVer[@]}"
+                do
+                    if [[ "${detected_version}" =~ $version ]]; then
+                        valid_version=true
+                        break
+                    fi
+                done
+                break
+            fi
+        done
+
+        local finalmsg
+        if [ "$valid_os" = true ]; then
+            log_write "${TICK} Distro:  ${COL_GREEN}${detected_os^}${COL_NC}"
+
+            if [ "$valid_version" = true ]; then
+                log_write "${TICK} Version: ${COL_GREEN}${detected_version}${COL_NC}"
+                finalmsg="${TICK} ${COL_GREEN}Distro and version supported${COL_NC}"
+            else
+                log_write "${CROSS} Version: ${COL_RED}${detected_version}${COL_NC}"
+                finalmsg="${CROSS} Error: ${COL_RED}${detected_os^} is supported but version ${detected_version} is currently unsupported ${COL_NC}(${FAQ_HARDWARE_REQUIREMENTS})${COL_NC}"
+            fi
+        else
+            log_write "${CROSS} Distro:  ${COL_RED}${detected_os^}${COL_NC}"
+            finalmsg="${CROSS} Error: ${COL_RED}${detected_os^} is not a supported distro ${COL_NC}(${FAQ_HARDWARE_REQUIREMENTS})${COL_NC}"
+        fi
+
+        # Print dig response and the final check result
+        log_write "${TICK} dig return code: ${COL_GREEN}${digReturnCode}${COL_NC}"
+        log_write "${INFO} dig response: ${response}"
+        log_write "${finalmsg}"
+    fi
+}
+
 diagnose_operating_system() {
    # error message in a variable so we can easily modify it later (or reuse it)
    local error_msg="Distribution unknown -- most likely you are on an unsupported platform and may run into issues."
-    local detected_os
-    local detected_version
-
    # Display the current test that is running
    echo_current_diagnostic "Operating system"

@@ -310,13 +380,8 @@ diagnose_operating_system() {

    # If there is a /etc/*release file, it's probably a supported operating system, so we can
    if ls /etc/*release 1> /dev/null 2>&1; then
-        # display the attributes to the user
-
-        detected_os=$(grep "\bID\b" /etc/os-release | cut -d '=' -f2 | tr -d '"')
-        detected_version=$(grep VERSION_ID /etc/os-release | cut -d '=' -f2 | tr -d '"')
-
-        log_write "${INFO} Distro: ${detected_os^}"
-        log_write "${INFO} Version: ${detected_version}"
+        # display the attributes to the user from the function made earlier
+        os_check
    else
        # If it doesn't exist, it's not a system we currently support and link to FAQ
        log_write "${CROSS} ${COL_RED}${error_msg}${COL_NC} (${FAQ_HARDWARE_REQUIREMENTS})"
@@ -414,9 +479,7 @@ run_and_print_command() {
    local output
    output=$(${cmd} 2>&1)
    # If the command was successful,
-    local return_code
-    return_code=$?
-    if [[ "${return_code}" -eq 0 ]]; then
+    if [[ $? -eq 0 ]]; then
        # show the output
        log_write "${output}"
    else
@@ -426,25 +489,13 @@ run_and_print_command() {
 }

 hardware_check() {
-    # Note: the checks are skipped if Pi-hole is running in a docker container
-
-    local skip_msg="${INFO} Not enough permissions inside Docker container ${COL_YELLOW}(skipped)${COL_NC}"
-
    echo_current_diagnostic "System hardware configuration"
-    if [ -n "${DOCKER_VERSION}" ]; then
-        log_write "${skip_msg}"
-    else
-        # Store the output of the command in a variable
-        run_and_print_command "lshw -short"
-    fi
+    # Store the output of the command in a variable
+    run_and_print_command "lshw -short"

    echo_current_diagnostic "Processor details"
-    if [ -n "${DOCKER_VERSION}" ]; then
-        log_write "${skip_msg}"
-    else
-        # Store the output of the command in a variable
-        run_and_print_command "lscpu"
-    fi
+    # Store the output of the command in a variable
+    run_and_print_command "lscpu"
 }

 disk_usage() {
@@ -757,24 +808,26 @@ dig_at() {
 process_status(){
    # Check to make sure Pi-hole's services are running and active
    echo_current_diagnostic "Pi-hole processes"
-
    # Local iterator
    local i
-
    # For each process,
    for i in "${PIHOLE_PROCESSES[@]}"; do
-        local status_of_process
-
        # If systemd
        if command -v systemctl &> /dev/null; then
            # get its status via systemctl
+            local status_of_process
            status_of_process=$(systemctl is-active "${i}")
        else
            # Otherwise, use the service command and mock the output of `systemctl is-active`
+            local status_of_process

-            # If it is a docker container, there is no systemctl or service. Do nothing.
+            # If DOCKER_VERSION is set, the output is slightly different (s6 init system on Docker)
            if [ -n "${DOCKER_VERSION}" ]; then
-                :
+                if service "${i}" status | grep -E '^up' &> /dev/null; then
+                    status_of_process="active"
+                else
+                    status_of_process="inactive"
+                fi
            else
            # non-Docker system
                if service "${i}" status | grep -E 'is\srunning' &> /dev/null; then
@@ -784,12 +837,8 @@ process_status(){
                fi
            fi
        fi
-
        # and print it out to the user
-        if [ -n "${DOCKER_VERSION}" ]; then
-            # If it's a Docker container, the test was skipped
-            log_write "${INFO} systemctl/service not installed inside docker container ${COL_YELLOW}(skipped)${COL_NC}"
-        elif [[ "${status_of_process}" == "active" ]]; then
+        if [[ "${status_of_process}" == "active" ]]; then
            # If it's active, show it in green
            log_write "${TICK} ${COL_GREEN}${i}${COL_NC} daemon is ${COL_GREEN}${status_of_process}${COL_NC}"
        else
@@ -806,8 +855,6 @@ ftl_full_status(){
    if command -v systemctl &> /dev/null; then
      FTL_status=$(systemctl status --full --no-pager pihole-FTL.service)
      log_write "   ${FTL_status}"
-    elif [ -n "${DOCKER_VERSION}" ]; then
-      log_write "${INFO} systemctl/service not installed inside docker container ${COL_YELLOW}(skipped)${COL_NC}"
    else
      log_write "${INFO} systemctl:  command not found"
    fi
@@ -861,6 +908,7 @@ parse_file() {
    # Get the lines that are in the file(s) and store them in an array for parsing later
    local file_info
    if [[ -f "$filename" ]]; then
+        #shellcheck disable=SC2016
        IFS=$'\r\n' command eval 'file_info=( $(cat "${filename}") )'
    else
        read -r -a file_info <<< "$filename"
@@ -1064,7 +1112,7 @@ show_FTL_db_entries() {
 }

 check_dhcp_servers() {
-    echo_current_diagnostic "Discovering active DHCP servers (takes 6 seconds)"
+    echo_current_diagnostic "Discovering active DHCP servers (takes 10 seconds)"

    OLD_IFS="$IFS"
    IFS=$'\n'
@@ -1148,7 +1196,7 @@ database_integrity_check(){
    local database="${1}"

    log_write "${INFO} Checking integrity of ${database} ... (this can take several minutes)"
-    result="$(pihole-FTL sqlite3 -ni "${database}" "PRAGMA integrity_check" 2>&1 & spinner)"
+    result="$(pihole-FTL "${database}" "PRAGMA integrity_check" 2>&1 & spinner)"
    if [[ ${result} = "ok" ]]; then
      log_write "${TICK} Integrity of ${database} intact"

@@ -1269,16 +1317,19 @@ upload_to_tricorder() {
        curl_to_tricorder
        # If we're not running in automated mode,
    else
-        echo ""
-        # give the user a choice of uploading it or not
-        # Users can review the log file locally (or the output of the script since they are the same) and try to self-diagnose their problem
-        read -r -p "[?] Would you like to upload the log? [y/N] " response
-        case ${response} in
-            # If they say yes, run our function for uploading the log
-            [yY][eE][sS]|[yY]) curl_to_tricorder;;
-            # If they choose no, just exit out of the script
-            *) log_write "    * Log will ${COL_GREEN}NOT${COL_NC} be uploaded to tricorder.\\n    * A local copy of the debug log can be found at: ${COL_CYAN}${PIHOLE_DEBUG_LOG}${COL_NC}\\n";exit;
-        esac
+        # if not being called from the web interface
+        if [[ ! "${WEBCALL}" ]]; then
+            echo ""
+            # give the user a choice of uploading it or not
+            # Users can review the log file locally (or the output of the script since they are the same) and try to self-diagnose their problem
+            read -r -p "[?] Would you like to upload the log? [y/N] " response
+            case ${response} in
+                # If they say yes, run our function for uploading the log
+                [yY][eE][sS]|[yY]) curl_to_tricorder;;
+                # If they choose no, just exit out of the script
+                *) log_write "    * Log will ${COL_GREEN}NOT${COL_NC} be uploaded to tricorder.\\n    * A local copy of the debug log can be found at: ${COL_CYAN}${PIHOLE_DEBUG_LOG}${COL_NC}\\n";exit;
+            esac
+        fi
    fi
    # Check if tricorder.pi-hole.net is reachable and provide token
    # along with some additional useful information
@@ -1298,8 +1349,13 @@ upload_to_tricorder() {
    # If no token was generated
    else
        # Show an error and some help instructions
-        log_write "${CROSS} ${COL_RED}There was an error uploading your debug log.${COL_NC}"
-        log_write "   * Please try again or contact the Pi-hole team for assistance."
+        # Skip this if being called from web interface and automatic mode was not chosen (users opt-out to upload)
+        if [[ "${WEBCALL}" ]] && [[ ! "${AUTOMATED}" ]]; then
+            :
+        else
+            log_write "${CROSS} ${COL_RED}There was an error uploading your debug log.${COL_NC}"
+            log_write "   * Please try again or contact the Pi-hole team for assistance."
+        fi
    fi
    # Finally, show where the log file is no matter the outcome of the function so users can look at it
    log_write "   * A local copy of the debug log can be found at: ${COL_CYAN}${PIHOLE_DEBUG_LOG}${COL_NC}\\n"
--- a/advanced/Scripts/piholeLogFlush.sh
+++ b/advanced/Scripts/piholeLogFlush.sh
@@ -9,12 +9,10 @@
 # Please see LICENSE file for your rights under this license.

 colfile="/opt/pihole/COL_TABLE"
-# shellcheck source="./advanced/Scripts/COL_TABLE"
 source ${colfile}

 readonly PI_HOLE_SCRIPT_DIR="/opt/pihole"
 utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source="./advanced/Scripts/utils.sh"
 source "${utilsfile}"

 # In case we're running at the same time as a system logrotate, use a
@@ -37,46 +35,6 @@ FTLFILE=$(getFTLConfigValue "files.log.ftl")
 if [ -z "$FTLFILE" ]; then
    FTLFILE="/var/log/pihole/FTL.log"
 fi
-WEBFILE=$(getFTLConfigValue "files.log.webserver")
-if [ -z "$WEBFILE" ]; then
-    WEBFILE="/var/log/pihole/webserver.log"
-fi
-
-# Helper function to handle log rotation for a single file
-rotate_log() {
-    # This function copies x.log over to x.log.1
-    # and then empties x.log
-    # Note that moving the file is not an option, as
-    # dnsmasq would happily continue writing into the
-    # moved file (it will have the same file handler)
-    local logfile="$1"
-    if [[ "$*" != *"quiet"* ]]; then
-        echo -ne "  ${INFO} Rotating ${logfile} ..."
-    fi
-    cp -p "${logfile}" "${logfile}.1"
-    echo " " > "${logfile}"
-    chmod 640 "${logfile}"
-    if [[ "$*" != *"quiet"* ]]; then
-        echo -e "${OVER}  ${TICK} Rotated ${logfile} ..."
-    fi
-}
-
-# Helper function to handle log flushing for a single file
-flush_log() {
-    local logfile="$1"
-    if [[ "$*" != *"quiet"* ]]; then
-        echo -ne "  ${INFO} Flushing ${logfile} ..."
-    fi
-    echo " " > "${logfile}"
-    chmod 640 "${logfile}"
-    if [ -f "${logfile}.1" ]; then
-        echo " " > "${logfile}.1"
-        chmod 640 "${logfile}.1"
-    fi
-    if [[ "$*" != *"quiet"* ]]; then
-        echo -e "${OVER}  ${TICK} Flushed ${logfile} ..."
-    fi
-}

 if [[ "$*" == *"once"* ]]; then
    # Nightly logrotation
@@ -88,16 +46,62 @@ if [[ "$*" == *"once"* ]]; then
        fi
        /usr/sbin/logrotate --force --state "${STATEFILE}" /etc/pihole/logrotate
    else
-        # Handle rotation for each log file
-        rotate_log "${LOGFILE}"
-        rotate_log "${FTLFILE}"
-        rotate_log "${WEBFILE}"
+        # Copy pihole.log over to pihole.log.1
+        # and empty out pihole.log
+        # Note that moving the file is not an option, as
+        # dnsmasq would happily continue writing into the
+        # moved file (it will have the same file handler)
+        if [[ "$*" != *"quiet"* ]]; then
+            echo -ne "  ${INFO} Rotating ${LOGFILE} ..."
+        fi
+        cp -p "${LOGFILE}" "${LOGFILE}.1"
+        echo " " > "${LOGFILE}"
+        chmod 640 "${LOGFILE}"
+        if [[ "$*" != *"quiet"* ]]; then
+            echo -e "${OVER}  ${TICK} Rotated ${LOGFILE} ..."
+        fi
+        # Copy FTL.log over to FTL.log.1
+        # and empty out FTL.log
+        if [[ "$*" != *"quiet"* ]]; then
+            echo -ne "  ${INFO} Rotating ${FTLFILE} ..."
+        fi
+        cp -p "${FTLFILE}" "${FTLFILE}.1"
+        echo " " > "${FTLFILE}"
+        chmod 640 "${FTLFILE}"
+        if [[ "$*" != *"quiet"* ]]; then
+            echo -e "${OVER}  ${TICK} Rotated ${FTLFILE} ..."
+        fi
    fi
 else
    # Manual flushing
-    flush_log "${LOGFILE}"
-    flush_log "${FTLFILE}"
-    flush_log "${WEBFILE}"
+
+    # Flush both pihole.log and pihole.log.1 (if existing)
+    if [[ "$*" != *"quiet"* ]]; then
+        echo -ne "  ${INFO} Flushing ${LOGFILE} ..."
+    fi
+    echo " " > "${LOGFILE}"
+    chmod 640 "${LOGFILE}"
+    if [ -f "${LOGFILE}.1" ]; then
+        echo " " > "${LOGFILE}.1"
+        chmod 640 "${LOGFILE}.1"
+    fi
+    if [[ "$*" != *"quiet"* ]]; then
+        echo -e "${OVER}  ${TICK} Flushed ${LOGFILE} ..."
+    fi
+
+    # Flush both FTL.log and FTL.log.1 (if existing)
+    if [[ "$*" != *"quiet"* ]]; then
+        echo -ne "  ${INFO} Flushing ${FTLFILE} ..."
+    fi
+    echo " " > "${FTLFILE}"
+    chmod 640 "${FTLFILE}"
+    if [ -f "${FTLFILE}.1" ]; then
+        echo " " > "${FTLFILE}.1"
+        chmod 640 "${FTLFILE}.1"
+    fi
+    if [[ "$*" != *"quiet"* ]]; then
+        echo -e "${OVER}  ${TICK} Flushed ${FTLFILE} ..."
+    fi

    if [[ "$*" != *"quiet"* ]]; then
        echo -ne "  ${INFO} Flushing database, DNS resolution temporarily unavailable ..."
--- a/advanced/Scripts/query.sh
+++ b/advanced/Scripts/query.sh
@@ -1,4 +1,9 @@
 #!/usr/bin/env sh
+# shellcheck disable=SC1090
+
+# Ignore warning about `local` being undefinded in POSIX
+# shellcheck disable=SC3043
+# https://github.com/koalaman/shellcheck/wiki/SC3043#exceptions

 # Pi-hole: A black hole for Internet advertisements
 # (c) 2023 Pi-hole, LLC (https://pi-hole.net)
@@ -17,11 +22,9 @@ domain=""

 # Source color table
 colfile="/opt/pihole/COL_TABLE"
-# shellcheck source="./advanced/Scripts/COL_TABLE"
 . "${colfile}"

 # Source api functions
-# shellcheck source="./advanced/Scripts/api.sh"
 . "${PI_HOLE_INSTALL_DIR}/api.sh"

 Help() {
--- a/advanced/Scripts/update.sh
+++ b/advanced/Scripts/update.sh
@@ -12,31 +12,26 @@

 # Variables
 readonly ADMIN_INTERFACE_GIT_URL="https://github.com/pi-hole/web.git"
+readonly ADMIN_INTERFACE_DIR="/var/www/html/admin"
 readonly PI_HOLE_GIT_URL="https://github.com/pi-hole/pi-hole.git"
 readonly PI_HOLE_FILES_DIR="/etc/.pihole"

+# shellcheck disable=SC2034
 SKIP_INSTALL=true

 # when --check-only is passed to this script, it will not perform the actual update
 CHECK_ONLY=false

-# shellcheck source="./automated install/basic-install.sh"
+# shellcheck disable=SC1090
 source "${PI_HOLE_FILES_DIR}/automated install/basic-install.sh"
-# shellcheck source=./advanced/Scripts/COL_TABLE
+# shellcheck disable=SC1091
 source "/opt/pihole/COL_TABLE"
-# shellcheck source="./advanced/Scripts/utils.sh"
-source "${PI_HOLE_INSTALL_DIR}/utils.sh"

 # is_repo() sourced from basic-install.sh
 # make_repo() sourced from basic-install.sh
 # update_repo() source from basic-install.sh
 # getGitFiles() sourced from basic-install.sh
 # FTLcheckUpdate() sourced from basic-install.sh
-# getFTLConfigValue() sourced from utils.sh
-
-# Honour configured paths for the web application.
-ADMIN_INTERFACE_DIR=$(getFTLConfigValue "webserver.paths.webroot")$(getFTLConfigValue "webserver.paths.webhome")
-readonly ADMIN_INTERFACE_DIR

 GitCheckUpdateAvail() {
    local directory
@@ -112,7 +107,6 @@ main() {
    web_update=false
    FTL_update=false

-
    # Install packages used by this installation script (necessary if users have removed e.g. git from their systems)
    package_manager_detect
    build_dependency_package
@@ -212,7 +206,7 @@ main() {
        echo ""
        echo -e "  ${INFO} Pi-hole Web Admin files out of date, updating local repo."
        getGitFiles "${ADMIN_INTERFACE_DIR}" "${ADMIN_INTERFACE_GIT_URL}"
-        echo -e "  ${INFO} If you had made any changes in '${ADMIN_INTERFACE_DIR}', they have been stashed using 'git stash'"
+        echo -e "  ${INFO} If you had made any changes in '/var/www/html/admin/', they have been stashed using 'git stash'"
    fi

    if [[ "${FTL_update}" == true ]]; then
@@ -221,7 +215,7 @@ main() {
    fi

    if [[ "${FTL_update}" == true || "${core_update}" == true ]]; then
-        ${PI_HOLE_FILES_DIR}/automated\ install/basic-install.sh --repair --unattended || \
+        ${PI_HOLE_FILES_DIR}/automated\ install/basic-install.sh --reconfigure --unattended || \
            echo -e "${basicError}" && exit 1
    fi

--- a/advanced/Scripts/updatecheck.sh
+++ b/advanced/Scripts/updatecheck.sh
@@ -39,12 +39,9 @@ function get_remote_hash() {
 }

 # Source the utils file for addOrEditKeyValPair()
-# shellcheck source="./advanced/Scripts/utils.sh"
+# shellcheck disable=SC1091
 . /opt/pihole/utils.sh

-ADMIN_INTERFACE_DIR=$(getFTLConfigValue "webserver.paths.webroot")$(getFTLConfigValue "webserver.paths.webhome")
-readonly ADMIN_INTERFACE_DIR
-
 # Remove the below three legacy files if they exist
 rm -f "/etc/pihole/GitHubVersions"
 rm -f "/etc/pihole/localbranches"
@@ -88,13 +85,13 @@ addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_CORE_HASH" "${GITHUB_CORE_HASH}"

 # get Web versions

-WEB_VERSION="$(get_local_version "${ADMIN_INTERFACE_DIR}")"
+WEB_VERSION="$(get_local_version /var/www/html/admin)"
 addOrEditKeyValPair "${VERSION_FILE}" "WEB_VERSION" "${WEB_VERSION}"

-WEB_BRANCH="$(get_local_branch "${ADMIN_INTERFACE_DIR}")"
+WEB_BRANCH="$(get_local_branch /var/www/html/admin)"
 addOrEditKeyValPair "${VERSION_FILE}" "WEB_BRANCH" "${WEB_BRANCH}"

-WEB_HASH="$(get_local_hash "${ADMIN_INTERFACE_DIR}")"
+WEB_HASH="$(get_local_hash /var/www/html/admin)"
 addOrEditKeyValPair "${VERSION_FILE}" "WEB_HASH" "${WEB_HASH}"

 GITHUB_WEB_VERSION="$(get_remote_version web "${WEB_BRANCH}")"
--- a/advanced/Scripts/utils.sh
+++ b/advanced/Scripts/utils.sh
@@ -1,4 +1,5 @@
 #!/usr/bin/env sh
+# shellcheck disable=SC3043 #https://github.com/koalaman/shellcheck/wiki/SC3043#exceptions

 # Pi-hole: A black hole for Internet advertisements
 # (c) 2017 Pi-hole, LLC (https://pi-hole.net)
@@ -87,8 +88,8 @@ getFTLConfigValue(){
 #######################
 setFTLConfigValue(){
  pihole-FTL --config "${1}" "${2}" >/dev/null
-  if [ $? -eq 5 ]; then
-    printf "  %s %s set by environment variable. Please unset it to use this function\n" "${CROSS}" "${1}"
+  if [[ $? -eq 5 ]]; then
+    echo -e "  ${CROSS} ${1} set by environment variable. Please unset it to use this function"
    exit 5
  fi
 }
--- a/advanced/Scripts/version.sh
+++ b/advanced/Scripts/version.sh
@@ -8,16 +8,20 @@
 # This file is copyright under the latest version of the EUPL.
 # Please see LICENSE file for your rights under this license.

-# Source the versions file populated by updatechecker.sh
+# Ignore warning about `local` being undefinded in POSIX
+# shellcheck disable=SC3043
+# https://github.com/koalaman/shellcheck/wiki/SC3043#exceptions
+
+# Source the versions file poupulated by updatechecker.sh
 cachedVersions="/etc/pihole/versions"

 if [ -f ${cachedVersions} ]; then
-    # shellcheck source=/dev/null
+    # shellcheck disable=SC1090
    . "$cachedVersions"
 else
    echo "Could not find /etc/pihole/versions. Running update now."
    pihole updatechecker
-     # shellcheck source=/dev/null
+    # shellcheck disable=SC1090
    . "$cachedVersions"
 fi

--- a/advanced/Templates/gravity.db.sql
+++ b/advanced/Templates/gravity.db.sql
@@ -43,8 +43,8 @@ CREATE TABLE adlist

 CREATE TABLE adlist_by_group
 (
-    adlist_id INTEGER NOT NULL REFERENCES adlist (id) ON DELETE CASCADE,
-    group_id INTEGER NOT NULL REFERENCES "group" (id) ON DELETE CASCADE,
+    adlist_id INTEGER NOT NULL REFERENCES adlist (id),
+    group_id INTEGER NOT NULL REFERENCES "group" (id),
    PRIMARY KEY (adlist_id, group_id)
 );

@@ -67,16 +67,11 @@ CREATE TABLE info
 );

 INSERT INTO "info" VALUES('version','19');
-/* This is a flag to indicate if gravity was restored from a backup
-    false = not restored,
-    failed = restoration failed due to no backup
-    other string = restoration successful with the string being the backup file used */
-INSERT INTO "info" VALUES('gravity_restored','false');

 CREATE TABLE domainlist_by_group
 (
-    domainlist_id INTEGER NOT NULL REFERENCES domainlist (id) ON DELETE CASCADE,
-    group_id INTEGER NOT NULL REFERENCES "group" (id) ON DELETE CASCADE,
+    domainlist_id INTEGER NOT NULL REFERENCES domainlist (id),
+    group_id INTEGER NOT NULL REFERENCES "group" (id),
    PRIMARY KEY (domainlist_id, group_id)
 );

@@ -91,8 +86,8 @@ CREATE TABLE client

 CREATE TABLE client_by_group
 (
-    client_id INTEGER NOT NULL REFERENCES client (id) ON DELETE CASCADE,
-    group_id INTEGER NOT NULL REFERENCES "group" (id) ON DELETE CASCADE,
+    client_id INTEGER NOT NULL REFERENCES client (id),
+    group_id INTEGER NOT NULL REFERENCES "group" (id),
    PRIMARY KEY (client_id, group_id)
 );

--- a/advanced/Templates/pihole-FTL-poststop.sh
+++ b/advanced/Templates/pihole-FTL-poststop.sh
@@ -3,7 +3,7 @@
 # Source utils.sh for getFTLConfigValue()
 PI_HOLE_SCRIPT_DIR='/opt/pihole'
 utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source="./advanced/Scripts/utils.sh"
+# shellcheck disable=SC1090
 . "${utilsfile}"

 # Get file paths
--- a/advanced/Templates/pihole-FTL-prestart.sh
+++ b/advanced/Templates/pihole-FTL-prestart.sh
@@ -3,32 +3,32 @@
 # Source utils.sh for getFTLConfigValue()
 PI_HOLE_SCRIPT_DIR='/opt/pihole'
 utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source="./advanced/Scripts/utils.sh"
+# shellcheck disable=SC1090
 . "${utilsfile}"

 # Get file paths
 FTL_PID_FILE="$(getFTLConfigValue files.pid)"

 # Ensure that permissions are set so that pihole-FTL can edit all necessary files
-mkdir -p /var/log/pihole
-chown -R pihole:pihole /etc/pihole/ /var/log/pihole/
+# shellcheck disable=SC2174
+mkdir -pm 0640 /var/log/pihole
+chown -R pihole:pihole /etc/pihole /var/log/pihole
+chmod -R 0640 /var/log/pihole
+chmod -R 0660 /etc/pihole

-# allow all users read version file (and use pihole -v)
-chmod 0644 /etc/pihole/versions
+# Logrotate config file need to be owned by root and must not be writable by group and others
+chown root:root /etc/pihole/logrotate
+chmod 0644 /etc/pihole/logrotate
+
+# allow all users to enter the directories
+chmod 0755 /etc/pihole /var/log/pihole

 # allow pihole to access subdirs in /etc/pihole (sets execution bit on dirs)
-find /etc/pihole/ /var/log/pihole/ -type d -exec chmod 0755 {} +
-# Set all files (except TLS-related ones) to u+rw g+r
-find /etc/pihole/ /var/log/pihole/ -type f ! \( -name '*.pem' -o -name '*.crt' \) -exec chmod 0640 {} +
-# Set TLS-related files to a more restrictive u+rw *only* (they may contain private keys)
-find /etc/pihole/ -type f \( -name '*.pem' -o -name '*.crt' \) -exec chmod 0600 {} +
-
-# Logrotate config file need to be owned by root
-chown root:root /etc/pihole/logrotate
+# credits https://stackoverflow.com/a/11512211
+find /etc/pihole -type d -exec chmod 0755 {} \;

 # Touch files to ensure they exist (create if non-existing, preserve if existing)
 [ -f "${FTL_PID_FILE}" ] || install -D -m 644 -o pihole -g pihole /dev/null "${FTL_PID_FILE}"
 [ -f /var/log/pihole/FTL.log ] || install -m 640 -o pihole -g pihole /dev/null /var/log/pihole/FTL.log
 [ -f /var/log/pihole/pihole.log ] || install -m 640 -o pihole -g pihole /dev/null /var/log/pihole/pihole.log
-[ -f /var/log/pihole/webserver.log ] || install -m 640 -o pihole -g pihole /dev/null /var/log/pihole/webserver.log
 [ -f /etc/pihole/dhcp.leases ] || install -m 644 -o pihole -g pihole /dev/null /etc/pihole/dhcp.leases
--- a/advanced/Templates/pihole-FTL.service
+++ b/advanced/Templates/pihole-FTL.service
@@ -12,7 +12,7 @@
 # Source utils.sh for getFTLConfigValue(), getFTLPID()
 PI_HOLE_SCRIPT_DIR="/opt/pihole"
 utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source="./advanced/Scripts/utils.sh"
+# shellcheck disable=SC1090
 . "${utilsfile}"


@@ -57,16 +57,13 @@ start() {
 stop() {
  if is_running; then
    kill "${FTL_PID}"
-    # Give FTL 60 seconds to gracefully stop
-    i=1
-    while [ "${i}" -le 60 ]; do
+    for i in 1 2 3 4 5; do
      if ! is_running; then
        break
      fi

      printf "."
      sleep 1
-      i=$((i + 1))
    done
    echo

--- a/advanced/Templates/pihole-FTL.systemd
+++ b/advanced/Templates/pihole-FTL.systemd
@@ -28,7 +28,7 @@ ExecReload=/bin/kill -HUP $MAINPID
 ExecStopPost=/opt/pihole/pihole-FTL-poststop.sh

 # Use graceful shutdown with a reasonable timeout
-TimeoutStopSec=60s
+TimeoutStopSec=10s

 # Make /usr, /boot, /etc and possibly some more folders read-only...
 ProtectSystem=full
--- a/advanced/bash-completion/pihole
+++ b/advanced/bash-completion/pihole
@@ -7,7 +7,7 @@ _pihole() {

    case "${prev}" in
        "pihole")
-            opts="allow allow-regex allow-wild deny checkout debug disable enable flush help logging query repair regex reloaddns reloadlists status tail uninstall updateGravity updatePihole version wildcard arpflush api"
+            opts="allow allow-regex allow-wild deny checkout debug disable enable flush help logging query reconfigure regex reloaddns reloadlists status tail uninstall updateGravity updatePihole version wildcard arpflush api"
            COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
        ;;
        "allow"|"deny"|"wildcard"|"regex"|"allow-regex"|"allow-wild")
--- a/install/basic-install.sh
+++ b/install/basic-install.sh
--- a/install/uninstall.sh
+++ b/install/uninstall.sh
@@ -8,13 +8,7 @@
 # This file is copyright under the latest version of the EUPL.
 # Please see LICENSE file for your rights under this license.

-# shellcheck source="./advanced/Scripts/COL_TABLE"
 source "/opt/pihole/COL_TABLE"
-# shellcheck source="./advanced/Scripts/utils.sh"
-source "/opt/pihole/utils.sh"
-
-ADMIN_INTERFACE_DIR=$(getFTLConfigValue "webserver.paths.webroot")$(getFTLConfigValue "webserver.paths.webhome")
-readonly ADMIN_INTERFACE_DIR

 while true; do
    read -rp "  ${QST} Are you sure you would like to remove ${COL_WHITE}Pi-hole${COL_NC}? [y/N] " answer
@@ -43,7 +37,6 @@ fi

 readonly PI_HOLE_FILES_DIR="/etc/.pihole"
 SKIP_INSTALL="true"
-# shellcheck source="./automated install/basic-install.sh"
 source "${PI_HOLE_FILES_DIR}/automated install/basic-install.sh"

 # package_manager_detect() sourced from basic-install.sh
@@ -60,9 +53,17 @@ removeMetaPackage() {
 }

 removePiholeFiles() {
-    # Remove the web interface of Pi-hole
+    # Only web directories/files that are created by Pi-hole should be removed
    echo -ne "  ${INFO} Removing Web Interface..."
-    ${SUDO} rm -rf "${ADMIN_INTERFACE_DIR}" &> /dev/null
+    ${SUDO} rm -rf /var/www/html/admin &> /dev/null
+
+
+    # If the web directory is empty after removing these files, then the parent html directory can be removed.
+    if [ -d "/var/www/html" ]; then
+        if [[ ! "$(ls -A /var/www/html)" ]]; then
+            ${SUDO} rm -rf /var/www/html &> /dev/null
+        fi
+    fi
    echo -e "${OVER}  ${TICK} Removed Web Interface"

    # Attempt to preserve backwards compatibility with older versions
@@ -93,9 +94,8 @@ removePiholeFiles() {
    echo -e "  ${TICK} Removed config files"

    # Restore Resolved
-    if [[ -e /etc/systemd/resolved.conf.orig ]] || [[ -e /etc/systemd/resolved.conf.d/90-pi-hole-disable-stub-listener.conf ]]; then
-        ${SUDO} cp -p /etc/systemd/resolved.conf.orig /etc/systemd/resolved.conf &> /dev/null || true
-        ${SUDO} rm -f /etc/systemd/resolved.conf.d/90-pi-hole-disable-stub-listener.conf
+    if [[ -e /etc/systemd/resolved.conf.orig ]]; then
+        ${SUDO} cp -p /etc/systemd/resolved.conf.orig /etc/systemd/resolved.conf
        systemctl reload-or-restart systemd-resolved
    fi

--- a/gravity.sh
+++ b/gravity.sh
@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+# shellcheck disable=SC1090

 # Pi-hole: A black hole for Internet advertisements
 # (c) 2017 Pi-hole, LLC (https://pi-hole.net)
@@ -15,13 +16,13 @@ export LC_ALL=C
 PI_HOLE_SCRIPT_DIR="/opt/pihole"
 # Source utils.sh for GetFTLConfigValue
 utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source=./advanced/Scripts/utils.sh
+# shellcheck disable=SC1090
 . "${utilsfile}"

 coltable="${PI_HOLE_SCRIPT_DIR}/COL_TABLE"
-# shellcheck source=./advanced/Scripts/COL_TABLE
+# shellcheck disable=SC1090
 . "${coltable}"
-# shellcheck source=./advanced/Scripts/database_migration/gravity-db.sh
+# shellcheck disable=SC1091
 . "/etc/.pihole/advanced/Scripts/database_migration/gravity-db.sh"

 basename="pihole"
@@ -29,9 +30,6 @@ PIHOLE_COMMAND="/usr/local/bin/${basename}"

 piholeDir="/etc/${basename}"

-# Gravity aux files directory
-listsCacheDir="${piholeDir}/listsCache"
-
 # Legacy (pre v5.0) list file locations
 whitelistFile="${piholeDir}/whitelist.txt"
 blacklistFile="${piholeDir}/blacklist.txt"
@@ -46,7 +44,6 @@ gravityDBcopy="${piholeGitDir}/advanced/Templates/gravity_copy.sql"

 domainsExtension="domains"
 curl_connect_timeout=10
-etag_support=false

 # Check gravity temp directory
 if [ ! -d "${GRAVITY_TMPDIR}" ] || [ ! -w "${GRAVITY_TMPDIR}" ]; then
@@ -57,12 +54,10 @@ fi
 # Set this only after sourcing pihole-FTL.conf as the gravity database path may
 # have changed
 gravityDBfile="${GRAVITYDB}"
-gravityDBfile_default="${piholeDir}/gravity.db"
+gravityDBfile_default="/etc/pihole/gravity.db"
 gravityTEMPfile="${GRAVITYDB}_temp"
 gravityDIR="$(dirname -- "${gravityDBfile}")"
 gravityOLDfile="${gravityDIR}/gravity_old.db"
-gravityBCKdir="${gravityDIR}/gravity_backups"
-gravityBCKfile="${gravityBCKdir}/gravity.db"

 fix_owner_permissions() {
  # Fix ownership and permissions for the specified file
@@ -96,21 +91,11 @@ gravity_build_tree() {

  if [[ "${status}" -ne 0 ]]; then
    echo -e "\\n  ${CROSS} Unable to build gravity tree in ${gravityTEMPfile}\\n  ${output}"
-    echo -e "  ${INFO} If you have a large amount of domains, make sure your Pi-hole has enough RAM available\\n"
    return 1
  fi
  echo -e "${OVER}  ${TICK} ${str}"
 }

-# Rotate gravity backup files
-rotate_gravity_backup() {
-  for i in {9..1}; do
-    if [ -f "${gravityBCKfile}.${i}" ]; then
-      mv "${gravityBCKfile}.${i}" "${gravityBCKfile}.$((i + 1))"
-    fi
-  done
-}
-
 # Copy data from old to new database file and swap them
 gravity_swap_databases() {
  str="Swapping databases"
@@ -126,32 +111,10 @@ gravity_swap_databases() {
  oldAvail=false
  if [ "${availableBlocks}" -gt "$((gravityBlocks * 2))" ] && [ -f "${gravityDBfile}" ]; then
    oldAvail=true
-    cp -p "${gravityDBfile}" "${gravityOLDfile}"
-  fi
-
-  # Drop the gravity and antigravity tables + subsequent VACUUM the current
-  # database for compaction
-  output=$({ printf ".timeout 30000\\nDROP TABLE IF EXISTS gravity;\\nDROP TABLE IF EXISTS antigravity;\\nVACUUM;\\n" | pihole-FTL sqlite3 -ni "${gravityDBfile}"; } 2>&1)
-  status="$?"
-
-  if [[ "${status}" -ne 0 ]]; then
-    echo -e "\\n  ${CROSS} Unable to clean current database for backup\\n  ${output}"
+    mv "${gravityDBfile}" "${gravityOLDfile}"
  else
-    # Check if the backup directory exists
-    if [ ! -d "${gravityBCKdir}" ]; then
-      mkdir -p "${gravityBCKdir}" && chown pihole:pihole "${gravityBCKdir}"
-    fi
-
-    # If multiple gravityBCKfile's are present (appended with a number), rotate them
-    # We keep at most 10 backups
-    rotate_gravity_backup
-
-    # Move the old database to the backup location
-    mv "${gravityDBfile}" "${gravityBCKfile}.1"
+    rm "${gravityDBfile}"
  fi
-
-
-  # Move the new database to the correct location
  mv "${gravityTEMPfile}" "${gravityDBfile}"
  echo -e "${OVER}  ${TICK} ${str}"

@@ -305,7 +268,7 @@ migrate_to_database() {
    fi

    # Check if gravity database needs to be updated
-    upgrade_gravityDB "${gravityDBfile}"
+    upgrade_gravityDB "${gravityDBfile}" "${piholeDir}"

    # Migrate list files to new database
    if [ -e "${adListFile}" ]; then
@@ -333,7 +296,7 @@ migrate_to_database() {
  fi

  # Check if gravity database needs to be updated
-  upgrade_gravityDB "${gravityDBfile}"
+  upgrade_gravityDB "${gravityDBfile}" "${piholeDir}"
 }

 # Determine if DNS resolution is available before proceeding
@@ -348,72 +311,17 @@ gravity_CheckDNSResolutionAvailable() {
    echo -e "  ${CROSS} DNS resolution is currently unavailable"
  fi

-  str="Waiting up to 120 seconds for DNS resolution..."
+  str="Waiting until DNS resolution is available..."
  echo -ne "  ${INFO} ${str}"
-
- # Default DNS timeout is two seconds, plus 1 second for each dot > 120 seconds
-  for ((i = 0; i < 40; i++)); do
-      if getent hosts github.com &> /dev/null; then
-        # If we reach this point, DNS resolution is available
-        echo -e "${OVER}  ${TICK} DNS resolution is available"
-        return 0
-      fi
-      # Append one dot for each second waiting
-      echo -ne "."
-      sleep 1
+  until getent hosts github.com &> /dev/null; do
+  # Append one dot for each second waiting
+    str="${str}."
+    echo -ne "  ${OVER}  ${INFO} ${str}"
+    sleep 1
  done

-  # DNS resolution is still unavailable after 120 seconds
-  return 1
-
-}
-
-# Function: try_restore_backup
-# Description: Attempts to restore the previous Pi-hole gravity database from a
-#              backup file. If a backup exists, it copies the backup to the
-#              gravity database file and prepares a new gravity database. If the
-#              restoration is successful, it returns 0. Otherwise, it returns 1.
-# Returns:
-#   0 - If the backup is successfully restored.
-#   1 - If no backup is available or if the restoration fails.
-try_restore_backup () {
-  local num filename timestamp
-  num=$1
-  filename="${gravityBCKfile}.${num}"
-  # Check if a backup exists
-  if [ -f "${filename}" ]; then
-    echo -e "  ${INFO} Attempting to restore previous database from backup no. ${num}"
-    cp "${filename}" "${gravityDBfile}"
-
-    # If the backup was successfully copied, prepare a new gravity database from
-    # it
-    if [ -f "${gravityDBfile}" ]; then
-      output=$({ pihole-FTL sqlite3 -ni "${gravityTEMPfile}" <<<"${copyGravity}"; } 2>&1)
-      status="$?"
-
-      # Error checking
-      if [[ "${status}" -ne 0 ]]; then
-        echo -e "\\n  ${CROSS} Unable to copy data from ${gravityDBfile} to ${gravityTEMPfile}\\n  ${output}"
-        gravity_Cleanup "error"
-      fi
-
-      # Get the timestamp of the backup file in a human-readable format
-      # Note that this timestamp will be in the server timezone, this may be
-      # GMT, e.g., on a Raspberry Pi where the default timezone has never been
-      # changed
-      timestamp=$(date -r "${filename}" "+%Y-%m-%d %H:%M:%S %Z")
-
-      # Add a record to the info table to indicate that the gravity database was restored
-      pihole-FTL sqlite3 "${gravityTEMPfile}" "INSERT OR REPLACE INTO info (property,value) values ('gravity_restored','${timestamp}');"
-      echo -e "  ${TICK} Successfully restored from backup (${gravityBCKfile}.${num} at ${timestamp})"
-      return 0
-    else
-      echo -e "  ${CROSS} Unable to restore backup no. ${num}"
-    fi
-  fi
-
-  echo -e "  ${CROSS} Backup no. ${num} not available"
-  return 1
+  # If we reach this point, DNS resolution is available
+  echo -e "${OVER}  ${TICK} DNS resolution is available"
 }

 # Retrieve blocklist URLs and parse domains from adlist.list
@@ -424,7 +332,33 @@ gravity_DownloadBlocklists() {
    echo -e "  ${INFO} Storing gravity database in ${COL_BOLD}${gravityDBfile}${COL_NC}"
  fi

-  local url domain str compression adlist_type directory success
+  # Retrieve source URLs from gravity database
+  # We source only enabled adlists, SQLite3 stores boolean values as 0 (false) or 1 (true)
+  mapfile -t sources <<<"$(pihole-FTL sqlite3 -ni "${gravityDBfile}" "SELECT address FROM vw_adlist;" 2>/dev/null)"
+  mapfile -t sourceIDs <<<"$(pihole-FTL sqlite3 -ni "${gravityDBfile}" "SELECT id FROM vw_adlist;" 2>/dev/null)"
+  mapfile -t sourceTypes <<<"$(pihole-FTL sqlite3 -ni "${gravityDBfile}" "SELECT type FROM vw_adlist;" 2>/dev/null)"
+
+  # Parse source domains from $sources
+  mapfile -t sourceDomains <<<"$(
+    # Logic: Split by folder/port
+    awk -F '[/:]' '{
+      # Remove URL protocol & optional username:password@
+      gsub(/(.*:\/\/|.*:.*@)/, "", $0)
+      if(length($1)>0){print $1}
+      else {print "local"}
+    }' <<<"$(printf '%s\n' "${sources[@]}")" 2>/dev/null
+  )"
+
+  local str="Pulling blocklist source list into range"
+  echo -e "${OVER}  ${TICK} ${str}"
+
+  if [[ -z "${sources[*]}" ]] || [[ -z "${sourceDomains[*]}" ]]; then
+    echo -e "  ${INFO} No source list found, or it is empty"
+    echo ""
+    unset sources
+  fi
+
+  local url domain str target compression adlist_type directory
  echo ""

  # Prepare new gravity database
@@ -456,55 +390,10 @@ gravity_DownloadBlocklists() {

  if [[ "${status}" -ne 0 ]]; then
    echo -e "\\n  ${CROSS} Unable to copy data from ${gravityDBfile} to ${gravityTEMPfile}\\n  ${output}"
-
-    # Try to attempt a backup restore
-    success=false
-    if [[ -d "${gravityBCKdir}" ]]; then
-      for i in {1..10}; do
-        if try_restore_backup "${i}"; then
-          success=true
-          break
-        fi
-      done
-    fi
-
-    # If none of the attempts worked, return 1
-    if [[ "${success}" == false ]]; then
-      pihole-FTL sqlite3 "${gravityTEMPfile}" "INSERT OR REPLACE INTO info (property,value) values ('gravity_restored','failed');"
-      return 1
-    fi
-
-    echo -e "  ${TICK} ${str}"
-  else
-    echo -e "${OVER}  ${TICK} ${str}"
+    return 1
  fi
-
-  # Retrieve source URLs from gravity database
-  # We source only enabled adlists, SQLite3 stores boolean values as 0 (false) or 1 (true)
-  mapfile -t sources <<<"$(pihole-FTL sqlite3 -ni "${gravityDBfile}" "SELECT address FROM vw_adlist;" 2>/dev/null)"
-  mapfile -t sourceIDs <<<"$(pihole-FTL sqlite3 -ni "${gravityDBfile}" "SELECT id FROM vw_adlist;" 2>/dev/null)"
-  mapfile -t sourceTypes <<<"$(pihole-FTL sqlite3 -ni "${gravityDBfile}" "SELECT type FROM vw_adlist;" 2>/dev/null)"
-
-  # Parse source domains from $sources
-  mapfile -t sourceDomains <<<"$(
-    # Logic: Split by folder/port
-    awk -F '[/:]' '{
-      # Remove URL protocol & optional username:password@
-      gsub(/(.*:\/\/|.*:.*@)/, "", $0)
-      if(length($1)>0){print $1}
-      else {print "local"}
-    }' <<<"$(printf '%s\n' "${sources[@]}")" 2>/dev/null
-  )"
-
-  local str="Pulling blocklist source list into range"
  echo -e "${OVER}  ${TICK} ${str}"

-  if [[ -z "${sources[*]}" ]] || [[ -z "${sourceDomains[*]}" ]]; then
-    echo -e "  ${INFO} No source list found, or it is empty"
-    echo ""
-    unset sources
-  fi
-
  # Use compression to reduce the amount of data that is transferred
  # between the Pi-hole and the ad list provider. Use this feature
  # only if it is supported by the locally available version of curl
@@ -515,15 +404,6 @@ gravity_DownloadBlocklists() {
    compression=""
    echo -e "  ${INFO} Libz compression not available\n"
  fi
-
-  # Check if etag is supported by the locally available version of curl
-  # (available as of curl 7.68.0, released Jan 2020)
-  # https://github.com/curl/curl/pull/4543 +
-  # https://github.com/curl/curl/pull/4678
-  if curl --help all | grep -q "etag-save"; then
-    etag_support=true
-  fi
-
  # Loop through $sources and download each one
  for ((i = 0; i < "${#sources[@]}"; i++)); do
    url="${sources[$i]}"
@@ -540,8 +420,8 @@ gravity_DownloadBlocklists() {
    fi

    # Save the file as list.#.domain
-    saveLocation="${listsCacheDir}/list.${id}.${domain}.${domainsExtension}"
-    activeDomains[i]="${saveLocation}"
+    saveLocation="${piholeDir}/list.${id}.${domain}.${domainsExtension}"
+    activeDomains[$i]="${saveLocation}"

    # Check if we can write to the save location file without actually creating
    # it (in case it doesn't exist)
@@ -573,12 +453,12 @@ gravity_DownloadBlocklists() {
    if [[ "${check_url}" =~ ${regex} ]]; then
      echo -e "  ${CROSS} Invalid Target"
    else
-      timeit gravity_DownloadBlocklistFromUrl "${url}" "${sourceIDs[$i]}" "${saveLocation}" "${compression}" "${adlist_type}" "${domain}"
+      timeit gravity_DownloadBlocklistFromUrl "${url}" "${sourceIDs[$i]}" "${saveLocation}" "${target}" "${compression}" "${adlist_type}" "${domain}"
    fi
    echo ""
  done

-  DownloadBlocklists_done=true
+  gravity_Blackbody=true
 }

 compareLists() {
@@ -607,8 +487,8 @@ compareLists() {

 # Download specified URL and perform checks on HTTP status and file content
 gravity_DownloadBlocklistFromUrl() {
-  local url="${1}" adlistID="${2}" saveLocation="${3}" compression="${4}" gravity_type="${5}" domain="${6}"
-  local modifiedOptions="" listCurlBuffer str httpCode success="" ip cmd_ext
+  local url="${1}" adlistID="${2}" saveLocation="${3}" target="${4}" compression="${5}" gravity_type="${6}" domain="${7}"
+  local heisenbergCompensator="" listCurlBuffer str httpCode success="" ip cmd_ext
  local file_path permissions ip_addr port blocked=false download=true

  # Create temp file to store content on disk instead of RAM
@@ -617,42 +497,43 @@ gravity_DownloadBlocklistFromUrl() {
  mv "${listCurlBuffer}" "${listCurlBuffer%.*}.phgpb"
  listCurlBuffer="${listCurlBuffer%.*}.phgpb"

-  # For all remote files, we try to determine if the file has changed to skip
-  # downloading them whenever possible.
-  if [[ $url != "file"* ]]; then
-    # Use the HTTP ETag header to determine if the file has changed if supported
-    # by curl. Using ETags is supported by raw.githubusercontent.com URLs.
-    if [[ "${etag_support}" == true ]]; then
-      # Save HTTP ETag to the specified file. An ETag is a caching related header,
-      # usually returned in a response. If no ETag is sent by the server, an empty
-      # file is created and can later be used consistently.
-      modifiedOptions="--etag-save ${saveLocation}.etag"
-
-      if [[ -f "${saveLocation}.etag" ]]; then
-        # This option makes a conditional HTTP request for the specific ETag read
-        # from the given file by sending a custom If-None-Match header using the
-        # stored ETag. This way, the server will only send the file if it has
-        # changed since the last request.
-        modifiedOptions="${modifiedOptions} --etag-compare ${saveLocation}.etag"
-      fi
-    fi
-
-    # Add If-Modified-Since header to the request if we did already download the
-    # file once
-    if [[ -f "${saveLocation}" ]]; then
-      # Request a file that has been modified later than the given time and
-      # date. We provide a file here which makes curl use the modification
-      # timestamp (mtime) of this file.
-      # Interstingly, this option is not supported by raw.githubusercontent.com
-      # URLs, however, it is still supported by many older web servers which may
-      # not support the HTTP ETag method so we keep it as a fallback.
-      modifiedOptions="${modifiedOptions} -z ${saveLocation}"
-    fi
+  # Determine if $saveLocation has read permission
+  if [[ -r "${saveLocation}" && $url != "file"* ]]; then
+    # Have curl determine if a remote file has been modified since last retrieval
+    # Uses "Last-Modified" header, which certain web servers do not provide (e.g: raw github urls)
+    # Note: Don't do this for local files, always download them
+    heisenbergCompensator="-z ${saveLocation}"
  fi

  str="Status:"
  echo -ne "  ${INFO} ${str} Pending..."
  blocked=false
+  case $(getFTLConfigValue dns.blocking.mode) in
+  "IP-NODATA-AAAA" | "IP")
+    # Get IP address of this domain
+    ip="$(dig "${domain}" +short)"
+    # Check if this IP matches any IP of the system
+    if [[ -n "${ip}" && $(grep -Ec "inet(|6) ${ip}" <<<"$(ip a)") -gt 0 ]]; then
+      blocked=true
+    fi
+    ;;
+  "NXDOMAIN")
+    if [[ $(dig "${domain}" | grep "NXDOMAIN" -c) -ge 1 ]]; then
+      blocked=true
+    fi
+    ;;
+  "NODATA")
+    if [[ $(dig "${domain}" | grep "NOERROR" -c) -ge 1 ]] && [[ -z $(dig +short "${domain}") ]]; then
+      blocked=true
+    fi
+    ;;
+  "NULL" | *)
+    if [[ $(dig "${domain}" +short | grep "0.0.0.0" -c) -ge 1 ]]; then
+      blocked=true
+    fi
+    ;;
+  esac
+
  # Check if this domain is blocked by Pi-hole but only if the domain is not a
  # local file or empty
  if [[ $url != "file"* ]] && [[ -n "${domain}" ]]; then
@@ -750,9 +631,8 @@ gravity_DownloadBlocklistFromUrl() {
  fi

  if [[ "${download}" == true ]]; then
-    # See https://github.com/pi-hole/pi-hole/issues/6159 for justification of the below disable directive
    # shellcheck disable=SC2086
-    httpCode=$(curl --connect-timeout ${curl_connect_timeout} -s -L ${compression} ${cmd_ext} ${modifiedOptions} -w "%{http_code}" "${url}" -o "${listCurlBuffer}" 2>/dev/null)
+    httpCode=$(curl --connect-timeout ${curl_connect_timeout} -s -L ${compression} ${cmd_ext} ${heisenbergCompensator} -w "%{http_code}" "${url}" -o "${listCurlBuffer}" 2>/dev/null)
  fi

  case $url in
@@ -795,21 +675,20 @@ gravity_DownloadBlocklistFromUrl() {
  # Determine if the blocklist was downloaded and saved correctly
  if [[ "${success}" == true ]]; then
    if [[ "${httpCode}" == "304" ]]; then
-      # Set list status to "unchanged/cached"
-      database_adlist_status "${adlistID}" "2"
      # Add domains to database table file
      pihole-FTL "${gravity_type}" parseList "${saveLocation}" "${gravityTEMPfile}" "${adlistID}"
+      database_adlist_status "${adlistID}" "2"
      done="true"
    # Check if $listCurlBuffer is a non-zero length file
    elif [[ -s "${listCurlBuffer}" ]]; then
-      # Move the downloaded list to the final location
-      mv "${listCurlBuffer}" "${saveLocation}"
-      # Ensure the file has the correct permissions
-      fix_owner_permissions "${saveLocation}"
-      # Compare lists if they are identical
-      compareLists "${adlistID}" "${saveLocation}"
+      # Determine if blocklist is non-standard and parse as appropriate
+      gravity_ParseFileIntoDomains "${listCurlBuffer}" "${saveLocation}"
+      # Remove curl buffer file after its use
+      rm "${listCurlBuffer}"
      # Add domains to database table file
      pihole-FTL "${gravity_type}" parseList "${saveLocation}" "${gravityTEMPfile}" "${adlistID}"
+      # Compare lists, are they identical?
+      compareLists "${adlistID}" "${saveLocation}"
      done="true"
    else
      # Fall back to previously cached list if $listCurlBuffer is empty
@@ -822,10 +701,9 @@ gravity_DownloadBlocklistFromUrl() {
    # Determine if cached list has read permission
    if [[ -r "${saveLocation}" ]]; then
      echo -e "  ${CROSS} List download failed: ${COL_LIGHT_GREEN}using previously cached list${COL_NC}"
-      # Set list status to "download-failed/cached"
-      database_adlist_status "${adlistID}" "3"
      # Add domains to database table file
      pihole-FTL "${gravity_type}" parseList "${saveLocation}" "${gravityTEMPfile}" "${adlistID}"
+      database_adlist_status "${adlistID}" "3"
    else
      echo -e "  ${CROSS} List download failed: ${COL_LIGHT_RED}no cached list available${COL_NC}"
      # Manually reset these two numbers because we do not call parseList here
@@ -835,6 +713,37 @@ gravity_DownloadBlocklistFromUrl() {
  fi
 }

+# Parse source files into domains format
+gravity_ParseFileIntoDomains() {
+  local src="${1}" destination="${2}"
+
+  # Remove comments and print only the domain name
+  # Most of the lists downloaded are already in hosts file format but the spacing/formatting is not contiguous
+  # This helps with that and makes it easier to read
+  # It also helps with debugging so each stage of the script can be researched more in depth
+  # 1) Convert all characters to lowercase
+  tr '[:upper:]' '[:lower:]' <"${src}" >"${destination}"
+
+  # 2) Remove carriage returns
+  # 3) Remove lines starting with ! (ABP Comments)
+  # 4) Remove lines starting with [ (ABP Header)
+  # 5) Remove lines containing ABP extended CSS selectors ("##", "#$#", "#@#", "#?#") and Adguard JavaScript (#%#) preceded by a letter
+  # 6) Remove comments (text starting with "#", include possible spaces before the hash sign)
+  # 7) Remove leading tabs, spaces, etc. (Also removes leading IP addresses)
+  # 8) Remove empty lines
+
+  sed -i -r \
+    -e 's/\r$//' \
+    -e 's/\s*!.*//g' \
+    -e 's/\s*\[.*//g' \
+    -e '/[a-z]\#[$?@%]{0,3}\#/d' \
+    -e 's/\s*#.*//g' \
+    -e 's/^.*\s+//g' \
+    -e '/^$/d' "${destination}"
+
+  fix_owner_permissions "${destination}"
+}
+
 # Report number of entries in a table
 gravity_Table_Count() {
  local table="${1}"
@@ -855,11 +764,11 @@ gravity_Table_Count() {
 gravity_ShowCount() {
  # Here we use the table "gravity" instead of the view "vw_gravity" for speed.
  # It's safe to replace it here, because right after a gravity run both will show the exactly same number of domains.
-  gravity_Table_Count "gravity" "gravity domains"
-  gravity_Table_Count "domainlist WHERE type = 1 AND enabled = 1" "exact denied domains"
-  gravity_Table_Count "domainlist WHERE type = 3 AND enabled = 1" "regex denied filters"
-  gravity_Table_Count "domainlist WHERE type = 0 AND enabled = 1" "exact allowed domains"
-  gravity_Table_Count "domainlist WHERE type = 2 AND enabled = 1" "regex allowed filters"
+  gravity_Table_Count "gravity" "gravity domains" ""
+  gravity_Table_Count "vw_blacklist" "exact denied domains"
+  gravity_Table_Count "vw_regex_blacklist" "regex denied filters"
+  gravity_Table_Count "vw_whitelist" "exact allowed domains"
+  gravity_Table_Count "vw_regex_whitelist" "regex allowed filters"
 }

 # Trap Ctrl-C
@@ -882,13 +791,13 @@ gravity_Cleanup() {
  # invalid_domains location
  rm "${GRAVITY_TMPDIR}"/*.ph-non-domains 2>/dev/null

-  # Ensure this function only runs when gravity_DownloadBlocklists() has completed
-  if [[ "${DownloadBlocklists_done:-}" == true ]]; then
-    # Remove any unused .domains/.etag/.sha files
-    for file in "${listsCacheDir}"/*."${domainsExtension}"; do
-      # If list is not in active array, then remove it and all associated files
+  # Ensure this function only runs when gravity_SetDownloadOptions() has completed
+  if [[ "${gravity_Blackbody:-}" == true ]]; then
+    # Remove any unused .domains files
+    for file in "${piholeDir}"/*."${domainsExtension}"; do
+      # If list is not in active array, then remove it
      if [[ ! "${activeDomains[*]}" == *"${file}"* ]]; then
-        rm -f "${file}"* 2>/dev/null ||
+        rm -f "${file}" 2>/dev/null ||
          echo -e "  ${CROSS} Failed to remove ${file##*/}"
      fi
    done
@@ -1008,33 +917,11 @@ timeit(){
  elapsed_time=$((end_time - start_time))

  # Display the elapsed time
-  printf "  %b--> took %d.%03d seconds%b\n" "${COL_BLUE}" $((elapsed_time / 1000)) $((elapsed_time % 1000)) "${COL_NC}"
+  printf "  %b--> took %d.%03d seconds%b\n" ${COL_BLUE} $((elapsed_time / 1000)) $((elapsed_time % 1000)) ${COL_NC}

  return $ret
 }

-migrate_to_listsCache_dir() {
-  # If the ${listsCacheDir} directory already exists, this has been done before
-  if [[ -d "${listsCacheDir}" ]]; then
-    return
-  fi
-
-  # If not, we need to migrate the old files to the new directory
-  local str="Migrating the list's cache directory to new location"
-  echo -ne "  ${INFO} ${str}..."
-  mkdir -p "${listsCacheDir}" && chown pihole:pihole "${listsCacheDir}"
-
-  # Move the old files to the new directory
-  if mv "${piholeDir}"/list.* "${listsCacheDir}/" 2>/dev/null; then
-    echo -e "${OVER}  ${TICK} ${str}"
-  else
-    echo -e "${OVER}  ${CROSS} ${str}"
-  fi
-
-  # Update the list's paths in the corresponding .sha1 files to the new location
-  sed -i "s|${piholeDir}/|${listsCacheDir}/|g" "${listsCacheDir}"/*.sha1 2>/dev/null
-}
-
 helpFunc() {
  echo "Usage: pihole -g
 Update domains from blocklists specified in adlists.list
@@ -1081,19 +968,13 @@ for var in "$@"; do
  "-t" | "--timeit") timed=true ;;
  "-r" | "--repair") repairSelector "$3" ;;
  "-u" | "--upgrade")
-    upgrade_gravityDB "${gravityDBfile}"
+    upgrade_gravityDB "${gravityDBfile}" "${piholeDir}"
    exit 0
    ;;
  "-h" | "--help") helpFunc ;;
  esac
 done

-# Check if DNS is available, no need to do any database manipulation if we're not able to download adlists
-if ! timeit gravity_CheckDNSResolutionAvailable; then
-  echo -e "   ${CROSS} No DNS resolution available. Please contact support."
-  exit 1
-fi
-
 # Remove OLD (backup) gravity file, if it exists
 if [[ -f "${gravityOLDfile}" ]]; then
  rm "${gravityOLDfile}"
@@ -1116,9 +997,6 @@ if [[ "${recover_database:-}" == true ]]; then
  timeit database_recovery "$4"
 fi

-# Migrate scattered list files to the new cache directory
-migrate_to_listsCache_dir
-
 # Move possibly existing legacy files to the gravity database
 if ! timeit migrate_to_database; then
  echo -e "   ${CROSS} Unable to migrate to database. Please contact support."
@@ -1129,11 +1007,16 @@ if [[ "${forceDelete:-}" == true ]]; then
  str="Deleting existing list cache"
  echo -ne "${INFO} ${str}..."

-  rm "${listsCacheDir}/list.*" 2>/dev/null || true
+  rm /etc/pihole/list.* 2>/dev/null || true
  echo -e "${OVER}  ${TICK} ${str}"
 fi

 # Gravity downloads blocklists next
+if ! timeit gravity_CheckDNSResolutionAvailable; then
+  echo -e "   ${CROSS} Can not complete gravity update, no DNS is available. Please contact support."
+  exit 1
+fi
+
 if ! gravity_DownloadBlocklists; then
  echo -e "   ${CROSS} Unable to create gravity database. Please try again later. If the problem persists, please contact support."
  exit 1
--- a/manpages/pihole.8
+++ b/manpages/pihole.8
@@ -23,7 +23,7 @@ pihole -r
 .br
 \fBpihole -g\fR
 .br
-\fBpihole\fR \fB-q\fR [options]
+\fBpihole\fR -\fBq\fR [options]
 .br
 \fBpihole\fR \fB-l\fR (\fBon|off|off noflush\fR)
 .br
@@ -43,7 +43,7 @@ pihole -r
 .br
 \fBpihole\fR \fBcheckout\fR repo [branch]
 .br
-\fBpihole\fR \fBapi\fR [verbose] endpoint
+\fBpihole\fR \api\fR endpoint
 .br
 \fBpihole\fR \fBhelp\fR
 .br
@@ -234,14 +234,10 @@ Available commands and options:
      branchname        Update subsystems to the specified branchname
 .br

-\fBapi\fR [verbose] endpoint
+\fBapi\fR endpoint
 .br
    Query the Pi-hole API at <endpoint>
 .br
-
-      verbose           Show authentication and status messages
-.br
-
 .SH "EXAMPLE"

 Some usage examples
@@ -327,11 +323,6 @@ Switching Pi-hole subsystem branches
    Queries FTL for the stats/summary endpoint
 .br

-\fBpihole api verbose stats/summary\fR
-.br
-    Same as above, but shows authentication and status messages
-.br
-
 .SH "COLOPHON"

 Get sucked into the latest news and community activity by entering Pi-hole's orbit. Information about Pi-hole, and the latest version of the software can be found at https://pi-hole.net.
--- a/41
+++ b/41
@@ -17,16 +17,13 @@ readonly PI_HOLE_SCRIPT_DIR="/opt/pihole"
 PI_HOLE_BIN_DIR="/usr/local/bin"

 readonly colfile="${PI_HOLE_SCRIPT_DIR}/COL_TABLE"
-# shellcheck source=./advanced/Scripts/COL_TABLE
 source "${colfile}"

 readonly utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
-# shellcheck source=./advanced/Scripts/utils.sh
 source "${utilsfile}"

 # Source api functions
 readonly apifile="${PI_HOLE_SCRIPT_DIR}/api.sh"
-# shellcheck source=./advanced/Scripts/api.sh
 source "${apifile}"

 versionsfile="/etc/pihole/versions"
@@ -34,7 +31,6 @@ if [ -f "${versionsfile}" ]; then
    # Only source versionsfile if the file exits
    # fixes a warning during installation where versionsfile does not exist yet
    # but gravity calls `pihole -status` and thereby sourcing the file
-    # shellcheck source=/dev/null
    source "${versionsfile}"
 fi

@@ -77,17 +73,19 @@ listFunc() {

 debugFunc() {
    local automated
+    local web
    local check_database_integrity
    # Pull off the `debug` leaving passed call augmentation flags in $1
    shift

    for value in "$@"; do
        [[ "$value"  == *"-a"* ]] && automated="true"
+        [[ "$value"  == *"-w"* ]] && web="true"
        [[ "$value"  == *"-c"* ]] && check_database_integrity="true"
        [[ "$value" == *"--check_database"* ]] && check_database_integrity="true"
    done

-  AUTOMATED=${automated:-} CHECK_DATABASE=${check_database_integrity:-} "${PI_HOLE_SCRIPT_DIR}"/piholeDebug.sh
+  AUTOMATED=${automated:-} WEBCALL=${web:-} CHECK_DATABASE=${check_database_integrity:-} "${PI_HOLE_SCRIPT_DIR}"/piholeDebug.sh
  exit 0
 }

@@ -111,11 +109,11 @@ updatePiholeFunc() {
  fi
 }

-repairPiholeFunc() {
+reconfigurePiholeFunc() {
  if [ -n "${DOCKER_VERSION}" ]; then
    unsupportedFunc
  else
-    /etc/.pihole/automated\ install/basic-install.sh --repair
+    /etc/.pihole/automated\ install/basic-install.sh --reconfigure
    exit 0;
  fi
 }
@@ -251,14 +249,12 @@ Time:
  data=$(PostFTLData "dns/blocking" "{ \"blocking\": ${1}, \"timer\": ${tt} }")

  # Check the response
-  local extra timer
-  extra=" forever"
-  timer="$(echo "${data}"| jq --raw-output '.timer' )"
+  local extra=" forever"
+  local timer="$(echo "${data}"| jq --raw-output '.timer' )"
  if [[ "${timer}" != "null" ]]; then
    extra=" for ${timer}s"
  fi
-  local str
-  str="Pi-hole $(echo "${data}" | jq --raw-output '.blocking')${extra}"
+  local str="Pi-hole $(echo "${data}" | jq --raw-output '.blocking')${extra}"

  # Logout from the API
  LogoutAPI
@@ -381,16 +377,14 @@ statusFunc() {

 tailFunc() {
  # Warn user if Pi-hole's logging is disabled
-  local logging_enabled
-  logging_enabled=$(getFTLConfigValue dns.queryLogging)
+  local logging_enabled=$(getFTLConfigValue dns.queryLogging)
  if [[ "${logging_enabled}" != "true" ]]; then
    echo "  ${CROSS} Warning: Query logging is disabled"
  fi
  echo -e "  ${INFO} Press Ctrl-C to exit"

  # Get logfile path
-  LOGFILE=$(getFTLConfigValue files.log.dnsmasq)
-  readonly LOGFILE
+  readonly LOGFILE=$(getFTLConfigValue files.log.dnsmasq)

  # Strip date from each line
  # Color blocklist/denylist/wildcard entries as red
@@ -406,10 +400,7 @@ tailFunc() {

 piholeCheckoutFunc() {
  if [ -n "${DOCKER_VERSION}" ]; then
-    echo -e "${CROSS} Function not supported in Docker images"
-    echo "Please build a custom image following the steps at"
-    echo "https://github.com/pi-hole/docker-pi-hole?tab=readme-ov-file#building-the-image-locally"
-    exit 0
+    unsupportedFunc
  else
    if [[ "$2" == "-h" ]] || [[ "$2" == "--help" ]]; then
      echo "Switch Pi-hole subsystems to a different GitHub branch
@@ -431,7 +422,6 @@ piholeCheckoutFunc() {
      exit 0
    fi

-    #shellcheck source=./advanced/Scripts/piholeCheckout.sh
    source "${PI_HOLE_SCRIPT_DIR}"/piholeCheckout.sh
    shift
    checkout "$@"
@@ -488,12 +478,11 @@ Debugging Options:
                        Add '-c' or '--check-database' to include a Pi-hole database integrity check
                        Add '-a' to automatically upload the log to tricorder.pi-hole.net
  -f, flush           Flush the Pi-hole log
-  -r, repair          Repair Pi-hole subsystems
+  -r, reconfigure     Reconfigure or Repair Pi-hole subsystems
  -t, tail [arg]      View the live output of the Pi-hole log.
                      Add an optional argument to filter the log
                      (regular expressions are supported)
  api <endpoint>      Query the Pi-hole API at <endpoint>
-                        Precede <endpoint> with 'verbose' option to show authentication and status messages


 Options:
@@ -546,7 +535,7 @@ case "${1}" in
  "--allow-wild" | "allow-wild"   ) need_root=0;;
  "-f" | "flush"                  ) ;;
  "-up" | "updatePihole"          ) ;;
-  "-r"  | "repair"                ) ;;
+  "-r"  | "reconfigure"           ) ;;
  "-l" | "logging"                ) ;;
  "uninstall"                     ) ;;
  "enable"                        ) need_root=0;;
@@ -589,7 +578,7 @@ case "${1}" in
  "-d" | "debug"                  ) debugFunc "$@";;
  "-f" | "flush"                  ) flushFunc "$@";;
  "-up" | "updatePihole"          ) updatePiholeFunc "$@";;
-  "-r"  | "repair"                ) repairPiholeFunc;;
+  "-r"  | "reconfigure"           ) reconfigurePiholeFunc;;
  "-g" | "updateGravity"          ) updateGravityFunc "$@";;
  "-l" | "logging"                ) piholeLogging "$@";;
  "uninstall"                     ) uninstallFunc;;
@@ -602,6 +591,6 @@ case "${1}" in
  "updatechecker"                 ) shift; updateCheckFunc "$@";;
  "arpflush"                      ) arpFunc "$@";;
  "-t" | "tail"                   ) tailFunc "$2";;
-  "api"                           ) shift; apiFunc "$@";;
+  "api"                           ) apiFunc "$2";;
  *                               ) helpFunc;;
 esac
--- a/test/_centos_10.Dockerfile
+++ b/test/_centos_10.Dockerfile
@@ -1,19 +0,0 @@
-FROM quay.io/centos/centos:stream10
-# Disable SELinux
-RUN echo "SELINUX=disabled" > /etc/selinux/config
-RUN yum install -y --allowerasing curl git initscripts
-
-ENV GITDIR=/etc/.pihole
-ENV SCRIPTDIR=/opt/pihole
-
-RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole
-ADD . $GITDIR
-RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $GITDIR/advanced/Scripts/COL_TABLE $SCRIPTDIR/
-ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR
-
-RUN true && \
-    chmod +x $SCRIPTDIR/*
-
-ENV SKIP_INSTALL=true
-
-#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_centos_9.Dockerfile
+++ b/test/_centos_9.Dockerfile
@@ -15,5 +15,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_debian_11.Dockerfile
+++ b/test/_debian_11.Dockerfile
@@ -12,5 +12,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_debian_12.Dockerfile
+++ b/test/_debian_12.Dockerfile
@@ -12,5 +12,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_fedora_40.Dockerfile
+++ b/test/_fedora_40.Dockerfile
@@ -13,5 +13,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_fedora_41.Dockerfile
+++ b/test/_fedora_41.Dockerfile
@@ -13,5 +13,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_ubuntu_20.Dockerfile
+++ b/test/_ubuntu_20.Dockerfile
@@ -12,5 +12,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_ubuntu_22.Dockerfile
+++ b/test/_ubuntu_22.Dockerfile
@@ -13,5 +13,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_ubuntu_23.Dockerfile
+++ b/test/_ubuntu_23.Dockerfile
@@ -1,5 +1,4 @@
-FROM fedora:42
-RUN dnf install -y git initscripts
+FROM buildpack-deps:lunar-scm

 ENV GITDIR=/etc/.pihole
 ENV SCRIPTDIR=/opt/pihole
@@ -8,10 +7,12 @@ RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole
 ADD . $GITDIR
 RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $GITDIR/advanced/Scripts/COL_TABLE $SCRIPTDIR/
 ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR
+ENV DEBIAN_FRONTEND=noninteractive

 RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/_ubuntu_24.Dockerfile
+++ b/test/_ubuntu_24.Dockerfile
@@ -13,5 +13,6 @@ RUN true && \
    chmod +x $SCRIPTDIR/*

 ENV SKIP_INSTALL=true
+ENV OS_CHECK_DOMAIN_NAME=dev-supportedos.pi-hole.net

 #sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \
--- a/test/requirements.txt
+++ b/test/requirements.txt
@@ -1,6 +1,6 @@
 pyyaml == 6.0.2
-pytest == 8.3.5
+pytest == 8.3.4
 pytest-xdist == 3.6.1
-pytest-testinfra == 10.2.2
-tox == 4.26.0
+pytest-testinfra == 10.1.1
+tox == 4.23.2
 pytest-clarity == 1.0.1
--- a/test/test_any_automated_install.py
+++ b/test/test_any_automated_install.py
@@ -89,10 +89,10 @@ def test_installPihole_fresh_install_readableFiles(host):
    export DEBIAN_FRONTEND=noninteractive
    umask 0027
    runUnattended=true
-    fresh_install=false
+    useUpdateVars=true
    source /opt/pihole/basic-install.sh > /dev/null
    runUnattended=true
-    fresh_install=false
+    useUpdateVars=true
    main
    /opt/pihole/pihole-FTL-prestart.sh
    """
@@ -119,6 +119,11 @@ def test_installPihole_fresh_install_readableFiles(host):
    assert exit_status_success == actual_rc
    check_leases = test_cmd.format("w", "/etc/pihole/dhcp.leases", piholeuser)
    actual_rc = host.run(check_leases).rc
+    # readable dns-servers.conf
+    assert exit_status_success == actual_rc
+    check_servers = test_cmd.format("r", "/etc/pihole/dns-servers.conf", piholeuser)
+    actual_rc = host.run(check_servers).rc
+    assert exit_status_success == actual_rc
    # readable install.log
    check_install = test_cmd.format("r", "/etc/pihole/install.log", piholeuser)
    actual_rc = host.run(check_install).rc
@@ -127,6 +132,10 @@ def test_installPihole_fresh_install_readableFiles(host):
    check_localversion = test_cmd.format("r", "/etc/pihole/versions", piholeuser)
    actual_rc = host.run(check_localversion).rc
    assert exit_status_success == actual_rc
+    # readable logrotate
+    check_logrotate = test_cmd.format("r", "/etc/pihole/logrotate", piholeuser)
+    actual_rc = host.run(check_logrotate).rc
+    assert exit_status_success == actual_rc
    # readable macvendor.db
    check_macvendor = test_cmd.format("r", "/etc/pihole/macvendor.db", piholeuser)
    actual_rc = host.run(check_macvendor).rc
@@ -245,7 +254,6 @@ def test_FTL_detect_no_errors(host, arch, detected_string, supported):
        {
            "-A /bin/sh": ("Tag_CPU_arch: " + arch, "0"),
            "-A /usr/bin/sh": ("Tag_CPU_arch: " + arch, "0"),
-            "-A /usr/sbin/sh": ("Tag_CPU_arch: " + arch, "0"),
        },
        host,
    )
@@ -466,6 +474,50 @@ def test_validate_ip(host):
    test_address("0.0.0.0#00001", False)


+def test_os_check_fails(host):
+    """Confirms install fails on unsupported OS"""
+    host.run(
+        """
+    source /opt/pihole/basic-install.sh
+    package_manager_detect
+    build_dependency_package
+    install_dependent_packages
+    cat <<EOT > /etc/os-release
+ID=UnsupportedOS
+VERSION_ID="2"
+EOT
+    """
+    )
+    detectOS = host.run(
+        """t
+    source /opt/pihole/basic-install.sh
+    os_check
+    """
+    )
+    expected_stdout = "Unsupported OS detected: UnsupportedOS"
+    assert expected_stdout in detectOS.stdout
+
+
+def test_os_check_passes(host):
+    """Confirms OS meets the requirements"""
+    host.run(
+        """
+    source /opt/pihole/basic-install.sh
+    package_manager_detect
+    build_dependency_package
+    install_dependent_packages
+    """
+    )
+    detectOS = host.run(
+        """
+    source /opt/pihole/basic-install.sh
+    os_check
+    """
+    )
+    expected_stdout = "Supported OS detected"
+    assert expected_stdout in detectOS.stdout
+
+
 def test_package_manager_has_pihole_deps(host):
    """Confirms OS is able to install the required packages for Pi-hole"""
    mock_command("dialog", {"*": ("", "0")}, host)
--- a/test/tox.centos_10.ini
+++ b/test/tox.centos_10.ini
@@ -1,10 +0,0 @@
-[tox]
-envlist = py3
-
-[testenv:py3]
-allowlist_externals = docker
-deps = -rrequirements.txt
-setenv =
-    COLUMNS=120
-commands = docker buildx build --load --progress plain -f _centos_10.Dockerfile -t pytest_pihole:test_container ../
-           pytest {posargs:-vv -n auto} ./test_any_automated_install.py ./test_any_utils.py ./test_centos_fedora_common_support.py
--- a/test/tox.fedora_42.ini
+++ b/test/tox.fedora_42.ini
@@ -1,10 +0,0 @@
-[tox]
-envlist = py3
-
-[testenv]
-allowlist_externals = docker
-deps = -rrequirements.txt
-setenv =
-    COLUMNS=120
-commands = docker buildx build --load --progress plain -f _fedora_42.Dockerfile -t pytest_pihole:test_container ../
-           pytest {posargs:-vv -n auto} ./test_any_automated_install.py ./test_any_utils.py ./test_centos_fedora_common_support.py
Author	SHA1	Message	Date
DL6ER	f0dd363c99	Explicitly log migration process Some checks are pending CodeQL / Analyze (pull_request) Waiting to run Details Test Supported Distributions / smoke-tests (pull_request) Waiting to run Details Test Supported Distributions / distro-test (centos_9) (pull_request) Blocked by required conditions Details Test Supported Distributions / distro-test (debian_11) (pull_request) Blocked by required conditions Details Test Supported Distributions / distro-test (debian_12) (pull_request) Blocked by required conditions Details Test Supported Distributions / distro-test (fedora_40) (pull_request) Blocked by required conditions Details Test Supported Distributions / distro-test (fedora_41) (pull_request) Blocked by required conditions Details Test Supported Distributions / distro-test (ubuntu_20) (pull_request) Blocked by required conditions Details Test Supported Distributions / distro-test (ubuntu_22) (pull_request) Blocked by required conditions Details Test Supported Distributions / distro-test (ubuntu_24) (pull_request) Blocked by required conditions Details Signed-off-by: DL6ER <dl6er@dl6er.de>	2024-12-23 08:52:56 +01:00
DL6ER	6cb8f55215	Merge branch 'fix/install' into fix/migration_test	2024-12-23 08:44:24 +01:00