Fix odd behaviour introduced into new version checking script (#5016)

Signed-off-by: Adam Warner <me@adamwarner.co.uk>

.github/workflows/test.yml

@@ -54,6 +54,7 @@ jobs:
             ubuntu_20,
             ubuntu_22,
             centos_8,
+            centos_9,
             fedora_35,
             fedora_36,
           ]
@@ -64,7 +65,7 @@ jobs:
         uses: actions/checkout@v3.1.0
 
       - name: Set up Python 3.10
-        uses: actions/setup-python@v4.2.0
+        uses: actions/setup-python@v4.3.0
         with:
           python-version: "3.10"
 

README.md

@@ -16,11 +16,11 @@
 
 The Pi-hole® is a [DNS sinkhole](https://en.wikipedia.org/wiki/DNS_Sinkhole) that protects your devices from unwanted content without installing any client-side software.
 
-- **Easy-to-install**: our versatile installer walks you through the process and takes less than ten minutes
+- **Easy-to-install**: our dialogs walk you through the simple installation process in less than ten minutes
 - **Resolute**: content is blocked in _non-browser locations_, such as ad-laden mobile apps and smart TVs
 - **Responsive**: seamlessly speeds up the feel of everyday browsing by caching DNS queries
 - **Lightweight**: runs smoothly with [minimal hardware and software requirements](https://docs.pi-hole.net/main/prerequisites/)
-- **Robust**: a command line interface that is quality assured for interoperability
+- **Robust**: a command-line interface that is quality assured for interoperability
 - **Insightful**: a beautiful responsive Web Interface dashboard to view and control your Pi-hole
 - **Versatile**: can optionally function as a [DHCP server](https://discourse.pi-hole.net/t/how-do-i-use-pi-holes-built-in-dhcp-server-and-why-would-i-want-to/3026), ensuring _all_ your devices are protected automatically
 - **Scalable**: [capable of handling hundreds of millions of queries](https://pi-hole.net/2017/05/24/how-much-traffic-can-pi-hole-handle/) when installed on server-grade hardware
@@ -60,7 +60,7 @@ Please refer to the [Pi-hole docker repo](https://github.com/pi-hole/docker-pi-h
 
 ## [Post-install: Make your network take advantage of Pi-hole](https://docs.pi-hole.net/main/post-install/)
 
-Once the installer has been run, you will need to [configure your router to have **DHCP clients use Pi-hole as their DNS server**](https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245) which ensures that all devices connecting to your network will have content blocked without any further intervention.
+Once the installer has been run, you will need to [configure your router to have **DHCP clients use Pi-hole as their DNS server**](https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245). This router configuration will ensure that all devices connecting to your network will have content blocked without any further intervention.
 
 If your router does not support setting the DNS server, you can [use Pi-hole's built-in DHCP server](https://discourse.pi-hole.net/t/how-do-i-use-pi-holes-built-in-dhcp-server-and-why-would-i-want-to/3026); be sure to disable DHCP on your router first (if it has that feature available).
 
@@ -70,7 +70,7 @@ As a last resort, you can manually set each device to use Pi-hole as their DNS s
 
 ## Pi-hole is free but powered by your support
 
-There are many reoccurring costs involved with maintaining free, open source, and privacy-respecting software; expenses which [our volunteer developers](https://github.com/orgs/pi-hole/people) pitch in to cover out-of-pocket. This is just one example of how strongly we feel about our software and the importance of keeping it maintained.
+There are many reoccurring costs involved with maintaining free, open-source, and privacy-respecting software; expenses which [our volunteer developers](https://github.com/orgs/pi-hole/people) pitch in to cover out-of-pocket. This is just one example of how strongly we feel about our software and the importance of keeping it maintained.
 
 Make no mistake: **your support is absolutely vital to help keep us innovating!**
 
@@ -132,9 +132,9 @@ Some of the statistics you can integrate include:
 
 Access the API via [`telnet`](https://github.com/pi-hole/FTL), the Web (`admin/api.php`) and Command Line (`pihole -c -j`). You can find out [more details over here](https://discourse.pi-hole.net/t/pi-hole-api/1863).
 
-### The Command Line Interface
+### The Command-Line Interface
 
-The [pihole](https://docs.pi-hole.net/core/pihole-command/) command has all the functionality necessary to fully administer the Pi-hole, without the need of the Web Interface. It's fast, user-friendly, and auditable by anyone with an understanding of `bash`.
+The [pihole](https://docs.pi-hole.net/core/pihole-command/) command has all the functionality necessary to fully administer the Pi-hole, without the need for the Web Interface. It's fast, user-friendly, and auditable by anyone with an understanding of `bash`.
 
 Some notable features include:
 

advanced/Scripts/piholeCheckout.sh

@@ -42,11 +42,6 @@ warning1() {
     esac
 }
 
-updateCheckFunc() {
-    /opt/pihole/updatecheck.sh
-    /opt/pihole/updatecheck.sh x remote
-}
-
 checkout() {
     local corebranches
     local webbranches
@@ -169,8 +164,8 @@ checkout() {
             exit 1
         fi
         checkout_pull_branch "${webInterfaceDir}" "${2}"
-        # Force an update of the updatechecker
-        updateCheckFunc
+        # Update local and remote versions via updatechecker
+        /opt/pihole/updatecheck.sh
     elif [[ "${1}" == "ftl" ]] ; then
         local path
         local oldbranch
@@ -185,8 +180,8 @@ checkout() {
             FTLinstall "${binary}"
             restart_service pihole-FTL
             enable_service pihole-FTL
-            # Force an update of the updatechecker
-            updateCheckFunc
+            # Update local and remote versions via updatechecker
+            /opt/pihole/updatecheck.sh
         else
             echo "  ${CROSS} Requested branch \"${2}\" is not available"
             ftlbranches=( $(git ls-remote https://github.com/pi-hole/ftl | grep 'heads' | sed 's/refs\/heads\///;s/ //g' | awk '{print $2}') )

advanced/Scripts/piholeDebug.sh

@@ -44,17 +44,12 @@ fi
 # shellcheck disable=SC1091
 . /etc/pihole/versions
 
-OBFUSCATED_PLACEHOLDER="<DOMAIN OBFUSCATED>"
-
 # FAQ URLs for use in showing the debug log
-FAQ_UPDATE_PI_HOLE="${COL_CYAN}https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249${COL_NC}"
-FAQ_CHECKOUT_COMMAND="${COL_CYAN}https://discourse.pi-hole.net/t/the-pihole-command-with-examples/738#checkout${COL_NC}"
 FAQ_HARDWARE_REQUIREMENTS="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/${COL_NC}"
 FAQ_HARDWARE_REQUIREMENTS_PORTS="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/#ports${COL_NC}"
 FAQ_HARDWARE_REQUIREMENTS_FIREWALLD="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/#firewalld${COL_NC}"
 FAQ_GATEWAY="${COL_CYAN}https://discourse.pi-hole.net/t/why-is-a-default-gateway-important-for-pi-hole/3546${COL_NC}"
 FAQ_FTL_COMPATIBILITY="${COL_CYAN}https://github.com/pi-hole/FTL#compatibility-list${COL_NC}"
-FAQ_BAD_ADDRESS="${COL_CYAN}https://discourse.pi-hole.net/t/why-do-i-see-bad-address-at-in-pihole-log/3972${COL_NC}"
 
 # Other URLs we may use
 FORUMS_URL="${COL_CYAN}https://discourse.pi-hole.net${COL_NC}"
@@ -73,7 +68,6 @@ WEB_SERVER_LOG_DIRECTORY="/var/log/lighttpd"
 WEB_SERVER_CONFIG_DIRECTORY="/etc/lighttpd"
 HTML_DIRECTORY="/var/www/html"
 WEB_GIT_DIRECTORY="${HTML_DIRECTORY}/admin"
-#BLOCK_PAGE_DIRECTORY="${HTML_DIRECTORY}/pihole"
 SHM_DIRECTORY="/dev/shm"
 ETC="/etc"
 
@@ -91,6 +85,7 @@ PIHOLE_LOGROTATE_FILE="${PIHOLE_DIRECTORY}/logrotate"
 PIHOLE_SETUP_VARS_FILE="${PIHOLE_DIRECTORY}/setupVars.conf"
 PIHOLE_FTL_CONF_FILE="${PIHOLE_DIRECTORY}/pihole-FTL.conf"
 PIHOLE_CUSTOM_HOSTS_FILE="${PIHOLE_DIRECTORY}/custom.list"
+PIHOLE_VERSIONS_FILE="${PIHOLE_DIRECTORY}/versions"
 
 # Read the value of an FTL config key. The value is printed to stdout.
 #
@@ -162,7 +157,8 @@ REQUIRED_FILES=("${PIHOLE_CRON_FILE}"
 "${PIHOLE_WEB_SERVER_ERROR_LOG_FILE}"
 "${RESOLVCONF}"
 "${DNSMASQ_CONF}"
-"${PIHOLE_CUSTOM_HOSTS_FILE}")
+"${PIHOLE_CUSTOM_HOSTS_FILE}"
+"${PIHOLE_VERSIONS_FILE}")
 
 DISCLAIMER="This process collects information from your Pi-hole, and optionally uploads it to a unique and random directory on tricorder.pi-hole.net.
 
@@ -240,15 +236,7 @@ compare_local_version_to_git_version() {
     local git_dir="${1}"
     # The named component of the project (Core or Web)
     local pihole_component="${2}"
-    # If we are checking the Core versions,
-    if [[ "${pihole_component}" == "Core" ]]; then
-        # We need to search for "Pi-hole" when using pihole -v
-        local search_term="Pi-hole"
-    elif [[ "${pihole_component}" == "Web" ]]; then
-        # We need to search for "AdminLTE" so store it in a variable as well
-        #shellcheck disable=2034
-        local search_term="AdminLTE"
-    fi
+
     # Display what we are checking
     echo_current_diagnostic "${pihole_component} version"
     # Store the error message in a variable in case we want to change and/or reuse it
@@ -261,43 +249,35 @@ compare_local_version_to_git_version() {
         log_write "${COL_RED}Could not cd into ${git_dir}$COL_NC"
         if git status &> /dev/null; then
             # The current version the user is on
-            local remote_version
-            remote_version=$(git describe --tags --abbrev=0);
+            local local_version
+            local_version=$(git describe --tags --abbrev=0);
             # What branch they are on
-            local remote_branch
-            remote_branch=$(git rev-parse --abbrev-ref HEAD);
+            local local_branch
+            local_branch=$(git rev-parse --abbrev-ref HEAD);
             # The commit they are on
-            local remote_commit
-            remote_commit=$(git describe --long --dirty --tags --always)
+            local local_commit
+            local_commit=$(git describe --long --dirty --tags --always)
             # Status of the repo
             local local_status
             local_status=$(git status -s)
             # echo this information out to the user in a nice format
-            # If the current version matches what pihole -v produces, the user is up-to-date
-            if [[ "${remote_version}" == "$(pihole -v | awk '/${search_term}/ {print $6}' | cut -d ')' -f1)" ]]; then
-                log_write "${TICK} ${pihole_component}: ${COL_GREEN}${remote_version}${COL_NC}"
-            # If not,
-            else
-                # echo the current version in yellow, signifying it's something to take a look at, but not a critical error
-                # Also add a URL to an FAQ
-                log_write "${INFO} ${pihole_component}: ${COL_YELLOW}${remote_version:-Untagged}${COL_NC} (${FAQ_UPDATE_PI_HOLE})"
-            fi
+            log_write "${TICK} Version: ${local_version}"
 
             # Print the repo upstreams
             remotes=$(git remote -v)
             log_write "${INFO} Remotes: ${remotes//$'\n'/'\n             '}"
 
             # If the repo is on the master branch, they are on the stable codebase
-            if [[ "${remote_branch}" == "master" ]]; then
+            if [[ "${local_branch}" == "master" ]]; then
                 # so the color of the text is green
-                log_write "${INFO} Branch: ${COL_GREEN}${remote_branch}${COL_NC}"
+                log_write "${INFO} Branch: ${COL_GREEN}${local_branch}${COL_NC}"
             # If it is any other branch, they are in a development branch
             else
                 # So show that in yellow, signifying it's something to take a look at, but not a critical error
-                log_write "${INFO} Branch: ${COL_YELLOW}${remote_branch:-Detached}${COL_NC} (${FAQ_CHECKOUT_COMMAND})"
+                log_write "${INFO} Branch: ${COL_YELLOW}${local_branch:-Detached}${COL_NC}"
             fi
             # echo the current commit
-            log_write "${INFO} Commit: ${remote_commit}"
+            log_write "${INFO} Commit: ${local_commit}"
             # if `local_status` is non-null, then the repo is not clean, display details here
             if [[ ${local_status} ]]; then
               # Replace new lines in the status with 12 spaces to make the output cleaner
@@ -331,22 +311,15 @@ compare_local_version_to_git_version() {
 }
 
 check_ftl_version() {
-    local ftl_name="FTL"
     local FTL_VERSION FTL_COMMIT FTL_BRANCH
-    echo_current_diagnostic "${ftl_name} version"
+    echo_current_diagnostic "FTL version"
     # Use the built in command to check FTL's version
-    FTL_VERSION=$(pihole-FTL -vv | grep -m 1 Version | awk '{printf $2}')
-    FTL_BRANCH=$(pihole-FTL -vv | grep -m 1 Branch | awk '{printf $2}')
-    FTL_COMMIT=$(pihole-FTL -vv | grep -m 1 Commit | awk '{printf $2}')
+    FTL_VERSION=$(pihole-FTL version)
+    FTL_BRANCH=$(pihole-FTL branch)
+    FTL_COMMIT=$(pihole-FTL --hash)
 
-    # Compare the current FTL version to the remote version
-    if [[ "${FTL_VERSION}" == "$(pihole -v | awk '/FTL/ {print $6}' | cut -d ')' -f1)" ]]; then
-        # If they are the same, FTL is up-to-date
-        log_write "${TICK} ${ftl_name}: ${COL_GREEN}${FTL_VERSION}${COL_NC}"
-    else
-        # If not, show it in yellow, signifying there is an update
-        log_write "${INFO} ${ftl_name}: ${COL_YELLOW}${FTL_VERSION}${COL_NC} (${FAQ_UPDATE_PI_HOLE})"
-    fi
+
+    log_write "${TICK} Version: ${FTL_VERSION}"
 
     # If they use the master branch, they are on the stable codebase
     if [[ "${FTL_BRANCH}" == "master" ]]; then
@@ -355,7 +328,7 @@ check_ftl_version() {
         # If it is any other branch, they are in a development branch
     else
         # So show that in yellow, signifying it's something to take a look at, but not a critical error
-        log_write "${INFO} Branch: ${COL_YELLOW}${FTL_BRANCH}${COL_NC} (${FAQ_CHECKOUT_COMMAND})"
+        log_write "${INFO} Branch: ${COL_YELLOW}${FTL_BRANCH}${COL_NC}"
     fi
 
     # echo the current commit
@@ -952,10 +925,21 @@ process_status(){
         else
             # Otherwise, use the service command and mock the output of `systemctl is-active`
             local status_of_process
-            if service "${i}" status | grep -E 'is\srunning' &> /dev/null; then
-                status_of_process="active"
+
+            # If DOCKER_VERSION is set, the output is slightly different (s6 init system on Docker)
+            if [ -n "${DOCKER_VERSION}" ]; then
+                if service "${i}" status | grep -E '^up' &> /dev/null; then
+                    status_of_process="active"
+                else
+                    status_of_process="inactive"
+                fi
             else
-                status_of_process="inactive"
+            # non-Docker system
+                if service "${i}" status | grep -E 'is\srunning' &> /dev/null; then
+                    status_of_process="active"
+                else
+                    status_of_process="inactive"
+                fi
             fi
         fi
         # and print it out to the user
@@ -1381,49 +1365,8 @@ spinner(){
     fi
 }
 
-obfuscated_pihole_log() {
-  local pihole_log=("$@")
-  local line
-  local error_to_check_for
-  local line_to_obfuscate
-  local obfuscated_line
-  for line in "${pihole_log[@]}"; do
-      # A common error in the pihole.log is when there is a non-hosts formatted file
-      # that the DNS server is attempting to read.  Since it's not formatted
-      # correctly, there will be an entry for "bad address at line n"
-      # So we can check for that here and highlight it in red so the user can see it easily
-      error_to_check_for=$(echo "${line}" | grep 'bad address at')
-      # Some users may not want to have the domains they visit sent to us
-      # To that end, we check for lines in the log that would contain a domain name
-      line_to_obfuscate=$(echo "${line}" | grep ': query\|: forwarded\|: reply')
-      # If the variable contains a value, it found an error in the log
-      if [[ -n ${error_to_check_for} ]]; then
-          # So we can print it in red to make it visible to the user
-          log_write "   ${CROSS} ${COL_RED}${line}${COL_NC} (${FAQ_BAD_ADDRESS})"
-      else
-          # If the variable does not a value (the current default behavior), so do not obfuscate anything
-          if [[ -z ${OBFUSCATE} ]]; then
-              log_write "   ${line}"
-          # Otherwise, a flag was passed to this command to obfuscate domains in the log
-          else
-              # So first check if there are domains in the log that should be obfuscated
-              if [[ -n ${line_to_obfuscate} ]]; then
-                  # If there are, we need to use awk to replace only the domain name (the 6th field in the log)
-                  # so we substitute the domain for the placeholder value
-                  obfuscated_line=$(echo "${line_to_obfuscate}" | awk -v placeholder="${OBFUSCATED_PLACEHOLDER}" '{sub($6,placeholder); print $0}')
-                  log_write "   ${obfuscated_line}"
-              else
-                  log_write "   ${line}"
-              fi
-          fi
-      fi
-  done
-}
-
 analyze_pihole_log() {
   echo_current_diagnostic "Pi-hole log"
-  local pihole_log_head=()
-  local pihole_log_tail=()
   local pihole_log_permissions
   local logging_enabled
 
@@ -1433,22 +1376,10 @@ analyze_pihole_log() {
       log_write "${INFO} Query logging is disabled"
       log_write ""
   fi
-  # Put the current Internal Field Separator into another variable so it can be restored later
-  OLD_IFS="$IFS"
-  # Get the lines that are in the file(s) and store them in an array for parsing later
-  IFS=$'\r\n'
+
   pihole_log_permissions=$(ls -lhd "${PIHOLE_LOG}")
   log_write "${COL_GREEN}${pihole_log_permissions}${COL_NC}"
-  mapfile -t pihole_log_head < <(head -n 20 ${PIHOLE_LOG})
-  log_write "   ${COL_CYAN}-----head of $(basename ${PIHOLE_LOG})------${COL_NC}"
-  obfuscated_pihole_log "${pihole_log_head[@]}"
-  log_write ""
-  mapfile -t pihole_log_tail < <(tail -n 20 ${PIHOLE_LOG})
-  log_write "   ${COL_CYAN}-----tail of $(basename ${PIHOLE_LOG})------${COL_NC}"
-  obfuscated_pihole_log "${pihole_log_tail[@]}"
-  log_write ""
-  # Set the IFS back to what it was
-  IFS="$OLD_IFS"
+  head_tail_log "${PIHOLE_LOG}" 20
 }
 
 curl_to_tricorder() {

advanced/Scripts/update.sh

@@ -216,9 +216,8 @@ main() {
     fi
 
     if [[ "${FTL_update}" == true || "${core_update}" == true || "${web_update}" == true ]]; then
-        # Force an update of the updatechecker
+        # Update local and remote versions via updatechecker
         /opt/pihole/updatecheck.sh
-        /opt/pihole/updatecheck.sh x remote
         echo -e "  ${INFO} Local version file information updated."
     fi
 

advanced/Scripts/updatecheck.sh

@@ -15,16 +15,30 @@ function get_local_branch() {
 }
 
 function get_local_version() {
-    # Return active branch
+    # Return active version
     cd "${1}" 2> /dev/null || return 1
-    git describe --long --dirty --tags 2> /dev/null || return 1
+    git describe --tags --always 2> /dev/null || return 1
+}
+
+function get_local_hash() {
+    cd "${1}" 2> /dev/null || return 1
+    git rev-parse --short HEAD || return 1
+}
+
+function get_remote_version() {
+    curl -s "https://api.github.com/repos/pi-hole/${1}/releases/latest" 2> /dev/null | jq --raw-output .tag_name || return 1
+}
+
+
+function get_remote_hash(){
+    git ls-remote "https://github.com/pi-hole/${1}" --tags "${2}" | awk '{print substr($0, 0,8);}' || return 1
 }
 
 # Source the setupvars config file
 # shellcheck disable=SC1091
 . /etc/pihole/setupVars.conf
 
-# Source the utils file
+# Source the utils file for addOrEditKeyValPair()
 # shellcheck disable=SC1091
 . /opt/pihole/utils.sh
 
@@ -46,54 +60,74 @@ if [[ ! "${DOCKER_TAG}" =~ $regex ]]; then
   unset DOCKER_TAG
 fi
 
-if [[ "$2" == "remote" ]]; then
-
-    if [[ "$3" == "reboot" ]]; then
+# used in cronjob
+if [[ "$1" == "reboot" ]]; then
         sleep 30
-    fi
+fi
 
-    GITHUB_CORE_VERSION="$(curl -s 'https://api.github.com/repos/pi-hole/pi-hole/releases/latest' 2> /dev/null | jq --raw-output .tag_name)"
-    addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_CORE_VERSION" "${GITHUB_CORE_VERSION}"
 
-    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
-        GITHUB_WEB_VERSION="$(curl -s 'https://api.github.com/repos/pi-hole/AdminLTE/releases/latest' 2> /dev/null | jq --raw-output .tag_name)"
-        addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_WEB_VERSION" "${GITHUB_WEB_VERSION}"
-    fi
+# get Core versions
 
-    GITHUB_FTL_VERSION="$(curl -s 'https://api.github.com/repos/pi-hole/FTL/releases/latest' 2> /dev/null | jq --raw-output .tag_name)"
-    addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_FTL_VERSION" "${GITHUB_FTL_VERSION}"
+CORE_VERSION="$(get_local_version /etc/.pihole)"
+addOrEditKeyValPair "${VERSION_FILE}" "CORE_VERSION" "${CORE_VERSION}"
 
-    if [[ "${DOCKER_TAG}" ]]; then
-        GITHUB_DOCKER_VERSION="$(curl -s 'https://api.github.com/repos/pi-hole/docker-pi-hole/releases/latest' 2> /dev/null | jq --raw-output .tag_name)"
-        addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_DOCKER_VERSION" "${GITHUB_DOCKER_VERSION}"
-    fi
+CORE_BRANCH="$(get_local_branch /etc/.pihole)"
+addOrEditKeyValPair "${VERSION_FILE}" "CORE_BRANCH" "${CORE_BRANCH}"
 
-else
+CORE_HASH="$(get_local_hash /etc/.pihole)"
+addOrEditKeyValPair "${VERSION_FILE}" "CORE_HASH" "${CORE_HASH}"
 
-    CORE_BRANCH="$(get_local_branch /etc/.pihole)"
-    addOrEditKeyValPair "${VERSION_FILE}" "CORE_BRANCH" "${CORE_BRANCH}"
+GITHUB_CORE_VERSION="$(get_remote_version pi-hole)"
+addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_CORE_VERSION" "${GITHUB_CORE_VERSION}"
 
-    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
-        WEB_BRANCH="$(get_local_branch /var/www/html/admin)"
-        addOrEditKeyValPair "${VERSION_FILE}" "WEB_BRANCH" "${WEB_BRANCH}"
-    fi
+GITHUB_CORE_HASH="$(get_remote_hash pi-hole "${CORE_BRANCH}")"
+addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_CORE_HASH" "${GITHUB_CORE_HASH}"
 
-    FTL_BRANCH="$(pihole-FTL branch)"
-    addOrEditKeyValPair "${VERSION_FILE}" "FTL_BRANCH" "${FTL_BRANCH}"
 
-    CORE_VERSION="$(get_local_version /etc/.pihole)"
-    addOrEditKeyValPair "${VERSION_FILE}" "CORE_VERSION" "${CORE_VERSION}"
+# get Web versions
 
-    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
-        WEB_VERSION="$(get_local_version /var/www/html/admin)"
-        addOrEditKeyValPair "${VERSION_FILE}" "WEB_VERSION" "${WEB_VERSION}"
-    fi
+if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
 
-    FTL_VERSION="$(pihole-FTL version)"
-    addOrEditKeyValPair "${VERSION_FILE}" "FTL_VERSION" "${FTL_VERSION}"
+    WEB_VERSION="$(get_local_version /var/www/html/admin)"
+    addOrEditKeyValPair "${VERSION_FILE}" "WEB_VERSION" "${WEB_VERSION}"
 
-    if [[ "${DOCKER_TAG}" ]]; then
-        addOrEditKeyValPair "${VERSION_FILE}" "DOCKER_VERSION" "${DOCKER_TAG}"
-    fi
+    WEB_BRANCH="$(get_local_branch /var/www/html/admin)"
+    addOrEditKeyValPair "${VERSION_FILE}" "WEB_BRANCH" "${WEB_BRANCH}"
+
+    WEB_HASH="$(get_local_hash /var/www/html/admin)"
+    addOrEditKeyValPair "${VERSION_FILE}" "WEB_HASH" "${WEB_HASH}"
+
+    GITHUB_WEB_VERSION="$(get_remote_version AdminLTE)"
+    addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_WEB_VERSION" "${GITHUB_WEB_VERSION}"
+
+    GITHUB_WEB_HASH="$(get_remote_hash AdminLTE "${WEB_BRANCH}")"
+    addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_WEB_HASH" "${GITHUB_WEB_HASH}"
 
 fi
+
+# get FTL versions
+
+FTL_VERSION="$(pihole-FTL version)"
+addOrEditKeyValPair "${VERSION_FILE}" "FTL_VERSION" "${FTL_VERSION}"
+
+FTL_BRANCH="$(pihole-FTL branch)"
+addOrEditKeyValPair "${VERSION_FILE}" "FTL_BRANCH" "${FTL_BRANCH}"
+
+FTL_HASH="$(pihole-FTL --hash)"
+addOrEditKeyValPair "${VERSION_FILE}" "FTL_HASH" "${FTL_HASH}"
+
+GITHUB_FTL_VERSION="$(get_remote_version FTL)"
+addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_FTL_VERSION" "${GITHUB_FTL_VERSION}"
+
+GITHUB_FTL_HASH="$(get_remote_hash FTL "${FTL_BRANCH}")"
+addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_FTL_HASH" "${GITHUB_FTL_HASH}"
+
+
+# get Docker versions
+
+if [[ "${DOCKER_TAG}" ]]; then
+    addOrEditKeyValPair "${VERSION_FILE}" "DOCKER_VERSION" "${DOCKER_TAG}"
+
+    GITHUB_DOCKER_VERSION="$(get_remote_version docker-pi-hole)"
+    addOrEditKeyValPair "${VERSION_FILE}" "GITHUB_DOCKER_VERSION" "${GITHUB_DOCKER_VERSION}"
+fi

advanced/Scripts/utils.sh

@@ -31,6 +31,9 @@ addOrEditKeyValPair() {
   local key="${2}"
   local value="${3}"
 
+  # touch file to prevent grep error if file does not exist yet
+  touch "${file}"
+
   if grep -q "^${key}=" "${file}"; then
     # Key already exists in file, modify the value
     sed -i "/^${key}=/c\\${key}=${value}" "${file}"
@@ -51,6 +54,9 @@ addKey(){
   local file="${1}"
   local key="${2}"
 
+  # touch file to prevent grep error if file does not exist yet
+  touch "${file}"
+
   if ! grep -q "^${key}" "${file}"; then
     # Key does not exist, add it.
     echo "${key}" >> "${file}"

advanced/Scripts/version.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env sh
 # Pi-hole: A black hole for Internet advertisements
 # (c) 2017 Pi-hole, LLC (https://pi-hole.net)
 # Network-wide ad blocking via your own hardware.
@@ -8,179 +8,104 @@
 # This file is copyright under the latest version of the EUPL.
 # Please see LICENSE file for your rights under this license.
 
-# Variables
-DEFAULT="-1"
-COREGITDIR="/etc/.pihole/"
-WEBGITDIR="/var/www/html/admin/"
-
 # Source the setupvars config file
 # shellcheck disable=SC1091
-source /etc/pihole/setupVars.conf
+. /etc/pihole/setupVars.conf
+
+# Source the versions file poupulated by updatechecker.sh
+cachedVersions="/etc/pihole/versions"
+
+if [ -f ${cachedVersions} ]; then
+    # shellcheck disable=SC1090
+    . "$cachedVersions"
+else
+    echo "Could not find /etc/pihole/versions. Running update now."
+    pihole updatechecker
+    # shellcheck disable=SC1090
+    . "$cachedVersions"
+fi
 
 getLocalVersion() {
-    # FTL requires a different method
-    if [[ "$1" == "FTL" ]]; then
-        pihole-FTL version
-        return 0
-    fi
-
-    # Get the tagged version of the local repository
-    local directory="${1}"
-    local version
-
-    cd "${directory}" 2> /dev/null || { echo "${DEFAULT}"; return 1; }
-    version=$(git describe --tags --always || echo "$DEFAULT")
-    if [[ "${version}" =~ ^v ]]; then
-        echo "${version}"
-    elif [[ "${version}" == "${DEFAULT}" ]]; then
-        echo "ERROR"
-        return 1
-    else
-        echo "Untagged"
-    fi
-    return 0
+    case ${1} in
+        "Pi-hole"   )  echo "${CORE_VERSION:=N/A}";;
+        "AdminLTE"  )  [ "${INSTALL_WEB_INTERFACE}" = true ] && echo "${WEB_VERSION:=N/A}";;
+        "FTL"       )  echo "${FTL_VERSION:=N/A}";;
+    esac
 }
 
 getLocalHash() {
-    # Local FTL hash does not exist on filesystem
-    if [[ "$1" == "FTL" ]]; then
-        echo "N/A"
-        return 0
-    fi
-
-    # Get the short hash of the local repository
-    local directory="${1}"
-    local hash
-
-    cd "${directory}" 2> /dev/null || { echo "${DEFAULT}"; return 1; }
-    hash=$(git rev-parse --short HEAD || echo "$DEFAULT")
-    if [[ "${hash}" == "${DEFAULT}" ]]; then
-        echo "ERROR"
-        return 1
-    else
-        echo "${hash}"
-    fi
-    return 0
+    case ${1} in
+        "Pi-hole"   )  echo "${CORE_HASH:=N/A}";;
+        "AdminLTE"  )  [ "${INSTALL_WEB_INTERFACE}" = true ] && echo "${WEB_HASH:=N/A}";;
+        "FTL"       )  echo "${FTL_HASH:=N/A}";;
+    esac
 }
 
 getRemoteHash(){
-    # Remote FTL hash is not applicable
-    if [[ "$1" == "FTL" ]]; then
-        echo "N/A"
-        return 0
-    fi
-
-    local daemon="${1}"
-    local branch="${2}"
-
-    hash=$(git ls-remote --heads "https://github.com/pi-hole/${daemon}" | \
-        awk -v bra="$branch" '$0~bra {print substr($0,0,8);exit}')
-    if [[ -n "$hash" ]]; then
-        echo "$hash"
-    else
-        echo "ERROR"
-        return 1
-    fi
-    return 0
+    case ${1} in
+        "Pi-hole"   )  echo "${GITHUB_CORE_HASH:=N/A}";;
+        "AdminLTE"  )  [ "${INSTALL_WEB_INTERFACE}" = true ] && echo "${GITHUB_WEB_HASH:=N/A}";;
+        "FTL"       )  echo "${GITHUB_FTL_HASH:=N/A}";;
+    esac
 }
 
 getRemoteVersion(){
-    # Get the version from the remote origin
-    local daemon="${1}"
-    local version
-    local cachedVersions
-    cachedVersions="/etc/pihole/versions"
-
-    #If the above file exists, then we can read from that. Prevents overuse of GitHub API
-    if [[ -f "$cachedVersions" ]]; then
-
-        # shellcheck disable=SC1090
-        . "$cachedVersions"
-
-        case $daemon in
-            "pi-hole"   )  echo "${GITHUB_CORE_VERSION}";;
-            "AdminLTE"  )  [[ "${INSTALL_WEB_INTERFACE}" == true ]] && echo "${GITHUB_WEB_VERSION}";;
-            "FTL"       )  echo "${GITHUB_FTL_VERSION}";;
-        esac
-
-        return 0
-    fi
-
-    version=$(curl --silent --fail "https://api.github.com/repos/pi-hole/${daemon}/releases/latest" | \
-        awk -F: '$1 ~/tag_name/ { print $2 }' | \
-        tr -cd '[[:alnum:]]._-')
-    if [[ "${version}" =~ ^v ]]; then
-        echo "${version}"
-    else
-        echo "ERROR"
-        return 1
-    fi
-    return 0
+    case ${1} in
+        "Pi-hole"   )  echo "${GITHUB_CORE_VERSION:=N/A}";;
+        "AdminLTE"  )  [ "${INSTALL_WEB_INTERFACE}" = true ] && echo "${GITHUB_WEB_VERSION:=N/A}";;
+        "FTL"       )  echo "${GITHUB_FTL_VERSION:=N/A}";;
+    esac
 }
 
 getLocalBranch(){
-    # Get the checked out branch of the local directory
-    local directory="${1}"
-    local branch
-
-    # Local FTL btranch is stored in /etc/pihole/ftlbranch
-    if [[ "$1" == "FTL" ]]; then
-        branch="$(pihole-FTL branch)"
-    else
-        cd "${directory}" 2> /dev/null || { echo "${DEFAULT}"; return 1; }
-        branch=$(git rev-parse --abbrev-ref HEAD || echo "$DEFAULT")
-    fi
-    if [[ ! "${branch}" =~ ^v ]]; then
-        if [[ "${branch}" == "master" ]]; then
-            echo ""
-        elif [[ "${branch}" == "HEAD" ]]; then
-            echo "in detached HEAD state at "
-        else
-            echo "${branch} "
-        fi
-    else
-        # Branch started in "v"
-        echo "release "
-    fi
-    return 0
+    case ${1} in
+        "Pi-hole"   )  echo "${CORE_BRANCH:=N/A}";;
+        "AdminLTE"  )  [ "${INSTALL_WEB_INTERFACE}" = true ] && echo "${WEB_BRANCH:=N/A}";;
+        "FTL"       )  echo "${FTL_BRANCH:=N/A}";;
+    esac
 }
 
 versionOutput() {
-    if [[ "$1" == "AdminLTE" && "${INSTALL_WEB_INTERFACE}" != true ]]; then
+    if [ "$1" = "AdminLTE" ] && [ "${INSTALL_WEB_INTERFACE}" != true ]; then
         echo "  WebAdmin not installed"
         return 1
     fi
 
-    [[ "$1" == "pi-hole" ]] && GITDIR=$COREGITDIR
-    [[ "$1" == "AdminLTE" ]] && GITDIR=$WEBGITDIR
-    [[ "$1" == "FTL" ]] && GITDIR="FTL"
-
-    [[ "$2" == "-c" ]] || [[ "$2" == "--current" ]] || [[ -z "$2" ]] && current=$(getLocalVersion $GITDIR) && branch=$(getLocalBranch $GITDIR)
-    [[ "$2" == "-l" ]] || [[ "$2" == "--latest" ]] || [[ -z "$2" ]] && latest=$(getRemoteVersion "$1")
-    if [[ "$2" == "-h" ]] || [[ "$2" == "--hash" ]]; then
-        [[ "$3" == "-c" ]] || [[ "$3" == "--current" ]] || [[ -z "$3" ]] && curHash=$(getLocalHash "$GITDIR") && branch=$(getLocalBranch $GITDIR)
-        [[ "$3" == "-l" ]] || [[ "$3" == "--latest" ]] || [[ -z "$3" ]] && latHash=$(getRemoteHash "$1" "$(cd "$GITDIR" 2> /dev/null && git rev-parse --abbrev-ref HEAD)")
+    [ "$2" = "-c" ] || [ "$2" = "--current" ] || [ -z "$2" ] && current=$(getLocalVersion "${1}") && branch=$(getLocalBranch "${1}")
+    [ "$2" = "-l" ] || [ "$2" = "--latest" ] || [ -z "$2" ] && latest=$(getRemoteVersion "${1}")
+    if [ "$2" = "--hash" ]; then
+        [ "$3" = "-c" ] || [ "$3" = "--current" ] || [ -z "$3" ] && curHash=$(getLocalHash "${1}") && branch=$(getLocalBranch "${1}")
+        [ "$3" = "-l" ] || [ "$3" = "--latest" ] || [ -z "$3" ] && latHash=$(getRemoteHash "${1}") && branch=$(getLocalBranch "${1}")
     fi
-    if [[ -n "$current" ]] && [[ -n "$latest" ]]; then
-        output="${1^} version is $branch$current (Latest: $latest)"
-    elif [[ -n "$current" ]] && [[ -z "$latest" ]]; then
-        output="Current ${1^} version is $branch$current"
-    elif [[ -z "$current" ]] && [[ -n "$latest" ]]; then
-        output="Latest ${1^} version is $latest"
-    elif [[ "$curHash" == "N/A" ]] || [[ "$latHash" == "N/A" ]]; then
-        output="${1^} hash is not applicable"
-    elif [[ -n "$curHash" ]] && [[ -n "$latHash" ]]; then
-        output="${1^} hash is $curHash (Latest: $latHash)"
-    elif [[ -n "$curHash" ]] && [[ -z "$latHash" ]]; then
-        output="Current ${1^} hash is $curHash"
-    elif [[ -z "$curHash" ]] && [[ -n "$latHash" ]]; then
-        output="Latest ${1^} hash is $latHash"
+
+    # We do not want to show the branch name when we are on master,
+    # blank out the variable in this case
+    if [ "$branch" = "master" ]; then
+        branch=""
+    else
+        branch="$branch "
+    fi
+
+    if [ -n "$current" ] && [ -n "$latest" ]; then
+        output="${1} version is $branch$current (Latest: $latest)"
+    elif [ -n "$current" ] && [ -z "$latest" ]; then
+        output="Current ${1} version is $branch$current"
+    elif [ -z "$current" ] && [ -n "$latest" ]; then
+        output="Latest ${1} version is $latest"
+    elif [ -n "$curHash" ] && [ -n "$latHash" ]; then
+        output="Local ${1} hash is $curHash (Remote: $latHash)"
+    elif [ -n "$curHash" ] && [ -z "$latHash" ]; then
+        output="Current local ${1} hash is $curHash"
+    elif [ -z "$curHash" ] && [ -n "$latHash" ]; then
+        output="Latest remote ${1} hash is $latHash"
+    elif [ -z "$curHash" ] && [ -z "$latHash" ]; then
+        output="Hashes for ${1} not available"
     else
         errorOutput
         return 1
     fi
 
-    [[ -n "$output" ]] && echo "  $output"
+    [ -n "$output" ] && echo "  $output"
 }
 
 errorOutput() {
@@ -189,9 +114,9 @@ errorOutput() {
 }
 
 defaultOutput() {
-    versionOutput "pi-hole" "$@"
+    versionOutput "Pi-hole" "$@"
 
-    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
+    if [ "${INSTALL_WEB_INTERFACE}" = true ]; then
         versionOutput "AdminLTE" "$@"
     fi
 
@@ -217,7 +142,7 @@ Options:
 }
 
 case "${1}" in
-    "-p" | "--pihole"    ) shift; versionOutput "pi-hole" "$@";;
+    "-p" | "--pihole"    ) shift; versionOutput "Pi-hole" "$@";;
     "-a" | "--admin"     ) shift; versionOutput "AdminLTE" "$@";;
     "-f" | "--ftl"       ) shift; versionOutput "FTL" "$@";;
     "-h" | "--help"      ) helpFunc;;

advanced/Scripts/webpage.sh

@@ -627,6 +627,14 @@ checkDomain()
     echo "${validDomain}"
 }
 
+escapeDots()
+{
+    # SC suggest bashism ${variable//search/replace}
+    # shellcheck disable=SC2001
+    escaped=$(echo "$1" | sed 's/\./\\./g')
+    echo "${escaped}"
+}
+
 addAudit()
 {
     shift # skip "-a"
@@ -702,6 +710,7 @@ RemoveCustomDNSAddress() {
     validHost="$(checkDomain "${host}")"
     if [[ -n "${validHost}" ]]; then
         if valid_ip "${ip}" || valid_ip6 "${ip}" ; then
+            validHost=$(escapeDots "${validHost}")
             sed -i "/^${ip} ${validHost}$/Id" "${dnscustomfile}"
         else
             echo -e "  ${CROSS} Invalid IP has been passed"
@@ -729,7 +738,12 @@ AddCustomCNAMERecord() {
     if [[ -n "${validDomain}" ]]; then
         validTarget="$(checkDomain "${target}")"
         if [[ -n "${validTarget}" ]]; then
-            echo "cname=${validDomain},${validTarget}" >> "${dnscustomcnamefile}"
+            if [ "${validDomain}" = "${validTarget}" ]; then
+                echo "  ${CROSS} Domain and target are the same. This would cause a DNS loop."
+                exit 1
+            else
+                echo "cname=${validDomain},${validTarget}" >> "${dnscustomcnamefile}"
+            fi
         else
             echo "  ${CROSS} Invalid Target Passed!"
             exit 1
@@ -755,7 +769,9 @@ RemoveCustomCNAMERecord() {
     if [[ -n "${validDomain}" ]]; then
         validTarget="$(checkDomain "${target}")"
         if [[ -n "${validTarget}" ]]; then
-            sed -i "/cname=${validDomain},${validTarget}$/Id" "${dnscustomcnamefile}"
+            validDomain=$(escapeDots "${validDomain}")
+            validTarget=$(escapeDots "${validTarget}")
+            sed -i "/^cname=${validDomain},${validTarget}$/Id" "${dnscustomcnamefile}"
         else
             echo "  ${CROSS} Invalid Target Passed!"
             exit 1

advanced/Templates/pihole.cron

@@ -28,6 +28,6 @@
 
 @reboot root /usr/sbin/logrotate --state /var/lib/logrotate/pihole /etc/pihole/logrotate
 
-# Pi-hole: Grab remote version every 24 hours
-59 17  * * *   root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker remote
-@reboot root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker remote reboot
+# Pi-hole: Grab remote and local version every 24 hours
+59 17  * * *   root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker
+@reboot root    PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker reboot

automated install/basic-install.sh

@@ -358,7 +358,7 @@ package_manager_detect() {
         # These variable names match the ones for apt-get. See above for an explanation of what they are for.
         PKG_INSTALL=("${PKG_MANAGER}" install -y)
         # CentOS package manager returns 100 when there are packages to update so we need to || true to prevent the script from exiting.
-        PKG_COUNT="${PKG_MANAGER} check-update | egrep '(.i686|.x86|.noarch|.arm|.src)' | wc -l || true"
+        PKG_COUNT="${PKG_MANAGER} check-update | grep -E '(.i686|.x86|.noarch|.arm|.src)' | wc -l || true"
         OS_CHECK_DEPS=(grep bind-utils)
         INSTALLER_DEPS=(git dialog iproute newt procps-ng which chkconfig ca-certificates)
         PIHOLE_DEPS=(cronie curl findutils sudo unzip libidn2 psmisc libcap nmap-ncat jq)
@@ -828,8 +828,11 @@ It is also possible to use a DHCP reservation, but if you are going to do that,
 
 # Configure networking via dhcpcd
 setDHCPCD() {
-    # Check if the IP is already in the file
-    if grep -q "${IPV4_ADDRESS}" /etc/dhcpcd.conf; then
+    # Regex for matching a non-commented static ip address setting
+    local regex="^[ \t]*static ip_address[ \t]*=[ \t]*${IPV4_ADDRESS}"
+
+    # Check if static IP is already set in file
+    if grep -q "${regex}" /etc/dhcpcd.conf; then
         printf "  %b Static IP already configured\\n" "${INFO}"
     # If it's not,
     else
@@ -2267,7 +2270,7 @@ get_binary_name() {
         local rev
         rev=$(uname -m | sed "s/[^0-9]//g;")
         local lib
-        lib=$(ldd "$(which sh)" | grep -E '^\s*/lib' | awk '{ print $1 }')
+        lib=$(ldd "$(command -v sh)" | grep -E '^\s*/lib' | awk '{ print $1 }')
         if [[ "${lib}" == "/lib/ld-linux-aarch64.so.1" ]]; then
             printf "%b  %b Detected AArch64 (64 Bit ARM) processor\\n" "${OVER}" "${TICK}"
             # set the binary to be used
@@ -2693,9 +2696,8 @@ main() {
     # Download and compile the aggregated block list
     runGravity
 
-    # Force an update of the updatechecker
+    # Update local and remote versions via updatechecker
     /opt/pihole/updatecheck.sh
-    /opt/pihole/updatecheck.sh x remote
 
     if [[ "${useUpdateVars}" == false ]]; then
         displayFinalMessage "${pw}"

gravity.sh

@@ -720,72 +720,25 @@ gravity_DownloadBlocklistFromUrl() {
 
 # Parse source files into domains format
 gravity_ParseFileIntoDomains() {
-  local src="${1}" destination="${2}" firstLine
+  local src="${1}" destination="${2}"
 
-  # Determine if we are parsing a consolidated list
-  #if [[ "${src}" == "${piholeDir}/${matterAndLight}" ]]; then
-    # Remove comments and print only the domain name
-    # Most of the lists downloaded are already in hosts file format but the spacing/formatting is not contiguous
-    # This helps with that and makes it easier to read
-    # It also helps with debugging so each stage of the script can be researched more in depth
-    # 1) Remove carriage returns
-    # 2) Convert all characters to lowercase
-    # 3) Remove comments (text starting with "#", include possible spaces before the hash sign)
-    # 4) Remove lines containing "/"
-    # 5) Remove leading tabs, spaces, etc.
-    # 6) Delete lines not matching domain names
-    < "${src}" tr -d '\r' | \
-    tr '[:upper:]' '[:lower:]' | \
-    sed 's/\s*#.*//g' | \
-    sed -r '/(\/).*$/d' | \
-    sed -r 's/^.*\s+//g' | \
-    sed -r '/([^\.]+\.)+[^\.]{2,}/!d' >  "${destination}"
-    chmod 644 "${destination}"
-    return 0
-  #fi
-
-  # Individual file parsing: Keep comments, while parsing domains from each line
-  # We keep comments to respect the list maintainer's licensing
-  read -r firstLine < "${src}"
-
-  # Determine how to parse individual source file formats
-  if [[ "${firstLine,,}" =~ (adblock|ublock|^!) ]]; then
-    # Compare $firstLine against lower case words found in Adblock lists
-    echo -e "  ${CROSS} Format: Adblock (list type not supported)"
-  elif grep -q "^address=/" "${src}" &> /dev/null; then
-    # Parse Dnsmasq format lists
-    echo -e "  ${CROSS} Format: Dnsmasq (list type not supported)"
-  elif grep -q -E "^https?://" "${src}" &> /dev/null; then
-    # Parse URL list if source file contains "http://" or "https://"
-    # Scanning for "^IPv4$" is too slow with large (1M) lists on low-end hardware
-    echo -ne "  ${INFO} Format: URL"
-
-    awk '
-      # Remove URL scheme, optional "username:password@", and ":?/;"
-      # The scheme must be matched carefully to avoid blocking the wrong URL
-      # in cases like:
-      #   http://www.evil.com?http://www.good.com
-      # See RFC 3986 section 3.1 for details.
-      /[:?\/;]/ { gsub(/(^[a-zA-Z][a-zA-Z0-9+.-]*:\/\/(.*:.*@)?|[:?\/;].*)/, "", $0) }
-      # Skip lines which are only IPv4 addresses
-      /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/ { next }
-      # Print if nonempty
-      length { print }
-    ' "${src}" 2> /dev/null > "${destination}"
-    chmod 644 "${destination}"
-
-    echo -e "${OVER}  ${TICK} Format: URL"
-  else
-    # Default: Keep hosts/domains file in same format as it was downloaded
-    output=$( { mv "${src}" "${destination}"; } 2>&1 )
-    chmod 644 "${destination}"
-
-    if [[ ! -e "${destination}" ]]; then
-      echo -e "\\n  ${CROSS} Unable to move tmp file to ${piholeDir}
-    ${output}"
-      gravity_Cleanup "error"
-    fi
-  fi
+  # Remove comments and print only the domain name
+  # Most of the lists downloaded are already in hosts file format but the spacing/formatting is not contiguous
+  # This helps with that and makes it easier to read
+  # It also helps with debugging so each stage of the script can be researched more in depth
+  # 1) Remove carriage returns
+  # 2) Convert all characters to lowercase
+  # 3) Remove comments (text starting with "#", include possible spaces before the hash sign)
+  # 4) Remove lines containing "/"
+  # 5) Remove leading tabs, spaces, etc.
+  # 6) Delete lines not matching domain names
+  < "${src}" tr -d '\r' | \
+  tr '[:upper:]' '[:lower:]' | \
+  sed 's/\s*#.*//g' | \
+  sed -r '/(\/).*$/d' | \
+  sed -r 's/^.*\s+//g' | \
+  sed -r '/([^\.]+\.)+[^\.]{2,}/!d' >  "${destination}"
+  chmod 644 "${destination}"
 }
 
 # Report number of entries in a table

test/_centos_9.Dockerfile

@@ -0,0 +1,18 @@
+FROM quay.io/centos/centos:stream9
+RUN yum install -y --allowerasing curl git initscripts
+
+ENV GITDIR /etc/.pihole
+ENV SCRIPTDIR /opt/pihole
+
+RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole
+ADD . $GITDIR
+RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/
+ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR
+
+RUN true && \
+    chmod +x $SCRIPTDIR/*
+
+ENV SKIP_INSTALL true
+ENV OS_CHECK_DOMAIN_NAME dev-supportedos.pi-hole.net
+
+#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \

test/test_any_automated_install.py

@@ -619,10 +619,15 @@ def test_FTL_detect_aarch64_no_errors(host):
     """
     # mock uname to return aarch64 platform
     mock_command("uname", {"-m": ("aarch64", "0")}, host)
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
     # mock ldd to respond with aarch64 shared library
-    mock_command("ldd", {"/bin/sh": ("/lib/ld-linux-aarch64.so.1", "0")}, host)
+    mock_command(
+        "ldd",
+        {
+            "/bin/sh": ("/lib/ld-linux-aarch64.so.1", "0"),
+            "/usr/bin/sh": ("/lib/ld-linux-aarch64.so.1", "0"),
+        },
+        host,
+    )
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh
@@ -647,10 +652,15 @@ def test_FTL_detect_armv4t_no_errors(host):
     """
     # mock uname to return armv4t platform
     mock_command("uname", {"-m": ("armv4t", "0")}, host)
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
     # mock ldd to respond with armv4t shared library
-    mock_command("ldd", {"/bin/sh": ("/lib/ld-linux.so.3", "0")}, host)
+    mock_command(
+        "ldd",
+        {
+            "/bin/sh": ("/lib/ld-linux.so.3", "0"),
+            "/usr/bin/sh": ("/lib/ld-linux.so.3", "0"),
+        },
+        host,
+    )
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh
@@ -675,10 +685,15 @@ def test_FTL_detect_armv5te_no_errors(host):
     """
     # mock uname to return armv5te platform
     mock_command("uname", {"-m": ("armv5te", "0")}, host)
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
     # mock ldd to respond with ld-linux shared library
-    mock_command("ldd", {"/bin/sh": ("/lib/ld-linux.so.3", "0")}, host)
+    mock_command(
+        "ldd",
+        {
+            "/bin/sh": ("/lib/ld-linux.so.3", "0"),
+            "/usr/bin/sh": ("/lib/ld-linux.so.3", "0"),
+        },
+        host,
+    )
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh
@@ -704,9 +719,14 @@ def test_FTL_detect_armv6l_no_errors(host):
     # mock uname to return armv6l platform
     mock_command("uname", {"-m": ("armv6l", "0")}, host)
     # mock ldd to respond with ld-linux-armhf shared library
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
-    mock_command("ldd", {"/bin/sh": ("/lib/ld-linux-armhf.so.3", "0")}, host)
+    mock_command(
+        "ldd",
+        {
+            "/bin/sh": ("/lib/ld-linux-armhf.so.3", "0"),
+            "/usr/bin/sh": ("/lib/ld-linux-armhf.so.3", "0"),
+        },
+        host,
+    )
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh
@@ -734,9 +754,14 @@ def test_FTL_detect_armv7l_no_errors(host):
     # mock uname to return armv7l platform
     mock_command("uname", {"-m": ("armv7l", "0")}, host)
     # mock ldd to respond with ld-linux-armhf shared library
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
-    mock_command("ldd", {"/bin/sh": ("/lib/ld-linux-armhf.so.3", "0")}, host)
+    mock_command(
+        "ldd",
+        {
+            "/bin/sh": ("/lib/ld-linux-armhf.so.3", "0"),
+            "/usr/bin/sh": ("/lib/ld-linux-armhf.so.3", "0"),
+        },
+        host,
+    )
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh
@@ -763,10 +788,15 @@ def test_FTL_detect_armv8a_no_errors(host):
     """
     # mock uname to return armv8a platform
     mock_command("uname", {"-m": ("armv8a", "0")}, host)
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
     # mock ldd to respond with ld-linux-armhf shared library
-    mock_command("ldd", {"/bin/sh": ("/lib/ld-linux-armhf.so.3", "0")}, host)
+    mock_command(
+        "ldd",
+        {
+            "/bin/sh": ("/lib/ld-linux-armhf.so.3", "0"),
+            "/usr/bin/sh": ("/lib/ld-linux-armhf.so.3", "0"),
+        },
+        host,
+    )
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh
@@ -789,8 +819,6 @@ def test_FTL_detect_x86_64_no_errors(host):
     """
     confirms only x86_64 package is downloaded for FTL engine
     """
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh
@@ -813,8 +841,6 @@ def test_FTL_detect_unknown_no_errors(host):
     """confirms only generic package is downloaded for FTL engine"""
     # mock uname to return generic platform
     mock_command("uname", {"-m": ("mips", "0")}, host)
-    # mock `which sh` to return `/bin/sh`
-    mock_command("which", {"sh": ("/bin/sh", "0")}, host)
     detectPlatform = host.run(
         """
     source /opt/pihole/basic-install.sh

test/tox.centos_9.ini

@@ -0,0 +1,8 @@
+[tox]
+envlist = py3
+
+[testenv]
+allowlist_externals = docker
+deps = -rrequirements.txt
+commands = docker build -f _centos_9.Dockerfile -t pytest_pihole:test_container ../
+           pytest {posargs:-vv -n auto} ./test_any_automated_install.py ./test_any_utils.py ./test_centos_fedora_common_support.py ./test_centos_common_support.py