Chapter 10: Troubleshooting

← Back to Index | Previous: Network and firewall | Next: Appendix →

10.1 Installer logs

Collect timestamps and the failing step name before contacting support. Prefer installer --status / --troubleshoot over ad-hoc host commands (Chapter 5 §5.6).

Installer self-update: Permission denied on /opt/deslicer/bin

Symptoms: replacing /opt/deslicer/bin/deslicer-dai-install.sh with published installer then mv: … Permission denied, and the run continues on an old version (for example 1.4.8).

Cause: The on-disk installer is under a root-owned directory. Older DAI installers treated a failed mv as success and re-executed the stale script.

Fix: Re-download to /tmp and run from there; then refresh the /opt copy with sudo install. See Chapter 5 §5.6. Published installer 1.4.11+ fails closed on replace failure and continues from the verified downloaded copy for that run.

10.2 Ansible temporary-file permission failures

Symptoms: Playbook fails while escalating to the service account with a message about temporary files and chmod: invalid operator.

Cause: The host is missing setfacl (package acl).

Expected path: Current DAI/DAP installers install acl during host preflight. Re-download the published installer and re-run.

If preflight cannot install the package (air-gapped / locked repos), install it once with sudo, then re-run:

# Ubuntu / Debian
sudo apt-get update && sudo apt-get install -y acl

# RHEL / Rocky / Alma / Amazon Linux 2023
sudo dnf install -y acl

command -v setfacl

10.3 docker.service fails to start on RHEL

Symptoms: DAI (or DAP) install fails during Docker host prep with Job for docker.service failed / dockerd is not responding, typically on RHEL 10 / Rocky / Alma after Docker CE packages are installed.

Cause: Docker’s networking stack needs kernel modules (br_netfilter, overlay, xt_addrtype, iptable_nat) and iptables-nft. Fresh RHEL / AWS images sometimes lack a loadable xt_addrtype (often from kernel-modules-extra on RHCK, or kernel-uek-modules-extra on UEK), or the host still runs an older kernel after a newer kernel RPM was installed. Oracle Linux with UEK may already have addrtype in-core — installing RHCK kernel-modules-extra is unnecessary and can fail on dnf exclude filters.

Typical journal line (kernels may already match):

iptables ... -m addrtype --dst-type LOCAL -j DOCKER: Warning: Extension addrtype revision 0 not supported, missing kernel module?

Expected path: DAI / DAP ansible bundle container_engine installs iptables-nft + Docker CE (major-pinned $releasever), asserts docker.service, runs depmod, and requires xt_addrtype capability (already loaded, built-in, or modprobe). Kernel *-modules-extra packages are soft-installed/reinstalled only when capability is missing (RHCK vs UEK selected from uname -r); dnf exclude filtering is non-fatal. Then start docker. If docker.service fails to start, the role reloads the networking modules and retries once; if it still fails and the running kernel differs from the newest installed kernel, it fails closed with a REBOOT REQUIRED banner — reboot and re-run the same enroll/install command. Docker repair is re-running the installer; Control's enroll OS tabs only carry the age CLI install and do not paste Docker commands. See also Chapter 3 Docker on RHEL.

Fix when journal mentions addrtype / xt_addrtype (even if uname -r matches the newest kernel):

uname -r
find /lib/modules/"$(uname -r)" -name 'xt_addrtype.ko*'
# RHCK (typical RHEL/Rocky/Alma):
sudo dnf reinstall -y "kernel-modules-extra-$(uname -r)" || sudo dnf reinstall -y kernel-modules-extra
# UEK (*uek* in uname -r / Oracle):
# sudo dnf reinstall -y "kernel-uek-modules-extra-$(uname -r)" || sudo dnf reinstall -y kernel-uek-modules-extra
sudo depmod -a
sudo modprobe xt_addrtype
sudo systemctl reset-failed docker
sudo systemctl enable --now docker
sudo docker info

After the installer stops for reboot (newest installed kernel ≠ uname -r):

sudo reboot
# Then re-run the same enroll/install command you used before the reboot.

Ubuntu / Debian / Amazon Linux 2023: this failure mode is uncommon; keep using the existing distro Docker paths (no reboot for modules is usually required).

10.4 Image pull / registry auth

Symptoms: unauthorized, denied, or pull timeouts during install.

Checks:

1. Confirm outbound HTTPS to container-registry.deslicer.io (Chapter 9)
2. Confirm container registry username/password in the provision package match what Deslicer issued (contact Deslicer to re-export the package if wrong)
3. Re-run install only after fixing credentials—do not paste passwords into chat

10.5 Age decrypt failures

Symptoms: DAI or DAP installer cannot decrypt provision.enc.yml, or DAI cannot read install-state.

Checks:

1. Age identity matches the public key used to encrypt the file
2. For DAP: you are decrypting the Control-issued bundle for this enroll, not an old handoff file
3. For DAI updates: /opt/deslicer/ai/install-state.enc.yml exists and the same identity is used
4. Identity file permissions allow the install user to read it

10.6 Control or app not reachable on HTTPS

Symptoms: Browser errors on https://<dai-host>/ or /control, while the host-local health check works.

Checks:

1. On the AI host: bash /opt/deslicer/bin/deslicer-dai-install.sh --status
2. Confirm DNS points at this host
3. Confirm certificate mode in Control (ACME needs port 80; PEM needs valid files under /etc/caddy/certs/)
4. Re-apply Web certificates using the Control-generated host command (Chapter 6)

10.7 ACME failures

Symptoms: Automatic HTTPS never becomes ready; Control or the installer report certificate challenge failures.

Checks:

1. Port 80 reachable from the public Internet to this host
2. No other process bound to 80/443
3. Hostname in install answers matches the DNS name you challenge

10.8 DAP enroll probe failures

Symptoms: deslicer-dap-install.sh cannot reach --dai-url.

Checks:

1. DAI Control is up on HTTPS
2. DAP host can resolve and reach the AI hostname (firewall between hosts)
3. Enroll token has not expired—mint a new token in Control

10.9 Fresh token vs existing volumes

Symptoms: Fresh install refuses because DAP volumes already exist, or Postgres auth fails after a rotated Fresh token.

Checks:

1. Prefer an Update token from Control when /opt/deslicer/dap already has data volumes
2. If a previous Fresh run left volumes + /opt/deslicer/dap/.env and the same Fresh token password still matches that .env, re-run the installer—it auto-resumes (preserves .env, finishes bootstrap; brings the stack up if observer-api is down)
3. Fresh that rotates secrets against a live volume whose .env password differs will refuse—wipe volumes for a clean Fresh, or use Update/Repair
4. Do not re-use a Fresh token that rotated the DB password against a live stack that still has the previous password

Encrypted secret verify failures: bootstrap task Verify DAP encrypted portal and registry rows decrypt must report a clear message (missing portal row / keyring mismatch). A Traceback unpacking true from set_config was a tooling bug—upgrade the DAP ansible tip / installer.

10.10 Update token cannot read compose .env

Symptoms: --update fails with missing or unreadable /opt/deslicer/dap/.env (often Permission denied for a non-root user).

Checks:

1. Run the Control-printed install command as the sudo-capable admin
2. Confirm .env exists under /opt/deslicer/dap with mode 0600 and is readable by the install path Control documents
3. Re-download the installer from Control if the host copy is older than the pin Control shows (deslicer-dap-install.sh --version)

10.11 DNS / dns_error vs Healthy

Symptoms: Control shows dns_error, failed, or Inactive while Retest shows Healthy.

Checks:

1. Public Observer hostname resolves from the AI host to the DAP host IP
2. On split installs, Deslicer AI probes the public Observer HTTPS URL Control shows—not a host-local name
3. After DNS/TLS is fixed, use Retest connection or Retry probe so Control sets install status to active and clears the stale error
4. If ACME backed off while DNS was wrong, restart Caddy on the DAP host after DNS is correct

10.12 Deslicer AI reports DAP down

Symptoms: Control or dashboard shows Observer unreachable.

Checks:

1. Open the Observer health URL Control shows for the backend
2. Confirm the backend URL in Control matches a URL the AI host can reach (split: public edge HTTPS on 443)
3. Confirm certificate trust on the AI host for that Observer URL
4. On each host, confirm the Compose project is running (docker compose under /opt/deslicer/ai and /opt/deslicer/dap)
5. Run Retry probe / Retest after fixing network or TLS so install status can become Active

10.13 Login returns 403 {"error":"Forbidden"}

Symptoms: Browser sign-in (email/password local login on https://<dai-host>/ or /control) fails immediately with:

{"error":"Forbidden"}

HTTP status 403. Wrong password returns 401 (Invalid credentials) instead—this section is only for 403 Forbidden.

Cause: Local login accepts the browser origin only when it exactly matches the public app URL configured for the Deslicer AI stack (no trailing slash). Typical mismatch after a hostname rename, DNS fix, or certificate-only apply that updated Caddy but not the application configuration.

Checks:

1. Note the URL in the address bar (scheme + host only), for example https://dai.example.com
2. On the AI host, compare the Caddy site name and the app URL:

# Public edge hostname (what Caddy serves)
sudo awk '/^[A-Za-z0-9][A-Za-z0-9._-]*[[:space:]]*\{/{ gsub(/\{/,"",$1); print $1; exit }' /etc/caddy/Caddyfile

# App-allowed origin (must match the browser URL)
sudo grep '^NEXT_PUBLIC_APP_URL=' /opt/deslicer/ai/.env

3. Confirm provision.yml / provision.enc.yml has the same public hostname in both: - deployment.urls.daiPublic (preferred; drives compile / Caddy) - app.domainName (keep in sync for handoffs)

4. Optional origin probe (expect 401 Invalid credentials when Origin matches; 403 when it does not):

curl -skS -X POST "https://<dai-host>/api/auth/local-db/login" \
  -H 'content-type: application/json' \
  -H 'origin: https://<dai-host>' \
  -d '{"email":"probe@example.com","password":"wrong"}'

Fix:

1. Correct daiPublic / domainName in the provision file (exact DNS name, including spelling). Contact Deslicer to re-export the package if you cannot edit the handoff safely.
2. Re-run a full DAI update so application configuration and containers pick up the new URL—omit --proxy-only:

bash /opt/deslicer/bin/deslicer-dai-install.sh \
  --provision /path/to/provision.yml \
  --age-identity ~/.config/age/keys.txt \
  --update

Confirm the installer log ends with a full install completion message, not only an edge-proxy apply.

3. Verify NEXT_PUBLIC_APP_URL matches the browser URL, then sign in again (hard-refresh if the tab still posts from an old origin).

Related failures (not this 403):

10.14 Control DAP page returns 500 (install-package cache)

Symptoms: https://<dai-host>/control/platform-integrations/dap or Updates → DAP fails with HTTP 500. Public application health may still be healthy.

Cause: The Control install-package cache directory is not writable by the Control service.

Checks / fix:

bash /opt/deslicer/bin/deslicer-dai-install.sh --status
bash /opt/deslicer/bin/deslicer-dai-install.sh --troubleshoot
bash /opt/deslicer/bin/deslicer-dai-install.sh --troubleshoot --fix-dap-bundle-cache

Then reload the Control DAP / Updates page.

10.15 DAP install package Apply / serve failures

Symptoms: Control shows Update available but Apply fails; DAP install fails on missing or mismatched checksum; enroll downloads return 503.

Checks:

1. Apply update requires outbound HTTPS from the AI host to registry.deslicer.io and artifact-registry.deslicer.io (Chapter 9). Air-gapped sites use Import package under Control → Updates → DAP instead.
2. If Apply says the package needs a newer Deslicer AI (minDaiVersion), update the DAI image first—playbook Apply cannot skip that gate.
3. Applying a release that is not the registry latest requires explicit confirmation in Control.
4. Messages about a missing package checksum, or HTTP 503 indicating no package is staged, mean the Control cache has no verified package. Fix with bash /opt/deslicer/bin/deslicer-dai-install.sh --repair, or Import package, then re-issue the enroll token.
5. Bundle download 503 with an integrity/cache message: the on-disk cache no longer matches the stored pin. Prefer Roll back, then Apply update again.
6. After Apply or repair seed, confirm Control → Updates → DAP shows a staged checksum before re-running DAP enroll / --update.
7. If Control itself 500s before you can open Updates, fix cache permissions first (§10.13).

Playbook-only hotfixes do not require a new Deslicer AI image. Schema or enroll API changes still do.

10.16 No match for argument: python3.13 on RHEL 10 (AWS)

Symptoms: Host prep fails with:

No match for argument: python3.13
No match for argument: python3.13-pip
Error: Unable to find a match: python3.13 python3.13-pip

Often paired with Unable to read consumer identity on AWS RHEL images (RHUI still serves BaseOS/AppStream).

Cause: RHEL 10 BaseOS/AppStream ships Python 3.12. python3.13 is an EPEL package and needs CodeReady Builder (CRB) enabled. A bare sudo dnf install -y python3.13 … without CRB+EPEL always fails. Age from Filippo can succeed while Python install fails.

Fix: do not install Python by hand. The DAP/DAI installer preflight (DAP 1.9.28+ / DAI 1.4.19+) enables CRB+EPEL and installs python3.13, python3.13-pip, curl, and acl itself — re-download the current install script and re-run the same enroll/install command. If preflight still fails, capture the full installer output and the result of:

dnf repolist --enabled
cat /etc/os-release

and see Chapter 3 / §10.17. Control's enroll OS tabs only carry the age CLI install; they are not a Python repair path.

10.17 Getting help

Contact support@deslicer.com with:

• Engagement / hostname
• Installer versions (deslicer-dai-install.sh / deslicer-dap-install.sh --version) and output of --status / --troubleshoot when relevant
• Applied DAP install-package version from Control (Updates), if relevant
• Redacted log excerpts (no passwords, tokens, or private keys)
• For login 403: browser URL and the configured public app URL (no secrets)

← Back to Index | Previous: Network and firewall | Next: Appendix →