Skip to main content

Runbook: webku Incident Triage

ServiceWebku (multi-tenant hotel website CMS)
Owner Team slack handle@bnl-dev-bali
Team's Slack Channel#bnl-teams-b
Alert Channel#bnl-critical-alerts
Alert sourceincident.io via the producer relay ([webku][env] <Exception> in <file>:<line>)

[!warning] Alerting is on a branch, not on master Ticket BNL-2367, branch feat/BNL-2367-setup-incident-io-alert-and-alert-doc (commits a481a4b9, 79e0b0f3). master is at e311c60f (v1.46.4) and has no alerting — production reports to Sentry only and pages nobody.

Everything in [[#1. Triage]] and [[#4. True Incident]] works today against Sentry and the logs. Only the "a page arrived" entry point is still ahead of you.

When BNL-2367 merges and INCIDENT_ALERT_ENABLED=true is set, delete this callout and update the verified-against line below.

Verified against branch 79e0b0f3 (master e311c60f, v1.46.4) on 2026-09-25.

Table of Contents​

  • [[#Important Links]]
  • [[#1. Triage]]
  • [[#2. Decision Point]]
  • [[#3. False Alarm]]
  • [[#4. True Incident]]
    • [[#4.1. Recover the System]]
    • [[#4.2. Clean up]]
ResourceWhere
Alert[incident.io — webku service view]
ArgoCDAuthentik dashboard → BNL ArgoCd — deploy state, sync history, pod health, live manifests
Grafana[Grafana] → dashboard UID a87fb0d919ec0ea5f6543124e16c42a5 (panel 1) — the CPU/memory panels the webku alerts fire from
OpenSearch[OpenSearch] — application logs, shipped by the vector sidecar
Sentry — unresolved webku issues[Sentry issue list, project=webku is:unresolved]
Sentry — releases[Sentry releases view] — for correlating an error with a deploy
Cloudflare Pages[Pages project list] — serves published tenant sites
Deploy pipeline[Bitbucket pipelines — bookandlink/webku]
Code repository[bitbucket.org/bookandlink/webku]

Kubernetes namespaces (from .kubernetes/env/*/grafana-alert.yaml):

EnvironmentNamespace
Productionprod-website
QA / Stagingstaging-webku

Workloads: webku (web), webku-static-generator (queue worker), webku-whitelabel-proxy, webku-migration (Job, ArgoCD sync hook).

[!note] The only Grafana alerts that exist for webku Four rules, defined in .kubernetes/env/*/grafana-alert.yaml and provisioned into Grafana:

RuleFires when
Prod Website BNL - CPU Usagenamespace CPU > 80% of quota, sustained 5m
Prod Website BNL - Memory Usagenamespace memory > 80% of quota, sustained 5m
Staging Website BNL - CPU Usagesame, staging-webku
Staging Website BNL - Memory Usagesame, staging-webku

They are resource-quota alerts, not error alerts — labelled app_name: webku, endpoint: slack. Nothing else in Grafana watches webku. If you are waiting for Grafana to tell you the app is throwing errors, it will not.

Engineering reference (repo sktbnl/AI-Doc-all-application, path team-b/docs/webku/):

DocumentUse it for
debugging-guide.mdSymptom-first triage, 37 numbered checks
known-issues.mdIs this already known? Check before debugging
publishing-pipeline.mdPublish internals, locks, timeout ordering
observability-and-jobs.mdSentry, Pulse, queue topology, log grep anchors
release-recovery.mdRestore vs rollback vs discard

Sibling runbooks in this vault:

  • [[corrupted-translation-json]] — the htmlspecialchars(): … array given fatal, and the repair command
  • [[cannot-read-properties-of-undefined]] — dashboard content-editor TypeError
  • [[google-analytics-permission-issue]] — analytics widget failures

1. Triage​

Goal: in under five minutes, establish blast radius and whether a release caused it. Do not read the stack trace yet — it tells you the fix, not the urgency.

Which tool answers which question:

QuestionTool
Is the app throwing errors, and how many tenants?Sentry
Did a deploy just happen, and did it sync cleanly?ArgoCD (Authentik → BNL ArgoCd)
Are pods healthy / restarting / OOM-killed?ArgoCD app view, or Grafana for the resource trend
What did the application actually log?OpenSearch
Is the published site up?curl against the tenant domain

webku has four surfaces that fail differently. Identify yours from the alert's file path or the Sentry transaction name:

SurfaceEntry point in the traceBlast radius
Public rendererPageController, themes.* viewsEvery visitor to every site — highest urgency
Publish pipelineSitePublishing, Publishable, deploy:cloudflareOne site cannot ship changes
DashboardDashboard\*ControllerAdmins blocked, visitors fine
Public forms APIAPI\StaticSiteEnquiries silently lost

1.1. Is a real tenant site up?​

curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://<tenant-domain>/en
  • Expected: 200 in under 2s.
  • Anything else — or a timeout — means visitors are affected. That outranks every other check.

1.2. How many tenants are hit?​

Open the Sentry issue → Tags → url / server_name.

  • One site → data-specific (a misconfigured or corrupted site). Lower urgency, see [[#Potential Cause 5 — One site only, corrupted per-locale JSON]].
  • Many sites → code or infrastructure. Treat as a real incident.

1.3. Did it start with a deploy?​

Compare the Sentry issue's first seen timestamp against the last sync in ArgoCD.

  1. Open Authentik dashboard → BNL ArgoCd and select the webku application for the affected environment.
  2. Read Sync Status and Last Sync on the app tile:
    • Synced / Healthy with a Last Sync well before the error started ⇒ not a deploy.
    • Last Sync within ~30 minutes of first-seen ⇒ assume the deploy caused it. This is the highest-yield question in the whole runbook.
    • OutOfSync, Degraded, or Progressing for more than a few minutes ⇒ the deploy itself is the incident — go to [[#Potential Cause 1 — A bad release]].
  3. Check the webku-migration Job. It runs php artisan migrate --force as an ArgoCD Sync hook (argocd.argoproj.io/hook: Sync, backoffLimit: 1), so it executes before the new pods are healthy. A failed migration fails the sync and can leave the old and new schema expectations mixed. Its status is visible in the ArgoCD resource tree.
  4. The image tag ArgoCD is deploying is in .kubernetes/env/<env>/deployment.yaml (webku:vX.Y.Z).

1.4. Is it still happening?​

In Sentry, set the issue graph to last 1 hour.

  • Flat / stopped → likely transient. Candidate for [[#3. False Alarm]].
  • Ongoing or climbing → [[#4. True Incident]].

1.5. Is the database healthy?​

Check this even if the error looks unrelated. In webku, MySQL backs application data and the cache, and sessions, and the job queue (CACHE_STORE, SESSION_DRIVER, QUEUE_CONNECTION are all database).

Primary — ArgoCD (Authentik → BNL ArgoCd): open the webku application and read the resource tree.

  • Expected: every workload Healthy; webku and webku-static-generator pods Running.
  • Repeated pod restarts, CrashLoopBackOff, or OOMKilled show here first.

Grafana — check whether either resource alert is firing for this environment:

  • Prod Website BNL - CPU Usage / Prod Website BNL - Memory Usage (prod, namespace prod-website)
  • Staging Website BNL - … (QA/staging, namespace staging-webku)
  • Both fire at >80% of the namespace quota sustained for 5 minutes. A firing memory alert alongside publish failures usually means the static-generator pod is being OOM-killed mid-build.

If you have direct cluster access, the same checks:

kubectl get pods -n prod-website -l app=webku
kubectl get pods -n prod-website -l app=webku-static-generator
  • Expected: all Running, RESTARTS not climbing.

Then check the Aurora dashboard for connection count and CPU.

[!danger] Absence of alerts is not evidence of health Alert delivery is queued on high-priority — the same MySQL-backed queue. If MySQL is down you may receive one alert, or none. There is an inline fallback for exactly this case, but do not trust silence during a suspected DB incident.

1.6. Is the alerting path itself working?​

Only meaningful once BNL-2367 is merged. From a pod terminal:

php artisan incident:test --dry-run

This prints every gate between you and incident.io as a table — INCIDENT_ALERT_ENABLED, hook URL, token presence, severity permission, delivery mode, and the count of pending high-priority jobs. It sends nothing.

  • Expected: every row reads ok.
  • Any row reading BLOCKS — … is your answer.
  • pending high-priority jobs > 0 means alerts are queued behind an idle worker.

2. Decision Point​

  • IF the tenant site returns 200 (1.1), the error has stopped (1.4), pods are Running (1.5), and it affects a single site with a known cause...

    • ➡️ Go to: [[#3. False Alarm]]

  • IF the tenant site is down or slow (1.1), OR the error is ongoing (1.4), OR it correlates with a release (1.3), OR it affects multiple tenants (1.2), OR pods/DB are unhealthy (1.5)...

    • ➡️ Go to: [[#4. True Incident]]

  • IF no alert arrived but you suspect an incident, OR incident:test showed a BLOCKS row (1.6)...

    • ➡️ Go to: [[#Potential Cause 6 — Alerts are not arriving]]

[!note] When in doubt, choose True Incident A false alarm costs you fifteen minutes. A missed incident costs a customer their website.

3. False Alarm​

A resolved alert is not a closed alert. A monitor that stays triggered will not re-fire if things get worse, so it must be explicitly handled.

3.1. Decide what to do with the monitor​

SituationAction
Error stopped, cause understood and harmlessResolve the Sentry issue so a recurrence re-opens it as new
Error stopped, cause unknownLeave it unresolved and watch for 30 minutes
Alert is flapping (fires/recovers repeatedly)Mute in incident.io for 2 hours, no longer
Error is real but expected and already handled by the appAdd it to the exclusion list — see [[#3.3. Suppressing a genuinely noisy alert]]

[!warning] Never silence without recording why An exclusion with no rationale becomes a permanent blind spot nobody remembers creating.

3.2. Post in Slack​

Copy into #bnl-teams-b:

webku alert — false alarm. [alert title] fired at [time]. Triage: tenant site returned 200, error count flat for [N] minutes, pods healthy. Cause: [one line]. Action taken: [resolved in Sentry / muted 2h]. No customer impact. Follow-up ticket: [link or "none needed"].

3.3. Suppressing a genuinely noisy alert​

In escalating order — prefer the narrowest:

  1. Fix the underlying error. Always right when available.
  2. Add the exception class to the exclusion list — IncidentReporter::IGNORED in app/Services/Incident/IncidentReporter.php. Currently ValidationException, AuthenticationException, AuthorizationException, TokenMismatchException, ModelNotFoundException. A code change, reviewed like any other.
  3. Lengthen the dedupe window — INCIDENT_DEDUPE_TTL (seconds, default 900). Config only, no deploy.
  4. Switch alerting off for the environment — INCIDENT_ALERT_ENABLED=false. Blunt and immediate. ⚠ Disables all paging for that environment; Sentry keeps recording, so you lose paging, not visibility.

3.4. Raise the follow-up ticket​

Title: RCA: webku noisy alert — [alert title] — [date]

Include: the Sentry link, first-seen and last-seen timestamps, tenant count from 1.2, which suppression step you took, and when it must be reviewed. Tag @bnl-dev-bali. Paste the ticket link into the Slack thread from 3.2.

4. True Incident​

Priority order: restore service → assess blast radius → communicate → clean up.

Announce first, in #bnl-critical-alerts:

🔴 Investigating webku incident. [alert title]. Blast radius: [N tenants / all visitors / publishing only]. Started [time], [correlates with release vX.Y.Z / no recent release]. I'm on it — updates in thread.

4.1. Recover the System​

Causes are ordered most-likely first. Stop at the first one that matches.

Potential Cause 1 — A bad release​

By far the most common, and the fastest to resolve.

Diagnostic Steps:

  1. Sentry issue → first seen. Compare with Last Sync in ArgoCD (Authentik → BNL ArgoCd).
  2. In ArgoCD, confirm which image tag is actually live — open the webku Deployment in the resource tree and read the container image, or: 
    kubectl get deployment webku -n prod-website -o jsonpath='{.spec.template.spec.containers[0].image}'
    • Expected: matches the version you believe is live.
  3. In Sentry, check whether the issue's Release tag is a single recent version. One release only ⇒ regression.
  4. Check the webku-migration Job in the ArgoCD resource tree. It is a Sync hook running php artisan migrate --force with backoffLimit: 1. If it failed, the schema may be partially migrated against either the old or the new code — that changes the remediation, because a plain rollback will not undo a migration.

Remediation Plan:

  1. Roll back rather than debug forward.
    • In ArgoCD: app → History and Rollback → select the previous successful revision → Rollback.
    • ⚠ Rollback re-runs the Sync hook, so webku-migration executes again. If the bad release included a schema migration, do not roll back until you have confirmed the migration is reversible — raise it with @bnl-dev-bali first.
    • If ArgoCD auto-sync is enabled, disable it before rolling back or it will immediately re-sync forward again.
  2. Announce:

    Incident: webku. Rolling back to [previous tag] — errors began within [N] minutes of [bad tag].

  3. Only investigate the root cause once traffic is healthy.

Verification:

  1. ArgoCD shows Synced / Healthy on the previous revision.
  2. Sentry error count for the issue drops to zero within 5 minutes of the rollback completing.
  3. curl from 1.1 returns 200.
  4. Confirm no publishing attempts were left mid-flight — go to [[#Potential Cause 3 — Publishing failed or is stuck]].

Potential Cause 2 — Database or core infrastructure unreachable​

Severity: Immediate. A MySQL problem is never only a database problem:

Also stops workingBecause
Login and every sessionSessions are DB rows
Host → site resolution, all cached lookupsCache is a DB table
Publish, restore, rollback, discardQueue is a DB table
Alert delivery itselfAlerts queue on high-priority

Diagnostic Steps:

  1. Confirm at the infrastructure layer, not from the app:
    • ArgoCD (Authentik → BNL ArgoCd): are the webku pods Healthy, or restarting?
    • OpenSearch: filter to the webku application logs and search SQLSTATE OR PDOException OR "Connection refused" over the incident window.
    • With cluster access: kubectl logs deploy/webku -n prod-website --tail=50 | grep -iE "SQLSTATE|PDOException|Connection refused"
  2. Open the Aurora dashboard in Grafana: connection count, CPU, failover events.
  3. In Sentry, PDOException maps to severity Immediate — its presence confirms the class of problem.

Remediation Plan:

  1. This is a platform escalation. Nothing in the webku repo fixes it.
  2. Page Platform in #bnl-critical-alerts with the Aurora dashboard link and the pod status output.
  3. While waiting, do not restart pods repeatedly — it adds connection churn.

Verification:

  1. Aurora reports healthy; pods stop restarting.
  2. Load the dashboard and log in — proves sessions and cache are working again.
  3. Then run [[#Potential Cause 3 — Publishing failed or is stuck]] — in-flight jobs died mid-run and will have left stuck rows.

Potential Cause 3 — Publishing failed or is stuck​

Severity: Business. A customer cannot get changes onto their live website.

[!note] Publishing is a live crawl The pipeline fetches the app's own rendered pages over HTTP at job-execution time. A publish failure is often caused by the public renderer being broken, not the pipeline. Re-check 1.1 first.

Diagnostic Steps:

  1. Get the site id from the alert or Sentry tags.
  2. Start at publishing_attempts, not the queue — status and message.error are the most specific signal available: 
    SELECT a.id, a.attempt, a.operation, a.status, a.started_at, a.finished_at,
           JSON_EXTRACT(a.message, '$.error') AS error
    FROM publishing_attempts a
    JOIN publishing_logs l ON l.id = a.publishing_log_id
    WHERE l.site_id = '<site_ulid>'
    ORDER BY a.id DESC LIMIT 5;
    • Healthy: latest row status = 'success' with finished_at set.
    • Stuck: status in pending|building|uploading|deploying with started_at more than 30 minutes ago.
  3. Then the step log — step-2 build, step-3 upload, step-4 deploy: 
    SELECT version, status, failed_at, message FROM publishing_logs
    WHERE site_id = '<site_ulid>' ORDER BY id DESC LIMIT 3;
  4. Then the worker. OpenSearch is the better tool here — the build may have happened on a pod that has since been replaced:
    • Filter to the webku application logs, search SitePublishing OR "deployment lock", scoped to the incident window.
    • With cluster access: kubectl logs deploy/webku-static-generator -n prod-website --tail=100 | grep -iE "SitePublishing|deployment lock"
  5. Check Grafana for a memory alert on this environment. A … - Memory Usage alert firing around the build time is the usual sign the static-generator pod was OOM-killed mid-crawl — which is exactly the case where SitePublishing::failed() never runs and the attempt is left stuck.

Remediation Plan:

If an attempt is stuck in a non-terminal status:

[!danger] There is no automatic recovery on master SitePublishing::failed() normally closes a crashed attempt, but it cannot run if the process died outright (OOM kill, container termination). ReleaseManager then refuses every subsequent publish for that site. A reaper command exists but is unmerged, in stash@{1}.

  1. Confirm no worker is actually running that job (check the logs above for an active build).
  2. Close the row by hand: 
    UPDATE publishing_attempts SET status = 'failed', finished_at = NOW(),
           message = JSON_OBJECT('error', 'Manually closed: worker died — INC-<id>')
    WHERE id = <attempt_id>;

    UPDATE publishing_logs SET status = 'failed', failed_at = NOW()
    WHERE id = <publishing_log_id> AND status NOT IN ('success','failed');
  3. Ask the customer to publish again, or publish on their behalf.

If the deploy reached Cloudflare but the site did not change: check the branch. deploy:cloudflare pins --branch to config('deploy.cloudflare.production_branch') (default master); any other branch lands as preview only — a clean success that never goes live.

Verification:

  1. The query in step 2 returns no non-terminal rows older than 30 minutes.
  2. A fresh publish reaches status = 'success'.
  3. The live site shows the change.

[!note] Timeout invariant — do not change one of these in isolation job timeout 1700s < deployment lock 1800s < DB_QUEUE_RETRY_AFTER 1900s. If the same publish appears to run twice, this ordering has been altered.

Potential Cause 4 — Public renderer failing for many tenants​

Severity: Immediate — every visitor to every site.

Diagnostic Steps:

  1. Test two different tenant domains (1.1). Both failing ⇒ systemic.
  2. Check whether the trace enters PageController or a themes.* view.
  3. Check the theme: a fault in a shared Blade component breaks every site using that theme, not all sites.
    • OpenSearch: search ViewException OR ErrorException over the incident window and group by site/URL.
    • With cluster access: kubectl logs deploy/webku -n prod-website --tail=100 | grep -iE "ViewException|ErrorException"

Remediation Plan:

  1. If it correlates with a release → [[#Potential Cause 1 — A bad release]], roll back.
  2. If it is a single theme, identify which sites use it before deciding urgency.
  3. Sites whose published Cloudflare artifact is current keep serving — the outage is on app-rendered traffic only. Say so in the incident channel; it materially changes reported impact.

Verification:

  1. Both tenant domains return 200.
  2. Sentry ViewException / ErrorException count returns to zero.

Potential Cause 5 — One site only, corrupted per-locale JSON​

Symptom: ViewException — htmlspecialchars(): Argument #1 ($string) must be of type string, array given. Fatal for that page on both the dashboard and the live app-served site.

Diagnostic Steps:

  1. Identify the site ULID and the failing view from the stack trace.
  2. Dry run first, from a pod terminal: 
    php artisan translation:fix --field=content --table=contents --site-id=<site_ulid> --dry-run

Remediation Plan: follow [[corrupted-translation-json]] — it carries the full procedure, the other affected tables, and the failure modes.

Two traps that catch people every time:

  • ⚠ 0 rows fixed but the page is still broken ⇒ the row's top-level keys are not locales, so the command does not recognise it. It needs manual repair — do not keep re-running.
  • ⚠ The repair does not publish itself. The command writes via DB::table(), bypassing the audit trail, so publishing returns 406 "nothing to publish". Make a trivial dashboard edit first, then publish.

⚠ Capture the row's value before fixing it. The write path that produces the nesting has never been identified, and the fix destroys the only evidence.

Verification: reload the page; then publish and confirm the live site.

Potential Cause 6 — Alerts are not arriving​

Diagnostic Steps:

  1. Run the gate table (1.6): php artisan incident:test --dry-run.

  2. Read the log — when alerting fails it logs and gives up by design, so it never breaks the request.

    OpenSearch, filtered to the webku application logs, search:

    "Incident hook rejected" OR "Incident hook unreachable" OR "IncidentReporter failed" OR "Incident alert dropped" OR "Could not queue incident"

    With cluster access:

    kubectl logs deploy/webku -n prod-website --tail=200 | grep -iE "Incident hook (rejected|unreachable)|IncidentReporter failed|Incident alert dropped|Could not queue incident"
Log lineMeaning
Incident hook rejected the alert + "body":"Cannot POST /incident-hooks"The producer host has no such route — not a webku fault
Incident hook rejected the alert + {"Severity":"Please fix this field value."}Severity casing. The producer requires Info/Daytime/Business/Immediate; lowercase is rejected. Fixed in 79e0b0f3 — seeing this means that commit is not deployed
Incident hook unreachableProducer down or network blocked
Incident alert dropped: info severity is not permittedINCIDENT_SEVERITY_OVERRIDE is set without INCIDENT_ALLOW_INFO — alerting is silently off
Could not queue incident alert, sending inlineQueue unavailable; the inline fallback ran
ReportIncident job failedDelivery exhausted its 3 attempts
  1. Check whether contact-form notifications are also missing. The same producer relay delivers both — if both are broken it is the producer, not webku's config.
  2. Verify the endpoint exists on that host. A healthy producer tells you nothing — /healthcheck returns 200 on hosts that have no hook route: 
    curl -s -o /dev/null -w "%{http_code}\n" https://producer.bookandlink.com/incident-hooks   # 405 = route exists (POST-only)
    curl -s -o /dev/null -w "%{http_code}\n" https://producer-rbmq.bnlstg.com/incident-hooks    # 404 = route absent

Remediation Plan:

  1. BLOCKS row in the gate table → fix that environment variable.
  2. Severity-casing rejection → deploy 79e0b0f3 or later.
  3. Producer 404 or unreachable → escalate to the producer service owner.
  4. ⚠ While the producer is down you are flying blind. Watch Sentry directly until it recovers, and say so in the incident channel.

Verification:

php artisan incident:test --severity=Info --title="Recovery check <date>"

Sends inline by default and deliberately — a queued alert is invisible until a worker runs, and "I triggered it and nothing arrived" is almost always an idle worker. Use --queue to exercise the real path.

Potential Cause 7 — Alert storm​

Treat as Immediate regardless of the alerts' own severity — a flooded pager means the next real incident is ignored.

Diagnostic Steps:

  1. Are the alerts different errors or the same one repeating?
    • Dedupe is per unique md5(class|file|line) with a 15-minute window, so a release breaking ten code paths legitimately produces ten alerts.
  2. Do the titles contain anything variable — an id, a timestamp, user input? Titles must be stable; IncidentReporter::titleFor() uses only class basename, file basename and line.
  3. Is an exception being thrown inside error handling? The static $reporting re-entrancy guard and isSelfOriginated() should prevent this.

Remediation Plan:

  1. Many different errors → a bad release. Roll back ([[#Potential Cause 1 — A bad release]]). Do not tune alerting.
  2. Unstable titles → a code bug in the reporter. Raise a ticket; do not patch live.
  3. Containment while you work: set INCIDENT_ALERT_ENABLED=false for the affected environment. Config change only — pods restart automatically, alerting stops within a minute or two, and Sentry keeps recording.
  4. Announce the suppression and set a reminder to re-enable.

Verification: alert rate returns to zero; Sentry still shows the underlying errors, so you retain visibility.

Potential Cause 8 — An error you do not recognise​

  1. Sentry first — full trace, request context, release, affected user and tenant counts.
  2. Which surface? Use the table in [[#1. Triage]].
  3. Application logs in OpenSearch — the grep anchors are listed in observability-and-jobs.md. Search the anchor, scoped to the incident window, rather than tailing a pod: the pod that logged it may already be gone.
  4. Already known? Check known-issues.md before debugging. Several recurring errors have documented root causes.
  5. Correlates with a release? Roll back rather than debug forward.
  6. Still stuck → escalate to @bnl-dev-bali with the Sentry link, the blast radius from 1.1–1.2, and what you have ruled out.

[!danger] Some failures never alert and never reach Sentry Exceptions swallowed by application code are invisible to both systems: SSOController::handleCallback's catch-all (every SSO failure becomes a login redirect), GoogleController's empty-200 catch (analytics errors look like "no data"), and GooglePlacesService's error returns. Silence on those paths is not evidence of health.

4.2. Clean up​

1. Assess the impact​

Failed publishes during the window:

SELECT l.site_id, l.version, l.status, l.failed_at
FROM publishing_logs l
WHERE l.failed_at BETWEEN '<start>' AND '<end>'
ORDER BY l.failed_at;

Attempts still stuck (must be zero before you close):

SELECT a.id, l.site_id, a.status, a.started_at
FROM publishing_attempts a
JOIN publishing_logs l ON l.id = a.publishing_log_id
WHERE a.status IN ('pending','building','uploading','deploying')
  AND a.started_at < NOW() - INTERVAL 30 MINUTE;

Queue backlog:

SELECT queue, COUNT(*) FROM jobs GROUP BY queue;
SELECT COUNT(*) FROM failed_jobs WHERE failed_at > '<start>';

Enquiries that were captured but not delivered. ⚠ Contact submissions are written before the notification is sent, so a producer outage leaves enquiries in the dashboard that nobody was emailed about:

SELECT site_id, COUNT(*) FROM contacts
WHERE updated_at BETWEEN '<start>' AND '<end>' GROUP BY site_id;

Record every count in the incident ticket.

2. Communicate​

Post in #bnl-critical-alerts, and cross-post to #bnl-teams-b:

✅ webku incident resolved. [alert title] Impact: [N] tenants, [start]–[end] ([duration]). [N] failed publishes, [N] enquiries delivered late. Cause: [one or two lines] Fix: [rollback to vX.Y.Z / config change / manual DB repair] Follow-up: [ticket link]

If customers were affected, hand the affected site list to CS as tenant domains, not site ULIDs.

3. Clean up state​

  1. Re-run failed publishes. Do not bulk-retry — publishing is a live crawl of the app's own pages, so a mass retry is a self-inflicted load spike. Ask affected customers to publish, or do it per site, spaced out.
  2. Clear stuck attempts using the UPDATE in Potential Cause 3. The second query in step 1 must then return zero rows.
  3. Retry failed jobs once the cause is resolved: 
    php artisan queue:failed
    php artisan queue:retry <uuid>          # prefer targeted retries over --all
  4. Re-enable anything you suppressed. If you set INCIDENT_ALERT_ENABLED=false or muted a monitor, turn it back on now and confirm with php artisan incident:test --dry-run.
  5. Verify a real tenant site one more time (1.1).
  6. Confirm ArgoCD is Synced / Healthy and, if you disabled auto-sync during a rollback, turn it back on.

4. Post-incident​

  • Raise the RCA ticket: RCA: webku [short description] - [date], tagged @bnl-dev-bali.
  • Attach the impact counts, the Sentry link, and the timeline.
  • If you hit a gap in this runbook — a command that did not work, a cause not listed — fix this file in the same ticket. That is the single highest-value follow-up action.

Appendix A — What this alerting cannot see​

Do not wait for a page that will never arrive:

Not coveredWhyWho covers it
Pod restarts, OOM kills, ingress 503Never reaches PHP — the app is not runningArgoCD (pod health) and Grafana (the CPU/memory quota alerts); Platform owns the fix
A published Cloudflare artifact serving a broken siteStatic hosting, outside this appCloudflare, Platform
Swallowed exceptionsCaught and discarded before any reporter sees themNothing — see Potential Cause 8
Slow-but-working requestsErrors only, no latency alertingPulse — ⚠ dashboard traffic only, not the public renderer. Grafana watches resource quota, not latency
Anything scheduled failing to run⚠ There is no scheduler on master. Nothing runs on a cadenceNothing

The 503 class has a known root cause in the deployment configuration (docs/issues/503-service-unavailable-root-cause.md in the webku repo). It is platform-owned; no application alerting will surface it.

Appendix B — Testing without paging anyone​

  • QA sends every alert at severity Info, which does not page — INCIDENT_SEVERITY_OVERRIDE=info plus INCIDENT_ALLOW_INFO=true. Config values are case-insensitive (canonicalSeverity() normalises them); the wire format is capitalised, because the producer rejects lowercase.
  • QA has its own routing target and its own credential — a second, independent reason testing cannot reach the live rotation. Confirm both before testing.
  • ⚠ Never copy the QA severity keys into production. Production sets neither. INCIDENT_SEVERITY_OVERRIDE without INCIDENT_ALLOW_INFO makes production stop alerting entirely, with no visible symptom except Incident alert dropped: info severity is not permitted in this environment. If anyone asks "why has webku not alerted lately?", grep for that line first.
php artisan incident:test --dry-run                        # gates + payload, sends nothing
php artisan incident:test --severity=Info                  # send now, inline (default)
php artisan incident:test --title="Daytime Test 4" --severity=Info
php artisan incident:test --queue                          # exercise the real queued path

The trigger route​

GET /test-sentry?token=<SENTRY_SECRET_KEY> (routes/web.php:26) throws a real exception on demand. Produces Daytime in production, Info in QA, titled [webku][<env>] Exception in web.php:32.

⚠ Two preconditions, and failing either is silent:

RequirementIf wrong
?token= must match SENTRY_SECRET_KEY exactly404 — identical to a route that does not exist
SENTRY_ENVIRONMENT must equal APP_ENV exactlyHTTP 200 with body Sentry is not active in this environment. Nothing thrown, nothing alerted. The more dangerous of the two, because 200 looks like success
  • Rate limited to 3/minute. The 4th returns 429 — an HttpException below 500, correctly excluded, raising no alert. Testing "repeats collapse into one alert" needs ~4 minutes, not 10 seconds.
  • The fingerprint never varies (class, file and line are constant), so repeat triggers give exactly one alert per cooldown window. You cannot raise two different test alerts from this route.
  • Both behaviours are pinned by tests/Feature/IncidentTriggerRouteTest.php.
  • ⚠ Never test in production. A test alert is indistinguishable from a real one to whoever is paged.

Appendix C — How an exception becomes a page​

Useful when the alert you received does not match what you expected.

exception thrown
  └─ bootstrap/app.php  →  IncidentReporter::report()
       1. $reporting static guard        — a failure inside alerting never alerts
       2. configured()                   — enabled + url + token, else stop
       3. shouldReport()                 — see exclusions below
       4. severityFor()                  — PDOException ⇒ Immediate
       │                                   trace hits SitePublishing / RestorePublishedVersion /
       │                                   RollbackChanges / DiscardSiteChanges ⇒ Business
       │                                   otherwise ⇒ Daytime
       5. resolveSeverity()              — INCIDENT_SEVERITY_OVERRIDE wins if set
       6. transmittable()                — Info requires INCIDENT_ALLOW_INFO
       7. claimFingerprint()             — Cache::add, 15 min, atomic across pods
       8. deliver()                      — queue on high-priority, inline fallback

Not reported at all: ValidationException, AuthenticationException, AuthorizationException, TokenMismatchException, ModelNotFoundException, any HttpException with status < 500, anything on a /health path, and anything originating inside App\Services\Incident\.

Gate order matters: the severity gate runs before the fingerprint claim, so a dropped alert never starts a cooldown window that would suppress a real one.

What the description carries: message (redacted), file:line, environment, release, request method + route, user id only (never name or email), Sentry event id, runbook URL. Redaction strips emails, token|secret|password|api_key|bearer assignments, and any 32+ character token-like string, then truncates at 500 characters.

Escalation​

SituationWho
webku application errors@bnl-dev-bali — #bnl-teams-b
Pods, database, ingress, networkingPlatform
ArgoCD access, or a sync you cannot roll backPlatform
Producer relay down or missing /incident-hooksProducer service owner
Cloudflare Pages / published-site deliveryPlatform

Lead with blast radius and whether it correlates with a release — not the stack trace. Those two facts determine the response; the trace only determines the fix.