Skip to main content

Runbook: Waiting Queue Warning [ALERT]

logo

ServiceOTA Worker
Owner Team slack handle@bnl-channel-integration @bnl-team-c
Team's Slack Channel#bnl-critical-alerts

Table of Contents​

ARI Partner Sync Monitoringhttps://portal.bookandlink.com/tools/sync-process

1. triage​

A. Dashboard Monitoring​

Steps:

  1. First, check the ARI Partner Sync Monitoring.
    ARI Partner Sync Monitoring

  2. If the queue shows a significant decrease:

    • Notify in the thread related to this error.
    • Inform stakeholders that the queue is decreasing, which indicates the system is working, but there are too many requests.
    • Ask for assistance from Mr. Sukerta or the team lead.
      info

  3. If there is no decrease, proceed to Monitoring OTA Worker Pod step

B. Monitoring OTA Worker Pod​

Steps:

  1. If the queue keeps piling up, check the pod in Argo CD.
  2. If the pod is unhealthy or has issues:
    • Contact the platform team immediately.
    • Report the incident.
  3. If the pod is healthy, but the queue continues to pile up, proceed to the next internal step.

C. Next Step​

Steps:

  1. If the queue still piles up even though the pod is healthy:
    • Check related application logs to identify potential errors or bottlenecks.
    • Report findings to the development or platform team.
  2. Ensure all relevant stakeholders receive updates on the queue status.
  3. Document all actions taken in the monitoring thread for audit and future issue analysis.
  4. If the problem persists, escalate to the manager or team lead for further action.

2. Decision Point​

This section helps you decide if this is a true incident or a false alarm.

  • IF the ARI Partner Sync queue shows a significant decrease and the OTA Worker Pod is healthy
  • IF the ARI Partner Sync queue does not decrease, OR the OTA Worker Pod is unhealthy, OR the queue keeps piling up despite the pod being healthy…

3. False Alarm​

A false alarm is not a "close and ignore" event. Follow the protocol below to safely handle the alert.

Immediate Action Plan​

  1. Monitor the queue actively:

  2. Decide on alert handling:
    Depending on the situation:

    • Keep triggered: Leave the monitor active to ensure it can alert again if the queue starts piling up.
    • Adjust threshold: Raise the alert threshold temporarily so that minor fluctuations do not trigger repeated false alarms.
    • Mute alert temporarily: Only if the monitor is flapping between triggered and recovered states, e.g., silence for 1 hour.

  3. Communicate in Slack:

Heads up: The [ALERT_NAME] triggered but the queue is decreasing, indicating no incident. Action taken: [KEPT/TRIGGERED / THRESHOLD ADJUSTED / MUTED FOR 1 HOUR]. Monitoring will continue for any escalation.

4. True Incident​

Context:
A true incident occurs when either:

  • The ARI Partner Sync queue does not decrease, or
  • The OTA Worker Pod is unhealthy, or
  • The queue keeps piling up even though the pod is healthy.

In this case, the priority is to restore service, assess impact, communicate, and perform cleanup.

4.1 Recover the System​

Step 1: Identify Root Cause​

Potential Cause 1: Pod Unhealthy​

Diagnostic Steps:

  1. Open Argo CD and check the OTA Worker Pod status: ota pod
  2. Confirm pod conditions: Ready vs CrashLoopBackOff or other errors. .

Remediation Plan:

  1. If pod is crashing, restart or redeploy the pod using Argo CD.
  2. Inform the team lead, Pak Sukerta, or the platform team