Skip to main content

Runbook: Validation Error - Partner Queue Full

ServiceCM-HUB
Owner Team slack handle@bnl-dev-bali
Team's Slack Channel#bnl-teams-b
Alert Channel#bnl-critical-alerts

Table of Contents​

1. Triage​

Quickly confirm if the Triplalink API is experiencing a temporary bottleneck.

  1. Check the specific error message in Slack:
    • Does the error include the text: "Please wait some time. XX request are waiting in the queue"?
    • If yes, this is a partner-side rate-limiting/congestion issue.
  2. Identify the Scope:
    • Is this happening for multiple Hotel IDs (e.g., HT25005870)? Check the #bnl-critical-alerts channel for a flood of similar messages.
  3. Check Partner Response:
    • Look at the ShortText field in the error. If the number of requests in the queue is increasing over time, the partner system is not clearing its backlog.

2. Decision Point​

  • IF the error explicitly states "requests are waiting in the queue"

3. False Alarm​

No false alarm for this issue

4. True Incident​

The partner API is currently saturated and cannot process further ARI updates, which may lead to inventory discrepancies.

4.1. Recover the System​

Follow these steps to ensure data synchronization:

Diagnostic Steps:

  1. Locate the failed ARI notification in the Slack channel.
  2. Verify the error message confirms the queue limit has been reached.

Remediation Plan:

  1. Initial Action: Click the Retry button on the Slack message to resend the ARI data from CM-HUB to Triplalink.
  2. If Error Persists: If after retry still receivng a "Queue Full" error in slack, wait for 10 minutes before attempting to click retry again. This allows the partner's system time to process its existing queue.

Verification:

  1. Monitor the logs/Slack for that specific Hotel ID.
  2. Verification is successful if no further errors occur for 15 minutes after the last successful retry click.

4.2. Clean up​

Once the system appears stable, perform the following housekeeping tasks:

  1. Acknowledge Notification: Add a checkmark emoji (✅) to the Slack notification/alert message to indicate that the error has been addressed.
  2. Final Monitoring: Continuously monitor the Slack notification channel for 30 minutes to ensure no new queue errors appear for the affected Hotel ID or Partner.