Skip to main content

Runbook: Repush Complete Invoice Using Cron

ServiceChannelku V1 / Channelku V2
Owner Team slack handle[@bnl-team-a]
Team's Slack Channel#bnl-ota-notification

Table of Contents​

Alert / Notification#bnl-ota-notification
channel where XML delivery failure notifications are sent
Debug Channel#bnl-debug
channel where successful XML deliveries by the cron are confirmed
Alerting Channel#bnl-debuging
channel where alerts are sent when cron retry also fails
EndpointComplete Invoice
Cron Job URLCron Job URL
OTA Service EndpointOTA Service

1. Triage​

Quickly confirm whether an XML delivery failure to the OTA service is actively occurring, or whether the retry cron has already resolved it automatically.

    1. Check whether data still exists in tmp_queues:**
    • Run the following query on the production database: 
      SELECT COUNT(*) FROM tmp_queues WHERE queue_name = 'complete_invoice';
    • Is the result greater than 0? If yes, there is data that has not yet been successfully sent to the OTA service.
    • To inspect the details: 
      SELECT data FROM tmp_queues ORDER BY created_at ASC LIMIT 20;

  • 3. Check the #bnl-debug channel for cron success confirmations:

    • Open #bnl-debug.
    • Are there any successful XML delivery messages in the last 5–10 minutes? A success message looks like this: 
      Response Execute Cron
      Booking ID :  
      Booking Date :
      URL :
      Generated XML :
    • If success messages are present and tmp_queues is empty, this is likely not an active incident.

  • 4. Check the #bnl-debuging channel for retry failure alerts:

    • Open #bnl-debuging.
    • Are there repeated retry failure alerts in the last 15–30 minutes? A retry failure alert looks like this: 
      Hi @bnl-team-a, Please be informed that this reservation failed to repush from cron:
      Booking ID : 
      Property ID : 
      Timestamp : 
      Let’s monitor this repush. Thank you.
    • If yes, note down the affected reservation_id values.

2. Decision Point​

Interpret the triage results using the following conditions:

  • IF tmp_queues is empty (COUNT = 0) AND #bnl-debug shows recent success messages...

  • IF tmp_queues is not empty (COUNT > 0) AND #bnl-debuging shows repeated retry failure alerts across multiple cycles (≥ 3 times in the last 15 minutes)...

  • IF tmp_queues is not empty BUT there are no repeated alerts in #bnl-debuging (cron is still actively retrying)...

    • ➡️ Wait for 1 more cron cycle (5 minutes), then re-run triage from Step 2 to confirm whether the cron has successfully cleared the queue.

3. False Alarm​

The tmp_queues is empty and #bnl-debug confirms the retry cron has successfully delivered all XML payloads to the OTA service. There is no active incident. However, this failure pattern should be monitored, as it indicates the OTA service remains vulnerable under high request volumes.

  1. Re-confirm that tmp_queues is truly empty: 
    SELECT COUNT(*) FROM tmp_queues WHERE queue_name = 'complete_invoice';
    Expected result: 0

4. True Incident​

The tmp_queues still contains unprocessed data and the retry cron has been consistently failing to deliver XML payloads to the OTA service across multiple consecutive cycles. The first priority is to ensure no data in tmp_queues is lost, then identify and resolve the root cause.

4.1. Recover the System​

The following are the most likely root causes, ordered from most common to least common.

Potential Cause 1: OTA Service is Down or Overloaded​

The OTA service is unresponsive or returning errors due to an excessive number of incoming requests.

Diagnostic Steps:

  1. Check whether the OTA service endpoint is reachable: 
    curl -o /dev/null -s -w "%{http_code}" https://ota.bookandlink.com/health
    Expected result: 200. If the result is 5xx, 000, or a timeout, the OTA service is experiencing issues.

Remediation Plan:

  1. If the OTA service is confirmed down, do not force a manual retry — allow the cron to run naturally every 5 minutes until the OTA service recovers.
  2. Monitor #bnl-debug — once the OTA service recovers, success confirmations will begin appearing automatically.

Verification:

  1. Wait for success messages to appear in #bnl-debug.
  2. Run the following query to confirm the queue is shrinking: 
    SELECT COUNT(*) FROM tmp_queues WHERE queue_name = 'complete_invoice';
  3. Confirm tmp_queues is empty before closing the incident.

4.2. Clean Up​

Once the system is stable and tmp_queues is empty, the next step is to measure the impact, communicate with stakeholders, and ensure no reservation was missed.

  1. Verify tmp_queues is completely clear:

    SELECT COUNT(*) FROM tmp_queues WHERE queue_name = 'complete_invoice';

    Expected result: 0

  2. Verify in #bnl-debug that the last XML messages received correspond to the reservations that were previously stuck in the queue.

  3. Create or update the post-mortem ticket with the following details:

    • Timestamp of the initial failure and time of resolution.
    • Identified root cause.
    • Total number of affected reservations.
    • Action items to prevent recurrence (e.g., retry limit, dead-letter queue, monitoring threshold for tmp_queues size).
    • Assign to @bnl-dev for review in the next sprint.