Skip to main content

Runbook: Atlas ARI Logs Processing

ServiceChannelku V2
Owner Team Slack Handle@bnl-team-a
Team's Slack Channel#bnl-team-a

Table of Contents​

This section links to the most relevant resources for an engineer responding to this alert.

Alerthttps://sentry.bookandlink.net/
DB Consolehttps://argocd.bnlstg.com/applications/prod-channelku-api?resource=
Atlas JSON URLhttps://api.atlas.tripla.jp
Code RepoRepository - Atlas Handler

1. Triage​

The goal of this section is to quickly confirm whether there is an active issue on the {CM_V2}/api/v1/ari-logs endpoint, or whether the alert was transient and has already self-resolved.

  • 1. Check endpoint status:

    • Send a test request to the endpoint using a valid propertyId and txId with the correct static token: 
      curl -X POST https://[CM_V2_HOST]/api/v1/ari-logs \
        -H "Content-Type: application/json" \
        -H "Authorization: [STATIC_TOKEN]" \
        -d '{"propertyId": "[VALID_PROPERTY_ID]", "txId": "[NEW_TX_ID]"}'
    • Expected response (HTTP 200): 
      { "status": "success" }
    • If the response is 401 Unauthorized → the token is incorrect or missing. Check the token configuration in step 2.
    • If the response is neither 200 nor 401 → continue to the next steps.

  • 2. Check static token validity:

    • Confirm that the token being used matches the value configured in the application's environment variables.
    • Check the token env configuration.
    • If the token is mismatched or has been rotated, this is not an incident — update the token on the caller side and retry the request.

  • 3. Check connectivity to the Atlas JSON URL:

    • Run the following command from inside the application server/container: 
        curl -X GET https://api.atlas.tripla.jp/api/tx-artifacts/[VALID_PROPERTY_ID]/[NEW_TX_ID]/ari.json \
        -H "x-api-key: [API_KEY]"
    • Expected response: HTTP 200 OK. If the result is a timeout or 5xx, Atlas is experiencing issues.

  • 4. Check the process_queue table for the reported txId:

    • Run the following query in the DB console: 
      SELECT * FROM process_queue WHERE uniqid = '[TX_ID]';
    • Record exists → this txId has been processed before. The system should skip the insert into process_queue and only proceed with inserting into request_details_v2 and update_log_{propertyId}. Continue to step 5.
    • No record → this is a new txId. The system should insert into all three tables (process_queue, request_details_v2, and update_log_{propertyId}). If any insertion failed, continue to step 7.

  • 5. Check the request_details_v2 table:

    • SELECT * FROM request_details_v2 WHERE uniqid = '[TX_ID]' ORDER BY timestamp DESC LIMIT 10;
    • If process_queue already has a record (from step 4) AND request_details_v2 also has a record → This is a duplicate request (Scenario 2). The system is working correctly by returning success without creating new entries.
    • If process_queue has a record but request_details_v2 does not → The follow-up insertion failed. Continue to step 7.
    • If neither table has a record → The initial insertion failed entirely. Continue to step 7.

  • 6. Check the logs.update_log_{propertyId} table:

    • SELECT * FROM `logs`.`update_log_[PROPERTY_ID]` WHERE flatfiles = '[TX_ID]';
    • Confirm that the latest history entry matches the timestamp of the reported request.

  • 7. Check the tmp_queues table for failed insert records:

    • This table stores the raw queries of records that failed to be processed into the main tables. It is the primary source for investigation and recovery during insertion errors.
    • SELECT * FROM tmp_queues WHERE queue_name = 'atlas-log';
    • If records exist here → it confirms that the insertion failed. The stored raw queries can be used for manual re-insertion into the appropriate tables.

2. Decision Point​

  • IF all triage checks passed — the endpoint responds 200, records are found in all expected tables, and there are no errors in the logs or records in tmp_queues...

  • IF the txId already exists in process_queue AND already exists in request_details_v2...

    • ➡️ This is the expected behavior for Scenario 2 (Duplicate Request). The system is working correctly — the insertion was skipped to prevent double logging. No further action is required.

  • IF the txId already exists in process_queue BUT does not yet exist in request_details_v2 or update_log_{propertyId}...

  • IF the txId does not exist in either process_queue or request_details_v2, and there is a record in tmp_queues...

  • IF the Atlas URL is unreachable, or the endpoint responds with something other than 200 or 401...

  • IF the endpoint responds with 401 Unauthorized, or a request without a token is not rejected (middleware not running)...

3. False Alarm​

The alert may have been triggered by a transient latency spike, a momentary DB connection issue, or a brief Atlas URL slowdown, but the system has since recovered. There is no active incident.

  1. Confirm the system is stable again:
    • Repeat the test request from the triage step. Confirm the response is 200 and records are successfully inserted into all expected tables.

4. True Incident​

When an active incident on the {CM_V2}/api/v1/ari-logs endpoint is confirmed, the first priority is to restore the system. Once stable, assess the blast radius and perform cleanup.

4.1. Recover the System​

The following are the potential root causes, ordered from most to least common.

Potential Cause 1: Atlas JSON URL Unreachable​

The endpoint failed to fetch ARI data from Atlas, preventing the process from continuing.

Diagnostic Steps:

  1. Run the following from inside the server/container: 
    curl -v https://api.atlas.tripla.jp
  2. Is the response a timeout, connection refused, or HTTP 5xx? If yes, this is the root cause.
  3. Check the application logs for error messages such as: Failed to fetch Atlas JSON or Connection timed out.

Remediation Plan:

  1. Contact the Atlas JSON URL owner team via Slack and report that the URL is unreachable.
  2. Do not re-process requests until the Atlas URL is accessible again, to avoid data inconsistencies.

Verification:

  1. Re-run curl -I https://api.atlas.tripla.jp and confirm it returns HTTP 200 OK.
  2. Send a test request to the endpoint and confirm the response is 200 and records are successfully inserted into all expected tables.

Potential Cause 2: Insertion Failure into request_details_v2 and/or update_log_{propertyId}​

There are two insertion failure scenarios based on the state of process_queue:

  • Partial failure: the txId already exists in process_queue (the request was previously processed), but the follow-up insertion into request_details_v2 or update_log_{propertyId} failed.
  • Total failure: a new txId where the insertion into all three tables (process_queue, request_details_v2, update_log_{propertyId}) failed entirely.

In both cases, the failed records are stored as raw queries in the tmp_queues table and can be used for recovery.

Diagnostic Steps:

  1. Check the tmp_queues table for the affected txId:

    SELECT * FROM tmp_queues WHERE queue_name = 'atlas-log';

    If records exist here, copy the stored raw queries — these will be used in the remediation step.

  2. Run the following query to verify the table is writable:

    INSERT INTO process_queue (uniqid, message, submit_timestamp) VALUES ('test-ping-[TIMESTAMP]', 'test', NOW());
    -- If successful, delete it immediately:
    DELETE FROM process_queue WHERE uniqid = 'test-ping-[TIMESTAMP]';

  3. Check for active deadlocks or connection pool exhaustion on the DB dashboard.

Remediation Plan:

  1. If the issue is a connection pool problem, restart the application service.

Verification:

  1. Repeat the insert test query above — confirm it succeeds without error.
  2. Confirm records exist in the expected tables based on the txId condition:
    • If the txId is new: check process_queue, request_details_v2, and logs.update_log_{propertyId} — all three must have records.
    • If the txId already existed in process_queue: check request_details_v2 and logs.update_log_{propertyId} — both must have records.
  3. Confirm there are no remaining pending records in tmp_queues for that txId.

Potential Cause 3: Static Token Authentication Failure​

Requests are being rejected because the token is invalid, missing, or has been rotated — resulting in 401 Unauthorized and no processing occurring. Alternatively, the authentication middleware is not running and requests without a token are being accepted.

Diagnostic Steps:

  1. Send a request without a token and confirm the system rejects it: 
    curl -X POST https://[CM_V2_HOST]/api/v1/ari-logs \
      -H "Content-Type: application/json" \
      -d '{"propertyId": "[VALID_PROPERTY_ID]", "txId": "[TEST_TX_ID]"}'
    Expected response: 401 Unauthorized. If the response is not 401, token validation is not running — there is a bug in the authentication middleware.
  2. Check the active token value in the application env configuration.

Remediation Plan:

  1. If the token has been rotated and the caller is using an old token, update the token on the caller side to match the latest value in the configuration.
  2. Review and fix the authentication middleware logic in the code: Repository - Atlas Handler

Verification:

  1. Send a request without a token — confirm the response is 401 Unauthorized.
  2. Send a request with the correct token — confirm the response is 200 and the process runs normally.

4.2. Clean up​

Once the system has recovered, use the following steps to assess the impact, communicate with stakeholders, and ensure no data was lost using the raw queries stored in the tmp_queues table.

  1. Identify all records that failed to insert via tmp_queues.

    • Run the following query to retrieve all failed records within the incident time window: 
      SELECT * FROM tmp_queues
      WHERE created_at BETWEEN '[INCIDENT_START_TIME]' AND '[INCIDENT_END_TIME]'
      ORDER BY created_at ASC;
    • Note the total count and list of affected txIds. Save the results to the incident ticket.

  2. Communicate the impact to stakeholders.

    • Post the following message in [#your-team-channel]:

      Post-Incident Update {CM_V2}/api/v1/ari-logs: The system is now stable. We have identified [COUNT] records that failed to process during the incident ([START_TIME] – [END_TIME]). Raw queries are available in the tmp_queues table and will be re-executed for recovery. No data was corrupted — only missing entries.

  3. Execute raw queries from tmp_queues to recover the failed data.

    • For each record found in tmp_queues, retrieve the raw query and execute it against the appropriate table. Check the txId condition before executing:
      • If the txId does not yet exist in process_queue → execute the raw queries to insert into all three tables: process_queue, request_details_v2, and logs.update_log_{propertyId}.
      • If the txId already exists in process_queue → execute only the raw queries to insert into request_details_v2 and logs.update_log_{propertyId}. Do not re-insert into process_queue.
    • Example of how to retrieve and execute raw queries from tmp_queues: 
      -- View the raw query for a specific txId
      SELECT data FROM tmp_queues WHERE queue_name = 'atlas-log';

      -- Execute the raw query manually after verifying it
    • After each raw query is successfully executed, delete the record from tmp_queues: 
      DELETE FROM tmp_queues WHERE queue_name = 'atlas-log';

  4. Verify the cleanup is complete.

    • Confirm all affected txIds now exist in the expected tables: 
      -- Check process_queue
      SELECT uniqid FROM process_queue WHERE uniqid IN ('[TX_ID_1]', '[TX_ID_2]', '...');

      -- Check request_details_v2
      SELECT uniqid FROM request_details_v2 WHERE uniqid IN ('[TX_ID_1]', '[TX_ID_2]', '...');
    • Confirm tmp_queues is clear of all affected records: 
      SELECT COUNT(*) FROM tmp_queues
      WHERE created_at BETWEEN '[INCIDENT_START_TIME]' AND '[INCIDENT_END_TIME]';
      -- Expected result: 0
    • Post a final confirmation in the Slack thread:

      Cleanup complete. All [COUNT] records have been successfully recovered from tmp_queues into the appropriate tables (process_queue, request_details_v2, logs.update_log_{propertyId}). The tmp_queues table is now clear.