Skip to main content

Runbook: Corrupted Translation JSON (nested locale keys)

ServiceWebku (multi-tenant hotel website CMS)
Owner Team slack handle@bnl-dev-bali
Team's Slack Channel#bnl-teams-b
Alert Channel#bnl-critical-alerts
Alert signatureIlluminate\View\ViewException — htmlspecialchars(): Argument #1 ($string) must be of type string, array given

Verified against branch 79e0b0f3 (master e311c60f, v1.46.4) on 2026-09-25. Everything on this page is current behaviour on master, not planned.

[!danger] Read [[#4.1. Recover the System]] before running translation:fix The repair command is lossy in three different ways and silently blanks triple-nested values. It also exits 0 when you forget a flag, so a scripted run reports success having done nothing. Identify which shape you have first — [[#1. Triage]] step 1.3 gives you the SQL.

Table of Contents​

  • [[#Important Links]]
  • [[#1. Triage]]
  • [[#2. Decision Point]]
  • [[#3. False Alarm]]
  • [[#4. True Incident]]
    • [[#4.1. Recover the System]]
    • [[#4.2. Clean up]]
ResourceLink
Alert[Sentry issue — ViewException htmlspecialchars array given]
Sentry — all ViewException, last 7 days[Sentry search: error.type:ViewException]
Logs[OpenSearch/Kibana view filtered to app=webku]
Affected dashboard URL/{locale}/dashboard/sites/{site}/content/edit?page={page}
Affected public URLhttps://{tenant-domain}/{locale}/{page}
Command sourceapp/Console/Commands/FixTranslationJson.php
Helper source__localize() in app/Helpers/helpers.php

Related runbooks:

  • [[incident-triage-runbook]] — start there if you were paged and do not yet know it is this error
  • [[cannot-read-properties-of-undefined]] — a different dashboard render failure
  • [[google-analytics-permission-issue]] — analytics widget failures

1. Triage​

Goal: confirm it is this error, identify the one affected site, and — critically — find out which corrupted shape you have before touching anything.

1.1. Confirm the signature​

From the Sentry issue or the log:

Illuminate\View\ViewException
htmlspecialchars(): Argument #1 ($string) must be of type string, array given
  • It is fatal at render time. The page does not partially render — the whole response dies with a 500.
  • The stack trace names a themes.* Blade view. That view tells you which page; it does not tell you which field.

1.2. Establish blast radius​

CheckHowMeaning
How many sites?Sentry issue → Tags → urlAlmost always one. Many sites ⇒ this is not the runbook you want
Is the live site down too?curl -s -o /dev/null -w "%{http_code}\n" https://<tenant-domain>/<locale>/<page>500 ⇒ visitors affected
Is the published artifact still serving?Load the same page on the Cloudflare-served domainA previously published site keeps working until the next publish

[!note] The published site and the app-served site fail independently Both render the same data through the same helpers, so both break — but the deployed Cloudflare artifact is a frozen copy. If the site was published before the corruption, visitors may be fine while the dashboard is broken. Establish which one is actually down before declaring customer impact.

1.3. Find the corrupted rows — and read their values​

[!warning] --dry-run only prints a count, not the values The command's entire output is Done. N rows fixed. It does not show you which rows or what would change. You cannot decide whether the repair is safe from its output alone — use the SQL below.

Start with the table the failing page reads (contents for page content; see [[#Appendix B — Table and field reference]] for the rest):

SELECT id, page, placement, name, content
FROM contents
WHERE site_id = '<site_ulid>'
  AND JSON_VALID(content)
  AND EXISTS (
    SELECT 1 FROM JSON_TABLE(JSON_KEYS(content), '$[*]' COLUMNS (k VARCHAR(16) PATH '$')) AS t
    WHERE JSON_TYPE(JSON_EXTRACT(content, CONCAT('$."', t.k, '"'))) = 'OBJECT'
  );

Returns every row where a top-level key holds an object instead of a string — the corruption signature.

If your MySQL version rejects JSON_TABLE, fall back to the cheap version and eyeball it:

SELECT id, page, placement, name, content FROM contents
WHERE site_id = '<site_ulid>' AND content LIKE '%{"en":{%';

Copy the full content value of every row it returns into the incident ticket now. See [[#4.2. Clean up]] — the repair destroys the only evidence of the write path.

1.4. Classify the shape​

This determines which remediation you use. Compare against the table below:

Stored shape__localize() returnsClassification
{"en": "Hello", "ja": "こんにちは"}"Hello"✅ healthy — not your problem
{"en": {"en": "Hello"}}["en" => "Hello"]Simple double-nest → [[#Potential Cause 1 — Simple double-nested locale (the common case)]]
{"en": {"en": "A", "ja": "B"}}arrayMulti-locale nest → lossy, [[#Potential Cause 2 — Multi-locale or cross-locale nesting (lossy)]]
{"en": {"ja": "Konnichiwa"}}arrayCross-locale nest → lossy, [[#Potential Cause 2 — Multi-locale or cross-locale nesting (lossy)]]
{"en": {"en": {"en": "Hello"}}}arrayTriple nest → ⚠ command blanks it, [[#Potential Cause 3 — Triple-nested (the command destroys this)]]
{"0": {...}} — top level is not a localearrayUnrecognised → command does nothing, [[#Potential Cause 4 — Top level is not locale-keyed (command does nothing)]]

1.5. Check the locale config in the environment you will run from​

php artisan tinker --execute="var_dump(config('services.localization.locales'));"
  • Expected: an array such as ["en","ja","id"].
  • NULL ⇒ VITE_SUPPORTED_LOCALES is unset in that environment and the command will crash with TypeError: array_intersect(): Argument #2 must be of type array, null given on the first translation row. Fix the environment before running anything.

2. Decision Point​

  • IF the error signature matches (1.1), affects one site (1.2), and 1.3 returned at least one row...

    • ➡️ Go to: [[#4. True Incident]], then pick your cause from the classification in 1.4.

  • IF the signature matches but 1.3 returned no rows in the table you checked...

    • ➡️ The corrupted field is in a different table. Work through [[#Appendix B — Table and field reference]], then return to 1.3.

  • IF the error affects many sites, or the message is not htmlspecialchars(): … array given...

    • ➡️ Wrong runbook. Go to [[incident-triage-runbook]].

  • IF the app-served page renders fine now and the published site is serving correctly...

    • ➡️ Go to: [[#3. False Alarm]]

3. False Alarm​

Two situations look like this incident and are not:

3.1. The page renders again on its own​

The corruption does not self-heal. If the page now works, someone edited that field through the dashboard, which overwrote the nested value with a clean one. The data is fixed; the write path that produced it is not.

  1. Confirm with the SQL in 1.3 — it should return zero rows.
  2. Do not resolve the Sentry issue as "no action". Leave it unresolved with a comment naming the site, so a recurrence groups against it.
  3. Raise the follow-up in 3.3 anyway. A self-resolved instance is still an instance.

3.2. Only the published artifact is affected​

The Cloudflare-served copy is broken but the app-served page is fine. That means the corruption existed at publish time and has since been fixed in the dashboard.

  1. Verify the app-served page is genuinely healthy (1.2).
  2. Republish the site — see [[#4.2. Clean up]] for the 406 trap.
  3. No data repair is needed.

3.3. Post in Slack and raise the follow-up​

Into #bnl-teams-b:

webku — corrupted translation JSON, no action needed. Site [tenant-domain], page [page]. The nested value was already overwritten by a dashboard edit; the SQL check returns zero rows. Sentry issue left unresolved so a recurrence re-groups. Follow-up: [ticket link] — the write path is still unidentified.

Ticket title: RCA: webku nested locale JSON on [tenant-domain] — [date], tagged @bnl-dev-bali.

4. True Incident​

Severity: Daytime, unless the failing page is a high-traffic tenant's homepage or the live public site is returning 500 — then escalate to Business.

Announce in #bnl-teams-b:

🔴 webku — corrupted translation JSON. Site [tenant-domain], page [page], [dashboard only / live site also down]. [N] rows affected. Repairing now.

4.1. Recover the System​

Work from your classification in 1.4. Only Potential Cause 1 is safe to repair with the command.

Potential Cause 1 — Simple double-nested locale (the common case)​

Shape: {"en": {"en": "Hello"}}. The data is intact, just one level too deep.

Diagnostic Steps:

  1. Confirm every row from 1.3 has the same locale nested inside itself and nothing else inside it.
  2. Confirm the locale config is populated (1.5).

Remediation Plan:

  1. Dry run, always:

    php artisan translation:fix --field=content --table=contents --site-id=<site_ulid> --dry-run
    • Expected: Done. N rows fixed. where N matches the row count from 1.3.
    • 0 rows fixed → ➡️ [[#Potential Cause 4 — Top level is not locale-keyed (command does nothing)]]
    • Invalid JSON at ID <id> → that row is not decodable JSON; repair it by hand.

    [!warning] Check for typos before trusting a clean result All three options are required, but a missing one prints an error and exits 0. A scripted or piped invocation looks successful having done nothing. Verify the count, not the exit status.

  2. Run for real by dropping --dry-run.

  3. ⚠ There is no transaction and no backup. Rows are written one at a time; an interruption leaves a partial repair. Re-running is safe (already-flat rows are counted as unchanged and not written).

Verification:

  1. Re-run the SQL from 1.3 — expect zero rows.
  2. Reload the dashboard page — expect a normal render.
  3. Reload the public app-served page — expect 200.
  4. ➡️ The fix is not yet on the published site. Go to [[#4.2. Clean up]].

Potential Cause 2 — Multi-locale or cross-locale nesting (lossy)​

Shapes: {"en": {"en": "A", "ja": "B"}} or {"en": {"ja": "Konnichiwa"}}.

Diagnostic Steps: the inner object holds more than one key, or a key that is not the outer locale. Verified behaviour of resolveNested():

InputCommand producesConsequence
{"en": {"en": "A", "ja": "B"}}{"en": "A"}"B" is discarded — unless a top-level ja already carries its own value
{"en": {"ja": "Konnichiwa"}}{"en": "Konnichiwa"}⚠ Japanese text is now stored as the English translation. No warning

The preference order is: the same locale nested inside itself → en → the first string found, whatever locale it belongs to → ''.

Remediation Plan:

  1. Do not run the command blind. Read every affected value from the 1.3 output first.
  2. If the discarded/contaminated content matters, repair by hand: 
    UPDATE contents
    SET content = JSON_SET(content, '$.en', 'Hello', '$.ja', 'こんにちは')
    WHERE id = <row_id>;
    Match the site's active locales — check configurations.locales for that site.
  3. If the extra content is genuinely redundant, the command is acceptable. Record that decision in the ticket.

Verification: as Potential Cause 1, plus read the rendered page in each active locale and confirm the text is in the right language.

Potential Cause 3 — Triple-nested (the command destroys this)​

Shape: {"en": {"en": {"en": "Hello"}}}.

[!danger] Verified data loss resolveNested() inspects only one level. The inner value is an array, not a string, at every branch — so it falls through to return ''. The command reports the row as fixed and the field becomes an empty string. The original text is gone, with no warning and no backup.

Diagnostic Steps:

  1. From the 1.3 output, look for a value containing {"en":{"en":{.
  2. Confirm with: 
    SELECT id, JSON_EXTRACT(content, '$.en.en') FROM contents WHERE id = <row_id>;
    • If that returns an object rather than a string, you have triple nesting.

Remediation Plan:

  1. Do not run translation:fix on this row.
  2. Extract the real value and write it back directly: 
    -- inspect first
    SELECT JSON_UNQUOTE(JSON_EXTRACT(content, '$.en.en.en')) FROM contents WHERE id = <row_id>;

    -- then repair
    UPDATE contents
    SET content = JSON_SET(content, '$.en', JSON_UNQUOTE(JSON_EXTRACT(content, '$.en.en.en')))
    WHERE id = <row_id>;
  3. Repeat per affected locale.
  4. If other rows on the same site are simple double-nests, you may run the command afterwards — but only once the triple-nested rows are already flat.

Verification: as Potential Cause 1, and confirm the field contains the expected text rather than an empty string.

Potential Cause 4 — Top level is not locale-keyed (command does nothing)​

Shape: anything whose top-level keys are not locales, e.g. {"0": {...}}.

Diagnostic Steps:

  1. The command reports Done. 0 rows fixed. while the page stays broken.
  2. isTranslationArray() requires at least one top-level key to intersect config('services.localization.locales'). No match ⇒ the row is skipped entirely, with no output.
  3. Compare the row's top-level keys against that config (1.5).

Remediation Plan:

  1. ⚠ Do not keep re-running the command. It will never touch this row.
  2. Repair by hand, rebuilding the expected locale-keyed shape: 
    UPDATE contents SET content = JSON_OBJECT('en', 'Hello', 'ja', 'こんにちは')
    WHERE id = <row_id>;
  3. This shape suggests a different write path from the ordinary nesting bug — say so in the ticket.

Verification: as Potential Cause 1.

Potential Cause 5 — The command crashes on startup​

Diagnostic Steps:

TypeError: array_intersect(): Argument #2 must be of type array, null given

isTranslationArray() passes config('services.localization.locales') straight into array_intersect. extractLocales() returns null when VITE_SUPPORTED_LOCALES is empty, so an environment missing that variable crashes on the first translation row.

Remediation Plan:

  1. Confirm with 1.5.
  2. Set VITE_SUPPORTED_LOCALES in that environment's config (format: en:English, ja:Japanese), or run the repair from an environment where it is set.
  3. Do not work around it by editing the command.

Verification: 1.5 returns an array; re-run the dry run.

Potential Cause 6 — The field is in metadata​

[!warning] translation:fix cannot reach the metadata table It is polymorphic (seoable_id / seoable_type) and has no site_id column. The command applies where('site_id', …) unconditionally, so the query fails on an unknown column. Its title, keywords and description are all locale-keyed arrays, so the same corruption is possible there.

Remediation Plan: manual repair only.

SELECT m.id, m.seoable_type, m.title, m.description
FROM metadata m
JOIN rooms r ON r.id = m.seoable_id AND m.seoable_type = 'App\\Models\\Room'
WHERE r.site_id = '<site_ulid>';

UPDATE metadata SET title = JSON_OBJECT('en', 'Deluxe Room') WHERE id = <row_id>;

Repeat the join per seoable_type (Room, Facility, SpecialOffer, Activity, CustomListing).

4.2. Clean up​

1. Capture the evidence — do this first​

⚠ The write path that produces the nesting has never been identified. The pre-fix value is the only evidence, and the repair destroys it.

If you have not already saved the rows from 1.3, and you have not yet run the repair:

SELECT id, page, placement, name, content FROM contents WHERE id IN (<ids>);

Paste the full values into the incident ticket, with the site ULID, the page, and the timestamp of the site's last dashboard edit:

SELECT id, actor, timestamp, JSON_EXTRACT(meta, '$.name') AS changed
FROM audit_logs WHERE site_id = '<site_ulid>' ORDER BY id DESC LIMIT 20;

That last query is the best available lead on who or what wrote the bad value.

2. Get the repair onto the published site​

[!danger] The repair creates no pending change — publishing will return 406 FixTranslationJson writes with DB::table()->update(), bypassing Eloquent. The Auditable trait boots on Eloquent's created/updated events, so no AuditLog row is written. ReleaseManager looks for audit rows with is_deployed = 0 and at_version = 'N/A'; with none, publishing returns 406 "nothing to publish" and the live site keeps the broken content.

  1. Make a trivial edit through the dashboard on that site — change a heading and change it back. That produces a real audit row.
  2. Confirm a pending change exists: 
    SELECT COUNT(*) FROM audit_logs
    WHERE site_id = '<site_ulid>' AND is_deployed = 0 AND at_version = 'N/A';
    • Expected: >= 1.
  3. Publish from the dashboard.
  4. Verify the published page — do not assume the publish carried the repair.

The same applies to every manual UPDATE in [[#4.1. Recover the System]]: raw SQL writes are invisible to the audit trail.

3. Communicate​

Into #bnl-teams-b:

✅ webku — corrupted translation JSON resolved. Site [tenant-domain], page [page]. Impact: [dashboard only / live site 500 for N minutes] Shape: [double-nest / triple-nest / cross-locale / non-locale top level], [N] rows Fix: [translation:fix / manual SQL], republished at [time] Data loss: [none / "B" discarded on N rows / field blanked and restored manually] Follow-up: [ticket link] — write path still unidentified

If content was lost or a translation ended up in the wrong language, tell CS so the customer can be asked to re-check that page.

4. Post-incident​

Ticket: RCA: webku nested locale JSON on [tenant-domain] — [date], tagged @bnl-dev-bali.

Attach: the captured pre-fix values, the audit-log excerpt, the shape classification, and which remediation you used.

Push for the read-side guard. One condition in __localize() turns this from a fatal page error into a blank field — see [[#Appendix C — Why there is no prevention today]]. Every instance of this incident is an argument for it.

Appendix A — Why the page dies​

Localized text is stored as a JSON object keyed by locale, cast to an array on the model:

protected $casts = [
    'content'     => 'json',   // Content
    'name'        => 'json',   // Room, Facility, SpecialOffer, Activity, CustomListing
    'description' => 'json',
];

Themes read it through __localize():

function __localize($object, $key = false, $subkey = false)
{
    $locale = request()->segment(1, 'en');
    if (empty($object)) return "";
    if (!$key) return $object[$locale] ?? "";
    // …
}

The helper returns whatever sits under the locale key, without asserting it is a string. Blade compiles {{ $x }} to e($x), which calls htmlspecialchars(). Handed an array, PHP 8 throws a TypeError, and Blade wraps it as Illuminate\View\ViewException.

The locale nested inside itself is the signature — {"en": {"en": …}}. In the simple case the data is not lost, just one level too deep.

Appendix B — Table and field reference​

Every localized JSON column that also carries site_id, so the command can reach it:

TableFields
contentscontent
rooms, facilities, special_offers, activitiesname, description
custom_listingsname, description
menustranslation
configurationscontact, listing_config
metadatatitle, keywords, description — ⚠ no site_id, see Potential Cause 6

Options — all three of --table, --field, --site-id are required and validated individually:

OptionMeaning
--tableRaw table name (contents, rooms, …)
--fieldThe column holding the corrupted translation JSON
--site-idSite ULID — always applied as where('site_id', …)
--dry-runCount what would change; writes nothing and prints no values

Rows the command silently skips:

ConditionBehaviour
Field is null / not setSkipped, no output
Value is not decodable JSONWarns Invalid JSON at ID <id>, skips
No top-level key matches a configured localeSkipped, no output — Potential Cause 4
Already flatCounted as unchanged, not written

Operational characteristics worth knowing: the command loads every row for the site with no chunking (a large contents table means a large memory footprint), runs without a transaction, and has no test coverage in tests/.

Appendix C — Why there is no prevention today​

The failure is a write-side problem caught at render time, and neither side guards it:

  • __localize() does not assert that what it returns is a string.
  • No validation rejects a nested locale structure on save.
  • No test covers the malformed shape.

Cheap hardening, in order of value:

  1. Make __localize() return '' when the resolved value is not a string. One condition, and it converts a fatal page error into a blank field — a bad render beats no render.
  2. Validate on save that locale values are scalar, so the corruption cannot enter the database.
  3. Log when the helper hits a non-string, so corruption is discoverable before a visitor finds it.

Until (1) exists, a single malformed row takes down a whole page for every visitor.

If you change the command:

  • Do not "fix" the helper by flattening at read time and stopping there. That masks the corruption permanently and leaves the bad data in place.
  • Make resolveNested() recursive, or make it refuse a nested array rather than returning '' — the current silent blanking is the worst available behaviour.
  • Return a non-zero exit code on missing options, so scripted runs fail loudly.
  • Guard isTranslationArray() against a null locale config.
  • Print the affected ids and values under --dry-run so the operator can make an informed decision, which is what this runbook currently needs SQL to work around.
  • A site_id-less mode would let it reach metadata.
  • Any repair path that writes through DB::table() inherits the no-audit, no-pending-change behaviour in 4.2.

Escalation​

SituationWho
The repair, the command, the write path@bnl-dev-bali — #bnl-teams-b
Live public site returning 500 for a tenant@bnl-dev-bali, escalate to Business severity
Database access for manual UPDATEPlatform, if you lack a pod terminal

Lead with the shape classification (1.4) and whether the live site is affected. Those two facts decide both the remediation and the urgency.