# Gateway QA Readiness Check — `gatewayapiqa.alpa.org` vs `gateway.alpa.org`

> **TLDR:** `scripts/gateway-readiness-check.sh` replays every gateway route the app calls
> against production (baseline) and the QA gateway — and optionally the old dev gateway — as
> the anonymous identity plus one Key Vault member per MEC (DAL, UAL, FDX by default), then
> reports per cell whether status, schema and (for config-like routes) values match. Reads run
> by default; the eight server-mutating routes run only as named round-trip write flows under
> `WRITES=`, against `WRITE_HOSTS` (default `prod,qa`; the vault members are test accounts). Output: `report.md`, `summary.json`,
> `junit.xml`, `raw/`. Pipeline: `AzureDevOps/azdo_gateway_readiness.yml` (on demand).
> Work item: AB#2575 "Update dev gateway endpoint to gatewayapiqa.alpa.org".
>
> **Date:** 2026-09-02 · **Branch:** `imp/blazor-hybrid` (scripts) · **Owner:** mobile team

---

## 1. What is being tested

The app selects its hosts in one place, `RestService.SetEndpoints()` (`ALPAMobile/Services/RestService.cs:72-104`),
from the single `bool DevEnv` flag (`AppProperties.Key_DevEnv`):

| DevEnv | `BaseAPIUrl` / `BaseAPIUrlV2` | `BaseAuthUrl` | Since |
|---|---|---|---|
| `false` (Production) | `https://gateway.alpa.org` / `…/v2` | `https://authsvc2.alpa.org` | — |
| `true` (Development) | `https://gatewayapiqa.alpa.org` / `…/v2` | `https://authapi.alpa.org` | PR 2181 / AB#2575, 2026-08-24 (was `gatewayapi.alpa.org`) |

Two things the flag does **not** move, both of which the check covers explicitly:

- `DynamicContentApiClient` (`ALPAMobile.Infrastructure/Remote/DynamicContentApiClient.cs:41`) is hardcoded
  to `https://gatewayapi.alpa.org` — the `dynamic.*` rows hit that host from every "host" column, so identical
  cells are expected there; the row exists to show the app still depends on the old dev host in production builds.
- The auth host is separate from the gateway. Tokens are signed identically (`iss: none`, `aud: ALPA`) and are
  accepted by all three gateways interchangeably (verified 2026-09-02), so one credential set serves every host.

### Route inventory (48 gateway routes the app calls, 8 of them writes; 76 catalog rows including request variants, chained follow-ups, anonymous-write checks and 3 PilotComms subscription helpers)

| Group | Routes | Client | Writes |
|---|---|---|---|
| RestService hand-rolled `HttpClient` | 29 | `RestService.cs` | `POST /api/log` (telemetry append) |
| FTDT sync | 4 | `ALPA.Services.FTDT.ApiClientNet7_0` 7.0.8 | create (POST, upsert), update (PUT), soft-delete (DELETE) |
| Jumpseat | 2 | `Jumpseat.ApiClient` 1.0.0 | — |
| PilotComms (v2) | 4 | `PilotComms.ApiClient.Net8` 1.3.12 | `PUT sub/toggleoptin/{id}`, `POST regis/createupdate` |
| Menu & Favorites (v2) | 5 | `MobileMenu.ApiClient` 1.0.11 | `POST favorites`, `DELETE favorites/{type}/{id}` |
| Dynamic content (hardcoded host) | 3 | `DynamicContentApiClient` | — |
| Theme | 1 | `ThemeApiClient` via `RequestProvider` | — |

The catalog is `scripts/gateway-readiness/endpoints.json` — one row per route with the call-site `source`,
auth requirement, compare dimensions, volatile-field ignores and capture rules. `--catalog-check` diffs the
catalog against `docs/postman/ALPAMobile.postman_collection.json` and every `"/api/…"` literal in
`RestService.cs`, so a new route cannot land without a catalog row (exit 1 on a gap).

Saved searches, notification read-state and profile edits are **device-local** in this app — there is no
server write for them, so nothing to compare.

## 2. Identities

| Identity | Credential | What it proves |
|---|---|---|
| `anonymous` | none | 401 parity on member-only routes; public routes (page banners, settings, release status, item types, airport codes) return the same shape. Uses the first MEC for the per-MEC theme / dynamic-site routes, like the app does before login. |
| `dal`, `ual`, `fdx` | `uitest-member-prod-<MEC>-username/password` from `ALPAMobile-UITEST-VAULT` (hydrated by `scripts/uitest-credentials.sh`) | Member-scoped routes (carrier, member, favorites, subscriptions, FTDT) per MEC. Any MEC with a stored pair works: `MECS=dal,ual,fdx,jbu`. |

Only `prod` credentials exist in the vault (checked 2026-09-02, 23 MECs); the QA gateway accepts a token
minted by either auth host, so `--cred-env prod` is the default and there is nothing to provision for QA.
Login happens once per identity per host through that host's own auth host, exactly like
`RestService.AuthenticateUserViaJWTAsync` (`POST /api/token/auth`, `Domain: ALPA2K`, id canonicalised like
`AlpaIdHelper`). A failed login SKIPs that identity's rows on that host instead of retrying — repeated 401s
lock the account.

## 3. Comparison semantics

Every cell records HTTP status, latency, size, a recursive schema signature and the body. Each non-baseline host
is compared to the baseline (`prod`, first in `HOSTS`):

| Verdict | Meaning | Fails the run by default |
|---|---|---|
| `MATCH` | same status, same schema, and same values where the catalog lists `values` | — |
| `VALUE_DRIFT` | same shape, different values on a config-like route (settings, release status, item types, reference lists, reps, committees, company info, theme) | no — **observation only, by decision (Jose, 2026-09-02)**; it is listed in the report and never fails the run |
| `SCHEMA_DRIFT` | key added/removed or scalar type changed (null vs value is a value difference, not schema) | yes |
| `STATUS_DRIFT` | different HTTP status | yes |
| `ERROR` | transport failure / timeout on either side (one retry on 502/503/504, mirroring `RetryTransientAsync`) | yes |
| `SKIP` | precondition missing: login failed, captured var absent (`docPath`, `subscriptionId`), route not applicable anonymously | — |

Data-bearing routes (documents, events, notifications, flight search, favorites list, FTDT list) compare
status + schema only: QA and prod databases legitimately hold different rows. Their value differences are still
listed in the report's drift details for the reader.

Volatile fields (`id`, `createdDateTime`, `updatedAt`, …) are dropped before value comparison
(`globalIgnoreKeys` in the catalog); per-route `ignore` paths drop payload blobs such as file contents.

## 4. Write flows (`WRITES=`)

Each flow is a round trip that restores what it can and records every step as its own row. They run only for
member identities, only on `WRITE_HOSTS` (default `prod,qa` since 2026-09-02; restrict with `WRITE_HOSTS=qa`).

| Flow | Steps | Leaves behind |
|---|---|---|
| `log` | `POST /api/log` one `GeneralDebug` event, `DeviceId: gateway-readiness-check`, `CorrelationKey: <runId>` | one telemetry row |
| `favorites` | list → add (`itemTypeId` = first item type, `itemId: gateway-readiness-check-<mec>`) → list → delete → list; report checks count before == count restored | nothing |
| `subscription` | list subscriptions → if the member has none, find an audience (`GET sub/getall/{audienceId}?PageSize=1` for ids 1–10; `aud/getall` is administrator-only) → `POST sub/create/ALPAID/{alpaId}/AUDIENCEID/{id}` → toggle opt-in → toggle back (report checks `optIn` returned to the original) → `DELETE sub/delete/{id}` → list again. **`sub/create` is administrator-only as well (403 for member tokens on prod and QA, measured 2026-09-02)**, so with the vault members the flow stops at the create step and reports it; the toggle runs only once an administrator has subscribed the test members (audience 1 `TestAudience` exists on both hosts for exactly this). The setup/teardown routes are in the NSwag client but not called by the app | nothing when the flow created the subscription; two audit rows when it toggled a pre-existing one |
| `push` | `POST regis/createupdate` with `InstallId: gateway-readiness-check-<mec>` and a 64-hex, APNS-shaped token derived from the marker (the service answers 500 "Error creating or updating the item." to a non-hex token — measured, not a QA defect) | one registration row per MEC per host, reused on every run (idempotent by InstallId) |
| `ftdt` | create duty period (`Note: gateway-readiness-check <runId>`) → update with `Version + 1` and a fresh `UpdatedAt` (the service 409s unless **both** exceed the stored values; the app does the same bump in `FTDTDutyPeriodService.SaveAndSyncAsync`) → body-less `DELETE` → if that fails, `PUT Deleted=true` fallback → list with `Deleted=true` | one soft-deleted duty period per run on the test account; the app's sync pulls it as deleted |

`WRITE_HOSTS=prod,qa` is allowed (decided 2026-09-02, §8): the vault members are test accounts. `push` and `ftdt`
leave marker rows on them; `log`, `favorites` and `subscription` leave nothing user-visible. Note that the FTDT store
is shared between the two gateways (§6), so `ftdt` rows written via either host show up on both.

## 5. How to run

```bash
# prerequisites: az login (Key Vault Secrets User on ALPAMobile-UITEST-VAULT), python3 ≥ 3.10, no pip installs
HOSTS=prod,qa            ./scripts/gateway-readiness-check.sh                # reads; anonymous + dal,ual,fdx
HOSTS=prod,qa,dev        ./scripts/gateway-readiness-check.sh                # three-host view
MECS=dal,ual,fdx,jbu     ./scripts/gateway-readiness-check.sh                # more MECs (any with a stored pair)
WRITES=log,favorites,subscription,push,ftdt ./scripts/gateway-readiness-check.sh   # + write flows on prod and qa
./scripts/gateway-readiness-check.sh --only member,app,favorites             # subset by group or route id
./scripts/gateway-readiness-check.sh --fail-on ERROR,STATUS_DRIFT,SCHEMA_DRIFT,VALUE_DRIFT
python3 scripts/gateway-readiness-check.py --catalog-check                  # catalog covers Postman + RestService.cs?
```

Output lands in `test-logs/gateway-readiness/<runId>/` (git-ignored): `report.md` (matrix per identity + drift
details), `summary.json` (every cell, machine-readable), `junit.xml` (one test per route × identity × host),
`raw/<host>/<identity>/<route>.json` (status, timing, body). No token or password is ever written.

Pipeline **ALPA Mobile Gateway Readiness Check** (`AzureDevOps/azdo_gateway_readiness.yml`) wraps the same
script: parameters `hosts`, `mecs`, `writes`, `writeHosts`, `failOn`; publishes JUnit to the Tests tab and the whole
output directory as the `gateway-readiness` artifact. Same service-connection prerequisite as the credential
check (`ALPA-Mobile-UITest-Vault`, Key Vault Secrets User). Register it once from Pipelines → New → existing YAML.

Postman users: `docs/postman/ALPAMobile-QA.postman_environment.json` (added 2026-09-02) is the
`gatewayapiqa` / `authapi` environment; the older `ALPAMobile-Dev` file remains the pre-AB#2575 `gatewayapi` host.

## 6. Baseline findings — 2026-09-02

Run `20260902T212906Z`: hosts prod (baseline), qa, dev; identities anonymous, dal, ual, fdx; reads only;
38 routes × 4 identities × 3 hosts. Verdict counts per compared host:

| host | ERROR | STATUS_DRIFT | SCHEMA_DRIFT | VALUE_DRIFT | MATCH | SKIP |
|---|---|---|---|---|---|---|
| qa | 0 | 12 | 13 | 21 | 105 | 4 |
| dev | 0 | 12 | 10 | 10 | 119 | 4 |

The 4 SKIPs are the anonymous identity on routes that need a member (`getalpano?id={email}`, `doc/downloadfile`,
FTDT list). Findings, most severe first (every one reproduces for all four identities unless stated):

### Blocking on QA (STATUS_DRIFT)

1. **Jumpseat is blocked by Cloudflare on the QA and dev gateways** — **cleared 2026-09-08**, 200 on all hosts, no challenge header (see the re-test section below). `GET /api/jumpseat/policies` and
   `GET /api/jumpseat/airlines` return prod 200, qa/dev **403 with a Cloudflare "Just a moment…" managed
   challenge** — with or without a bearer, for every User-Agent tried (the app's `ALPADocs/… CFNetwork`, Safari,
   curl, none). Other `/api/*` routes on the same host are not challenged, so it is a WAF rule scoped to
   `/api/jumpseat/*`. The app cannot answer a browser challenge: Jumpseat is unusable against `gatewayapiqa`.
   Backend/infra: exempt `/api/jumpseat/*` from the managed challenge on `gatewayapiqa.alpa.org` and `gatewayapi.alpa.org`.

   **Not a harness artefact — confirmed from two more clients on 2026-09-02:**
   - *The app itself* (Debug build 5.0.16, iPhone 17 simulator, MauiDevFlow network capture): logged in as the
     DAL test member, Settings → Toggle Environment (re-login via `authapi`, every other route on
     `gatewayapiqa` 200), then Jumpseat → Airline Policies. `RestService.GetAirlineJumpseatPoliciesAsync`
     → `GET https://gatewayapiqa.alpa.org/api/jumpseat/policies` → **403**, `Server: cloudflare`,
     `cf-mitigated: challenge`, `cf-ray: a350090a6b386783-ATL`, 8.9 KB "Just a moment…" HTML; the screen
     renders empty. Request headers were the app's normal set (`Authorization: Bearer`, `Accept:
     application/json`, `sentry-trace`, `baggage`). Toggling the same session back to Production:
     `GET https://gateway.alpa.org/api/jumpseat/policies` → 200 in 0.8–1.4 s, airline list rendered.
     Capture: `test-logs/gateway-readiness/app-jumpseat-policies-qa.txt`, screenshots `sim-03-jumpseat-qa.png`
     / `sim-04-jumpseat-prod.png`.
   - *Newman* (Postman CLI, `scripts`-free ad-hoc collection, QA environment file): anonymous and bearer
     `GET /api/jumpseat/airlines` and bearer `/policies` → 403 `cf-mitigated: challenge`; control
     `GET /api/settings/list` with the same bearer → 200. Same collection against the Prod environment → all 200,
     served by `Microsoft-IIS/10.0` directly — production is not fronted by Cloudflare at all.
2. **Flight Finder notifications fail on QA and dev** — **cleared 2026-09-08**, 200 on all hosts. `POST /api/notifications/list` (`RestService.cs:1201`,
   `GetFlightNotificationListAsync`) returns prod 200, qa/dev **500** `"Unexpected during notification retrieval.
   Please try again later."` for anonymous and all three members.

### Response-shape differences (SCHEMA_DRIFT)

1. `GET /api/settings/list` — qa and dev **drop `genericSettings`** (prod carries the Jumpseat `EtiquetteHtml`
   entry). qa additionally rewrites `adfs.mappingList` (`jumpseatwebdev.alpa.org` → `login.microsoftonline.com`,
   `default2` → `dartqa.alpa.org`), replaces `auth.authorizationScript` with a promise-based script, and trims
   `defaultFeatures` from 3 to 2 GUIDs. Consumers: `AppConfigSettings`, ADFS auto-login WebView.
2. `GET /api/member/getuser` — qa and dev **add `rank`** (e.g. `First Officer`) and return **null** for
   preferences prod populates: `is_hotel_form_active`, `toolbox_email`, `grievance_form_email_address` (DAL, UAL,
   FDX), plus `is_mec_hotel_list_user` (UAL) / `is_mec_contacts_list_user` (FDX). Consumer: `Member.preferences`
   → MEC hotel / contacts / grievance gating.
3. `GET /api/FlightSearchV2/GetAirportAndCityCodes` — qa and dev **add `timeZoneId`** per airport (additive; the
   `AirportLocation` DTO ignores it today).
4. `GET /v2/api/doc/getdocuments` — qa **adds `categories[].preFetchCount`** (additive; dev does not have it).
   qa also answers in 4–5 s where prod takes 18–29 s and dev 15–26 s for the same member.

### Configuration differences (VALUE_DRIFT on config-like routes)

1. `GET /api/national/getkcmairports` — every `mapUrl` moves from `https://app2.alpa.org/sites/kcm/Airports/…`
   (prod) to `https://alpa5.sharepoint.com/sites/app_kcm/Airports/…` (qa, dev). `MapCacheService` downloads these
   anonymously; SharePoint URLs need a session, so KCM airport maps are expected to fail to cache on QA.
2. `GET /v2/api/mobilecontent/menuitems/getallforuser` — prod 35 items, qa 17, with different `path` values
   (`/settings` on prod vs `SettingsPage` on qa), glyphs and `showOnHomeScreen` flags. dev matches prod. The QA menu
   is a different menu configuration, not a filtered prod menu.
3. `GET /v2/api/mobilecontent/favorites/itemtypes` — prod 14 item types, qa 4. dev matches prod.
4. `GET /api/pagebanner/list` — qa serves 1 banner to authenticated members, prod 0 (anonymous: 0 on both).
5. `GET /api/appconfig/getrelease?appkey=ALPA&version=5.0.13` (smoke run) — prod answers `REQUIRED` ("A new app
   version is required"), qa `OK`. With the current `5.0.16` all hosts agree. QA has no forced-update rule for
   older builds.
6. `GET /api/national/getcontinentlist`, `GET /api/national/getaccidenttopics` — SharePoint-rendered HTML bodies
   differ in wrapper class ids and a few countries' content on qa/dev. Content edits, not shape.

### Flight search chain — 2026-09-04 (DAL; prod, qa, dev)

The four Flight Search routes were first exercised with static bodies, which proved little: `FindFlightInfo` for a
made-up DL 123 returned an empty `legs` list on every host. The catalog now chains them the way the app does
(`flight.search` captures the first result's leg and the first three flights; the follow-ups build their bodies
from those captures and assert a non-empty result):

| Step | Body | prod | qa | dev | Result |
|---|---|---|---|---|---|
| `POST /api/FlightSearchV2/SearchFlights` | ATL → DEN, tomorrow, non-stop + one-stop | 200, 25 flights, 4.3 s | 200, 25 flights, 3.2 s | 200, 25 flights, 3.7 s | MATCH |
| `POST /api/FlightSearchV2/FindFlightInfo` | first leg: departure airport + scheduled date + IATA carrier + flight number (`FlightLegMappers.ToDto`) | 200, 1 leg, `lastStatus: Scheduled`, `statusFresh: true` | same | same | MATCH |
| `POST /api/FlightSearchV2/FindFlightsInfo` | first three flights re-serialised like `Flight.ToAPIFormattedJson` (only those four fields per leg) | 200, 3 flights, 2 legs each, `statusUpdated: true` | same | same | MATCH |
| `POST /api/notifications/list` | `AudienceList: ["fs-<scheduleKey of that leg>"]` (the flight-change push audience, `PushNotificationHandler.cs:156`) | 200, 0 items | 200, 0 items | 200, 0 items | MATCH — nothing has registered that schedule, so an empty list is the expected answer |

**Per-mode searches (NONSTOP / ONE STOP / MULTILEG tabs)** — `JumpseatFlightFinderSearchPageViewModel.cs:986-988`
sends exactly one of `isNonStopSearch` / `isOneStopSearch` / `isTwoStopSearch` per tab with `lightningMode: true`.
Each mode is a separate row with a leg-count assertion (`assertEachLen`) on the search result and on its
`FindFlightsInfo` refresh; a violation is the `ASSERT_FAIL` verdict.

| Mode | Route | prod | qa | dev | Leg assertion |
|---|---|---|---|---|---|
| Nonstop | ATL → DEN | 11 flights, 0.4 s | 11, 0.2 s | 11, 0.3 s | 11/11 have exactly 1 leg; refresh 3/3 |
| One stop | ATL → DEN | 25 flights, 2.1 s | 25, 1.8 s | 25, 2.0 s | 25/25 have exactly 2 legs; refresh 3/3 |
| Multileg | ATL → ANC | 25 flights, 1.7 s | 25, 2.1 s | 25, 2.3 s | 25/25 have 3 or more legs; refresh 3/3 |

All MATCH across hosts (2026-09-04, DAL).

`FindFlightsInfo` rejects the raw search-result flight objects with 400 ("Required property 'carrierCode' not found
in JSON") on every host — that is the contract, not drift: the app strips every leg field marked
`JsonIgnoreSerialization` before sending. The `timeZoneId` schema drift on `GetAirportAndCityCodes` (qa, dev) remains.

### Coverage additions — 2026-09-04 (DAL + anonymous; prod, qa, dev)

Six gaps from the coverage review, each now a catalog row; the run that added them (`coverage-adds`, `coverage-adds-2`):

| Addition | Row(s) | Result |
|---|---|---|
| KCM map image fetch | `national.kcmMapImage0`, `…10` — GET the `mapUrl` of the 1st and 11th airport from that host's own `getkcmairports`, no bearer like `MapCacheService` | prod (`app2.alpa.org`) **200**; qa and dev (`alpa5.sharepoint.com`) **403** for anonymous and members — **KCM airport maps cannot be cached from the QA gateway**. `ASSERT_FAIL` |
| DNN document download | `doc.downloadFileDnn` — first document whose `source` is `DNN` | prod 200; qa and dev **SKIP: no DNN document** — the QA/dev `getdocuments` feed holds only the 42 Sitecore documents against 812 on prod (770 DNN). `getdocuments` compares status + schema, so this only shows as the skip |
| Flight Search variants | `flight.search.city` (isCity), `.page2`, `.sortTotalTripTime`, `.avoidConnection` [DFW], `.requireConnection` [ORD], `.requiredAirlines`, `.cargo`; `flight.findFlightInfo.multileg` (all legs of the first multileg itinerary) | all 200 with 25 flights and MATCH on every host; avoid/require assertions 25/25; `findFlightInfo.multileg` 3 legs. `requiredAirlines` takes airline **names** (`Delta Air Lines`, 25/25 legs Delta) — the IATA code form `DL` returns 0 flights on every host |
| Theme conditional fetch | `theme.byMecSince` (`?since=1`) | 404 on every host, same as the unconditional row: no theme published for DAL yet |
| FTDT paging | `ftdt.byLastUpdatedAt.page1` / `.page2` (Count=10, from 01-01-2012, as `FTDTDataSyncHelper` pages) | page 1: 9 rows on every host (the marker duty periods; same rows via all three gateways — shared store); page 2 empty; the AB#2364 fields appear on qa/dev only |
| Anonymous write parity | `favorites.addAnonymous`, `ftdt.createAnonymous` (expect 401), `pilotcomms.registerPushAnonymous`, `log.postEventAnonymous` | favorites and FTDT: 401 on every host. **`regis/createupdate` accepts an anonymous POST on every host (200, row stored with `userId: unauthenticated`)** — parity holds, backend to confirm that is intended. `/api/log` anonymous: 200 everywhere (the app logs before login) |

### Re-test 2026-09-08 — `gateway.alpa.org` vs `gatewayapi.alpa.org` (anonymous, DAL, UAL, FDX; reads)

Run `prod-vs-dev`: 259 cells compared, **196 MATCH, 0 STATUS_DRIFT**, 9 SCHEMA_DRIFT, 18 VALUE_DRIFT, 8 ASSERT_FAIL, 28 SKIP.

**Cleared since 2026-09-02** (verified on `gatewayapi` by the run and on `gatewayapiqa` by an anonymous spot-check the same day):

- `GET /api/jumpseat/policies` and `/airlines` — **200 on prod, qa and dev**, no `cf-mitigated` header. The Cloudflare managed challenge is gone. dev serves a superset: 105 airlines vs 103 (adds `Aero Air`, `Global Crossing Airlines`) and 137 policies vs 130 (airline ids 1, 67, 110, 135, 145, 148, 150 only on dev) — staged content, VALUE_DRIFT by design.
- `POST /api/notifications/list` — 200 on all three hosts (was 500 on qa/dev).
- `GET /api/settings/list` on dev — now identical to prod including `genericSettings` (qa not re-run for values).
- documents, menu items, favorites item types, page banners on dev — identical to prod.

**Still different on dev** (same as 2026-09-02): KCM `mapUrl` on SharePoint and the map images 403 (8 ASSERT_FAIL); `member/getuser` adds `rank` and nulls three preferences (DAL, UAL, FDX); `GetAirportAndCityCodes` adds `timeZoneId`; FTDT list carries the AB#2364 fields (dev ahead of prod); continent list / accident topics HTML wrapper ids.

### Observations that are not QA drift

- `GET /v2/api/pilotcomms/sub/getall/{audienceId}` is readable by any member token and returns **other members'
  subscription rows** (id, audienceId, user, fullName, optIn — 29 rows for audience 1 on prod). Backend to confirm
  that is intended; the app never calls it.
- `GET /api/jumpseat/policies` and `/airlines` answer **200 anonymously on prod** (722 KB of policy HTML with no
  bearer). The route is member content in the app; backend to confirm whether anonymous access is intended.
- `DynamicContentApiClient` rows are identical across hosts by construction (hardcoded `gatewayapi.alpa.org`);
  `GET /api/mobilecontent/dynamic/page/{id}` returns 404 for the first page of the `alpa` site on that host —
  `DynamicFeedQueries` resolves pages from the site payload first, so the app does not notice.
- Latency: qa is slower than prod on `getkcmairports` (1.4–1.6 s vs 0.4–0.5 s) and `getcontinentlist`
  (5.8–13 s vs 2.0–2.3 s); faster on `getdocuments` (above). Flight search is comparable (2.8–4.9 s vs 2.2–3.8 s).

### Parity confirmed (MATCH on qa and dev for every identity)

`auth.token`, `app.releaseStatus` (5.0.16), `member.c2aList`, `member.pacInfo`, `carrier.mecEvents`,
`carrier.lecEvents`, `carrier.mecReps`, `carrier.lecReps`, `carrier.companyInfo`, `carrier.mecAirports`,
`carrier.committees`, `national.kcmAirlines`, `national.resourceLinks`, `flight.search`, `flight.findFlightInfo`,
`flight.findFlightsInfo`, `pilotcomms.notifList`, `pilotcomms.subscriptions`, `favorites.list`, `theme.byMec`,
`dynamic.siteAlpa`, `dynamic.siteMec`, `dynamic.themeActive` — and every anonymous 401 on member-only routes
(`getuser`, `c2alist`, `pacinfo`, `getdocuments`, `favorites`, carrier routes, FTDT, PilotComms) is a 401 on all
three hosts. `ftdt.byLastUpdatedAt`, `doc.downloadFile` and `member.alpaNoByEmail` match for the three members.

### Write flows — 2026-09-02 (DAL, UAL, FDX on **prod and QA**; prod writes authorised by Jose, test accounts)

Run `writes-prod-qa`: every write step executed on both hosts and compared cell by cell. Earlier QA-only runs
(`writes-qa` … `writes-qa-5`) were used to measure the FTDT rules below.

| Flow | prod | qa | Verdict | Note |
|---|---|---|---|---|
| `log` | 200 | 200 | MATCH ×3 | — |
| `favorites` add / list / remove / list | 201 / +1 / 204 / restored | same | MATCH ×3 | separate stores (favorite ids prod 46–48, qa 11–13) |
| `push` `regis/createupdate` | 200 | 200 | MATCH ×3 | separate stores (registration ids prod 79505–79507, qa 3229–3231); 500 "Error creating or updating the item." for a non-hex token on both — format validation, not drift |
| `subscription` | 403 at `sub/create` (no subscription on prod yet) | **toggle 200 → 200**, `optIn` true → false → true ×3 | WRITE_OK ×6 on qa | 2026-09-04: Jose subscribed DAL, UAL, FDX to audience `Sept-2026-Test-Aud` in the admin console — that landed in **QA's** PilotComms store only (qa `sub/getallforuser` shows it, prod still `[]`). The toggle round trip is now exercised on qa; prod runs once the same membership exists there |
| `ftdt` create | 201 | 201 | SCHEMA_DRIFT ×3 | qa echoes `ImmediatePositioningMinutes`, `PreFdpSleepOpportunityHours`, `PreFdpSelfDeclaredInsufficient`; prod omits them — **AB#2364 backend gap still open on prod today** |
| `ftdt` update (`Version + 1`, fresh `UpdatedAt`) | 200 | 200 | SCHEMA_DRIFT ×3 | same three fields; 409 on either host for any other combination (7 variants measured) |
| `ftdt` soft-delete `DELETE /{id}` | **409** | **409** | MATCH ×3 | **service behaviour on both hosts, not QA drift** — see below |
| `ftdt` fallback `PUT Deleted=true` | 200 | 200 | SCHEMA_DRIFT ×3 | harness workaround; every marker row verified `Deleted: true` via both hosts, none live |

**FTDT is one database behind both gateways.** The marker duty periods created through `gatewayapiqa` and through
`gateway` are the same rows: identical `Id` sets and contiguous ids across the two hosts (DAL 169525…169544, UAL
169526…169546, FDX 169527…169548), and a delete through one host is visible through the other. The QA gateway only
serves a newer FTDT API (the AB#2364 fields). Consequence: **every FTDT write a tester makes with DevEnv on lands in
production duty-period data** — the FTDT beta-gate rule "P0 sync pushes seed data to prod accounts" applies to the
QA gateway too. PilotComms and MobileMenu/Favorites are separate stores on QA.

**FTDT soft-delete of an edited duty period fails on prod and QA.** The app deletes through
`FTDTDutyPeriodService.DeleteAndSyncAsync` (`Deleted = true`, `Version++`, `UpdatedAt = UtcNow`) →
`FTDTDataSyncHelper.SoftDeleteFDP` → `RestService.SoftDeleteDutyPeriodAsync` → body-less
`DELETE /api/ftdt/dutyperiod/{id}?Deleted=true&version=1.0`. That DELETE answers 409 on **both hosts** for any duty
period whose stored `Version` is above 1.0 (edited at least once since creation) and 200 for a never-edited one.
`RestService.cs:641` routes a 409 into `HandleConflict`, which **hard-deletes the local record and saves the server's
copy — with `Deleted = false`**. Expected member-visible effect: edit a duty period, delete it, sync, and it comes back.
Needs (a) an app-side confirmation on a device (edit → delete → cold relaunch, FTDT walkthrough), (b) a backend fix
to the DELETE handler's concurrency check (it appears to compare against a default `Version 1.0` rather than the stored
record), and (c) a note that `UpdatedAt`/`CreatedDate` come back date-only from create/update on both hosts.

## 7. Run log

| Date | Run | Hosts | Identities | Writes | Result |
|---|---|---|---|---|---|
| 2026-09-02 | smoke (`--only member,app,doc,favorites,theme,dynamic`) | prod, qa, dev | anonymous, dal | none | harness validated end to end; `docPath` capture fixed to `fileID` |
| 2026-09-02 | `20260902T212906Z` full reads | prod, qa, dev | anonymous, dal, ual, fdx | none | qa: 12 STATUS / 13 SCHEMA / 21 VALUE / 105 MATCH; dev: 12 / 10 / 10 / 119 — findings in §6 |
| 2026-09-02 | Jumpseat 403 cross-check: app on simulator (MauiDevFlow) + Newman | prod, qa | dal (+ anonymous in Newman) | none | Cloudflare challenge reproduced from the app's own `RestService` and from Postman CLI; prod control 200 both ways |
| 2026-09-02 | `writes-qa` … `writes-qa-5` (5 iterations while the FTDT update/delete rules were being measured) | prod, qa | dal, ual, fdx | log, favorites, push, ftdt on qa | log/favorites/push OK; FTDT create+update OK once Version and UpdatedAt are bumped; body-less DELETE 409 on edited records (§6); subscription flow skipped (no subscriptions on the test accounts) |
| 2026-09-02 | `writes-subscription`, `writes-subscription-2` | prod, qa | dal, ual, fdx | subscription on prod and qa | `aud/getall` 403; audience found via `sub/getall/1` (`TestAudience`); `sub/create` **403** on both hosts — administrator-only; toggle not reached |
| 2026-09-04 | `flight-chain`, `flight-chain-2` | prod, qa, dev | anonymous, dal | none | Flight Search chained from a live search result: FindFlightInfo 1 leg, FindFlightsInfo 3 flights, schedule-keyed notification list, all MATCH; raw flight objects → 400 on every host (contract) |
| 2026-09-04 | `flight-modes` | prod, qa, dev | dal | none | NONSTOP / ONE STOP / MULTILEG searches + per-mode refresh: leg counts 1 / 2 / ≥3 hold on every host (11 / 25 / 25 flights), all MATCH |
| 2026-09-04 | `coverage-adds`, `coverage-adds-2` | prod, qa, dev | anonymous, dal | none | 18 new rows: KCM maps 403 on qa/dev, no DNN documents on qa/dev, search variants all MATCH, anonymous push registration accepted everywhere (§6) |
| 2026-09-04 | `writes-subscription-3` | prod, qa | dal, ual, fdx | subscription on prod and qa | qa: toggle → optIn false → toggle back → true, restored, all three members (WRITE_OK ×6); prod: still no subscription → `sub/create` 403 as before |
| 2026-09-08 | `prod-vs-dev` | prod, dev | anonymous, dal, ual, fdx | none | 196 MATCH / 0 STATUS_DRIFT / 9 SCHEMA / 18 VALUE / 8 ASSERT_FAIL; Jumpseat challenge and notifications 500 cleared on qa and dev (§6) |
| 2026-09-04 | pipeline 103 run 5239 | prod, qa | anonymous only | none | first pipeline run: service connection `ALPA-Mobile-UITest-Vault` authenticated but its identity has no role on the vault → no member credentials; 64 routes anonymous, 2 STATUS_DRIFT (Jumpseat), 2 ASSERT_FAIL (KCM maps), JUnit published |
| 2026-09-02 | `writes-prod-qa` | prod, qa | dal, ual, fdx | all five on **prod and qa** | 36 MATCH / 15 SCHEMA_DRIFT (the AB#2364 fields) / 0 STATUS_DRIFT; DELETE 409 reproduces on prod; FTDT database shared between hosts (§6). Marker rows on the DAL / UAL / FDX accounts: 9 / 7 / 7, all soft-deleted, verified via both hosts |

## 8. Decisions

Decided 2026-09-02 (Jose):

- **VALUE_DRIFT is an observation, never a gate.** The run fails on ERROR / STATUS_DRIFT / SCHEMA_DRIFT / WRITE_FAIL
  only; configuration differences stay in the report for the reader.
- **Write flows may run against production.** The `uitest-member-prod-*` accounts are test accounts; changing their
  production data is acceptable. `WRITE_HOSTS=prod,qa` is the normal invocation for a full readiness pass. Keep the
  marker conventions (`Note`/`InstallId`/`itemId` = `gateway-readiness-check…`) so the rows stay identifiable.
- **Subscriptions: the flow provisions its own where the token allows, otherwise it toggles what exists.** `sub/create`
  is administrator-only (403 for member tokens), so the test members were subscribed by hand: DAL, UAL and FDX are in
  audience `Sept-2026-Test-Aud` on **QA** since 2026-09-04 and the toggle round trip runs there. The prod PilotComms
  store is separate and still has no subscription for them — add the same membership on prod to exercise the prod
  side; until then the prod cells report the 403 at `sub/create`.
- **Cadence: manual, tied to API change.** No schedule. Run the check (reads on `prod,qa`, all five write flows)
  whenever a major API endpoint change is planned or has landed on either gateway — before the app build that
  consumes it, and again after the backend deploy — and attach `report.md` to the work item.

Still open:

1. **FTDT shared database.** Decide whether the QA gateway should get its own FTDT store, or whether "DevEnv writes
   reach production duty-period data" is accepted and documented for testers (§6).
2. **FTDT DELETE 409 on edited records.** Backend ticket plus an app-side device confirmation (§6).
3. **`DynamicContentApiClient` host.** The check makes the hardcoded `gatewayapi.alpa.org` visible; splitting it
   Dev/Prod is the TODO at `DynamicContentApiClient.cs:39-40`, not this check's scope.