Add transjakarta-live-data research topic

Verified TransJakarta realtime stack: anonymous MQTT bus feed (wss://mqtt.tj.co.id:8084/mqtt), guest REST API (tijeapi.transjakarta.co.id), official GTFS. Cross-capture verification vs third-party tracker, connection recipes, snapshot tool.

transjakarta-live-data/analysis.md

@@ -0,0 +1,49 @@
+# Analysis — architecture and verification evidence
+
+## How the stack works
+
+1. Buses carry GPS devices; vehicles are identified by fleet body numbers (`MYS-…`, `TJ-…`, `KWK-…`, `DMR-…`, `PRM-…`; the prefix appears to be operator/depot).
+2. TransJakarta's backend ingests positions and publishes each bus to its MQTT broker; REST state backs the official app.
+3. Client flow: login (guest works) → `POST /v1/bus` with your location → you receive nearby buses **plus MQTT topic wildcards for the map cells around you** → subscribe those on the broker → live positions stream in. REST covers routes/schedules; GTFS covers static geometry.
+4. The upstream path from vehicle hardware into the broker is inferred — it is not observable from outside.
+
+## The MQTT feed
+
+- Broker: `wss://mqtt.tj.co.id:8084/mqtt` — MQTT 3.1.1 over WebSocket Secure, **anonymous** (no username/password; verified with independent clients).
+- Topic schema: `/mobile_armada/<grid-cell>/<route-code>/<trip-id>/<bus-body-no>`
+  - Real captured example: `/mobile_armada/0021-0014/JAK.49/JAK.49-L03/KWK-1518`
+  - `<grid-cell>` = `{lat-index}-{lng-index}`, a ~0.01° (~1.1 km) map cell. Verified by probing the discovery API with offset coordinates: ±0.1° of latitude/longitude shifts the matching index by ±10 (index grows southward and eastward).
+- Retained messages: subscribing delivers ~2,000+ last-known states within seconds, then live updates continue. Per-bus refresh observed at ~8–75 s (some active vehicles faster; many ~25–37 s); delivery is bursty.
+- Fleet size observed live: 3,300–3,722 buses.
+
+### Payload schema (complete key list, captured verbatim)
+
+`latitude`, `longitude`, `snap_latitude`, `snap_longitude`, `bearing`, `estimated_time_next_stops`, `estimated_distance_next_stops`, `curr_stops`, `next_stops`, `prev_stops`, `passengers_status`, `route_code`, `route_long_name`, `route_color`, `route_text_color`, `trip_short_name`, `trip_headsign`, `bus_type`, `type`, `trip_id`, `bus_body_no`, `stops[]`, `timestamp`, `last_update`, `eta_by_parent_stop_id`, `next_parent_gate`, `next_parent_gate_eta`, `next_gate`, `next_gate_eta`, `direction`, `eta`.
+
+- `stops[]` items: `name`, `stop_id`, `latitude`, `longitude`, `stop_sequence`, `time_from_prev`, `time_wait`, `headsign`, `parent_stop_id`, `parent_stop_name`, `eta`.
+- `timestamp` is epoch milliseconds (e.g. `1791797549000`). Stop IDs (`B00436P`-style) match GTFS.
+- Some fields are null for some vehicles (e.g. gate fields, `passengers_status`, `last_update`).
+
+## The REST API
+
+- `POST /v1/auth/login/guest` → `token` + `refresh_token` + `user{email: <random>@guest.tije.com, is_guest: true}`. Token lifetime ~24 h. Headers used: `User-Agent: okhttp/4.12.0`, `X-App-OS: android`, `X-App-Version: 2.10.2`, `X-Device-ID: <string>`.
+- `POST /v1/bus` body `{latitude, longitude, radius, event: "change_loc"}` → `data { buses[], polyline, session_id, stops[], topics[] }`. `topics` = the MQTT wildcards for nearby cells (e.g. `/mobile_armada/0018-0022/#` …). This is the app's discovery mechanism.
+- `GET /v1/bus?latitude=&longitude=&radius=` → array of nearby bus records (~15 at radius=1 near Monas; radius units unverified).
+
+## Verification case: oyenza serves this same data
+
+Simultaneous captures (node WS client on oyenza's stream + node MQTT client on TJ's broker):
+
+- 1,794 buses common to both (of 1,823 seen on the broker in the same window).
+- 100% agreement on `trip_id` and `route_code`; 99.7% on headsign.
+- Position deltas: median 8.9 m, p90 42 m, 1,783/1,794 ≤ 100 m — consistent with reading the same moving buses a few seconds apart; no other explanation fits.
+- Fleet overlap 97.8% over a 75 s window (2,869/2,933); remainder = capture-window skew.
+- Category taxonomy maps 1:1, including the edge case: the app's whole "other" bucket (2 buses) = the only two untyped buses in the feed.
+
+oyenza's own layer (what it adds): fleet-state backend; browser protocol `hello/feed/snap/delta` with positions packed as integers ×10⁵; deltas every 3.00 s (~1,000 position records each); full snapshot 3,722 buses; its own REST extras; a replay mode. Static layer from the official GTFS (240 routes, 698 shapes, 7,493 stops at capture). Its About panel states: "Posisi bus — feed langsung TransJakarta" and "Tidak berafiliasi".
+
+## Confidence
+
+- **Established (verified by direct capture):** guest token; both bus endpoints; anonymous MQTT access; topic schema + grid decoding; payload schema; retained behavior; GTFS file; Traccar-based bustrack; oyenza data identity.
+- **Community-documented (not independently re-verified):** ~105 REST endpoint count; Tara chatbot WS; Kong gateway; refresh-token flow.
+- **Inferred:** vehicle→broker ingest path; oyenza's exact subscription method (whole-tree vs per-cell — data identity holds either way); `radius` units.

transjakarta-live-data/index.md

@@ -0,0 +1,43 @@
+---
+title: "TransJakarta live data — verified stack: MQTT stream, app REST API, GTFS"
+description: "How TransJakarta realtime bus data works and how to use it — anonymous MQTT broker, guest REST API, official GTFS. Verified by cross-capture, Oct 2026."
+status: concluded
+category: technical
+tags: [transjakarta, mqtt, realtime, gtfs, reverse-engineering, transit-data]
+draft: true
+created: 2026-10-12
+updated: 2026-10-12
+---
+
+# TransJakarta live data — verified stack
+
+**What this is:** a verified map of TransJakarta's public realtime data services — the MQTT live-bus stream, the official app REST API, and the GTFS static feed — plus ready-to-use connection recipes.
+
+**Origin:** reverse-engineering `transit.oyenza.com` (a third-party tracker) to determine what it uses. The trail led to TransJakarta's own services, which are reachable directly. Cross-capture verification proved the third-party tracker serves the same data as the official feed — see [[transjakarta-live-data/analysis|Analysis]].
+
+## TL;DR — the three services
+
+1. **Live positions (MQTT):** `wss://mqtt.tj.co.id:8084/mqtt` — anonymous, no key needed. Topics `/mobile_armada/<cell>/<route>/<trip>/<fleet-no>`; one JSON payload per bus.
+2. **App REST API:** `https://tijeapi.transjakarta.co.id` — guest JWT via `POST /v1/auth/login/guest` (no account). `POST /v1/bus` returns nearby buses **and** the MQTT topics to subscribe.
+3. **Static schedule:** `https://gtfs.transjakarta.co.id/files/file_gtfs.zip` (~2.4 MB, open download; version observed 27 Jul 2026).
+
+## Key verified numbers (Oct 12, 2026)
+
+- Live fleet: ~3,300–3,720 buses (BRT, Mikrotrans, Angkutan Umum Integrasi, Transjabodetabek, Royaltrans, Rusun).
+- oyenza cross-capture: 1,794 common buses, 100% trip/route ID agreement, median position delta 8.9 m, 97.8% fleet overlap over 75 s.
+- Broker: ~2,000+ retained messages delivered instantly on subscribe; per-bus refresh ~8–75 s in samples.
+- Third-party tracker's own browser stream: full snapshot (3,722 buses) + deltas every 3.00 s.
+
+## Using it
+
+- Copy-paste recipes (curl / node): [[transjakarta-live-data/recipes|Recipes]]
+- Ready tool: [[transjakarta-live-data/scripts/tj-snapshot.mjs|scripts/tj-snapshot.mjs]] — capture a live snapshot to JSON.
+- Evidence archive: `workspace/transjakarta-live-captures/` (raw captures, scripts, bundles).
+
+## Files
+
+- [[transjakarta-live-data/sources|Sources]] — endpoints, repos, references
+- [[transjakarta-live-data/analysis|Analysis]] — architecture + verification evidence
+- [[transjakarta-live-data/methodology|Methodology]] — how it was verified (repeatable)
+- [[transjakarta-live-data/recipes|Recipes]] — connection examples
+- [[transjakarta-live-data/next-steps|Next steps]] — risks + future ideas

transjakarta-live-data/methodology.md

@@ -0,0 +1,27 @@
+# Methodology — how this was verified
+
+Repeatable procedure (executed Oct 11–12, 2026). Tools: `curl`, `node` (packages `ws`, `mqtt`), `jq`.
+
+## Steps
+
+1. **Map the target:** fetch the site's HTML + JS bundles with curl; read the CSP header (`connect-src` names allowed WebSocket endpoints); grep bundles for protocol strings → found oyenza's own WS (`/api/live`) and its message types (`hello/feed/snap/delta`).
+2. **Find static-data citations:** `static.json` referenced the official GTFS zip URL → static layer source identified.
+3. **Find the upstream:** search GitHub for projects in the same ecosystem (`tj-live`, `open-tj`) → they document the REST + MQTT + GTFS stack.
+4. **Simultaneous cross-capture:**
+   - `oyenza-ws.mjs` — WS client on `wss://transit.oyenza.com/api/live`
+   - `mqtt-cap.mjs` / `mqtt-long.mjs` — MQTT client on `wss://mqtt.tj.co.id:8084/mqtt`, subscribing `/mobile_armada/#`
+   - Analysis with `compare.mjs` / `compare2.mjs` (ID sets, field agreement, position deltas), `seg1.mjs` (topic segments), `mqtt-rate.mjs` (rate profile).
+5. **Decode the topic grid:** four `POST /v1/bus` probes at ±0.1° offsets → index-shift mapping (≈0.01°/unit).
+6. **Cadence check:** re-connect to oyenza's WS and record delta intervals (`oy2.mjs`) → exactly ~3.00 s.
+
+## Key results
+
+- 1,794 common buses; 100% trip/route ID match; median position delta 8.9 m; p90 42 m; 97.8% fleet overlap (75 s window).
+- Broker: ~2,000+ retained messages instantly on subscribe; per-bus refresh ~8–75 s in samples.
+- oyenza deltas: 3000–3013 ms gaps, ~1,000 position records each.
+
+## Artifacts
+
+- All scripts + raw captures archived at `workspace/transjakarta-live-captures/` (4.6 MB): oyenza page + bundles, oyenza WS captures, MQTT captures, comparison outputs, rate profiles, tije REST responses (guest token redacted).
+- Scratch copies live under `~/.hermes/profiles/research/cache/scratch/oyenza/` (auto-pruned after ~24 h idle — use the archive instead).
+- Reusable snapshot tool: [[transjakarta-live-data/scripts/tj-snapshot.mjs|scripts/tj-snapshot.mjs]]

transjakarta-live-data/next-steps.md

@@ -0,0 +1,23 @@
+# Next steps / open questions
+
+## Open questions
+
+- Exact ingest path from vehicle GPS hardware into the broker — not observable from outside.
+- oyenza's exact subscription method (whole-tree subscribe vs per-cell sweep) — unresolved; the served data is identical to the broker's either way.
+- Full REST surface: ~105 endpoints per open-tj (routes, stops, schedules, fares, facilities, Tara chatbot). Worth enumerating if a project needs them.
+- bustrack.transjakarta.co.id auth model (Traccar) — session probe returned 404; likely internal, but could be explored further.
+- Whether anonymous MQTT access is intentional or an oversight — affects long-term viability.
+
+## Risks
+
+- Access policy can change without notice; all third-party trackers depend on the same two entry points.
+- Per-bus refresh observed ~8–75 s; apps showing 1–3 s updates re-batch/interpolate the same data — plan for the real cadence.
+- No API versioning or deprecation contract exists for third-party consumers.
+
+## Ideas for future use
+
+- Personal realtime dashboard / alerts ("bus X is N stops from my stop").
+- Fleet analytics from archived captures: headway estimation, bus-bunching detection, corridor coverage over time.
+- Continuous capture into a time-series store → historic delay/performance analysis.
+- Combine feed positions with GTFS schedules for ETA modeling.
+- Cross-validate against the bustrack/Traccar system if access ever opens.

transjakarta-live-data/recipes.md

@@ -0,0 +1,71 @@
+# Recipes — using TransJakarta live data
+
+Prerequisites: `curl` + `jq` (REST), `node` + `npm install mqtt` (live stream). Any MQTT 3.1.1-over-WSS client works (e.g. MQTTX, paho with websockets transport).
+
+## 1. Live stream (MQTT, anonymous)
+
+```js
+// save as live.mjs; run: npm i mqtt && node live.mjs
+import mqtt from 'mqtt';
+
+const client = mqtt.connect('wss://mqtt.tj.co.id:8084/mqtt', {
+  protocolVersion: 4, clean: true, connectTimeout: 15000
+});
+
+// whole fleet: '/mobile_armada/#' — or narrow to map cells, e.g. '/mobile_armada/0018-0022/#'
+client.on('connect', () => client.subscribe('/mobile_armada/#', { qos: 0 }));
+client.on('message', (topic, payload) => {
+  const b = JSON.parse(payload.toString());
+  console.log(b.bus_body_no, b.route_code, b.trip_id, b.latitude, b.longitude, b.bearing);
+});
+```
+
+Notes:
+- Expect a retained burst (~2,000 messages) right after subscribing, then live updates.
+- `clean: true` avoids stale queues; add simple reconnect handling for long runs.
+- Subscribe to specific cells instead of `#` if you only care about one area.
+
+Ready-made snapshot tool: [[transjakarta-live-data/scripts/tj-snapshot.mjs|scripts/tj-snapshot.mjs]]
+`node tj-snapshot.mjs --seconds 60 --out snapshot.json`
+
+## 2. REST — guest token + nearby buses
+
+```bash
+# 1) guest token (no account)
+TOKEN=$(curl -s -X POST 'https://tijeapi.transjakarta.co.id/v1/auth/login/guest' \
+  -H 'Content-Type: application/json' -H 'Accept: application/json' \
+  -H 'User-Agent: okhttp/4.12.0' -H 'X-App-OS: android' -H 'X-App-Version: 2.10.2' \
+  -H 'X-Device-ID: tj-research-01' \
+  -d '{"device_id":"tj-research-01"}' | jq -r .token)
+
+# 2) nearby buses (Monas-area example)
+curl -s 'https://tijeapi.transjakarta.co.id/v1/bus?latitude=-6.2088&longitude=106.8456&radius=1' \
+  -H "Authorization: Bearer $TOKEN" -H 'Accept: application/json' \
+  -H 'User-Agent: okhttp/4.12.0' -H 'X-App-OS: android' -H 'X-App-Version: 2.10.2' \
+  -H 'X-Device-ID: tj-research-01' | jq '.data[0]'
+
+# 3) discovery: which MQTT topics cover a location?
+curl -s -X POST 'https://tijeapi.transjakarta.co.id/v1/bus' \
+  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H 'Accept: application/json' \
+  -H 'User-Agent: okhttp/4.12.0' -H 'X-App-OS: android' -H 'X-App-Version: 2.10.2' \
+  -H 'X-Device-ID: tj-research-01' \
+  -d '{"latitude":-6.2088,"longitude":106.8456,"radius":1,"event":"change_loc"}' | jq '.data.topics'
+```
+
+- Any stable `X-Device-ID` string appears accepted; tokens last ~24 h.
+- `radius` units unverified (try 1–50).
+
+## 3. Static schedule (GTFS)
+
+```bash
+curl -LO https://gtfs.transjakarta.co.id/files/file_gtfs.zip
+unzip -d gtfs file_gtfs.zip   # stops.txt, routes.txt, trips.txt, shapes.txt, stop_times.txt, ...
+```
+
+Join strategy: bus positions arrive with `trip_id` / `route_code`; use GTFS for stop names, shapes, and schedules. Feed stop IDs (`B00436P`-style) match GTFS `stop_id`.
+
+## Cautions
+
+- Undocumented infrastructure: endpoints can change or close without notice. Access is currently open and shared with the official app path — be gentle (one connection goes a long way; avoid aggressive polling).
+- Do not open many parallel broker connections or reconnect in tight loops.
+- Guest tokens expire; log in again rather than hammering the refresh endpoint.

transjakarta-live-data/scripts/tj-snapshot.mjs

@@ -0,0 +1,63 @@
+#!/usr/bin/env node
+// tj-snapshot.mjs — capture a live snapshot of the TransJakarta bus feed (MQTT over WSS).
+//
+// The broker (wss://mqtt.tj.co.id:8084/mqtt) is anonymous. This tool connects,
+// subscribes, and records the last known state of every bus it sees.
+//
+// Setup:  npm install mqtt        (in this directory, or a parent dir)
+// Usage:  node tj-snapshot.mjs [--topic /mobile_armada/#] [--seconds 60] [--out snapshot.json]
+//
+// Output: summary on stdout; with --out, a JSON file:
+//   { capturedAt, durationSec, topic, messages, busCount, buses: { <bus_body_no>: {...} } }
+//
+// Notes: undocumented infrastructure — be gentle; one connection is enough.
+//        Whole fleet: '/mobile_armada/#'. One area: '/mobile_armada/0018-0022/#'.
+
+import mqtt from 'mqtt';
+import fs from 'fs';
+
+const args = process.argv.slice(2);
+const getArg = (name, fallback) => {
+  const i = args.indexOf(name);
+  return i >= 0 && args[i + 1] ? args[i + 1] : fallback;
+};
+
+const topic = getArg('--topic', '/mobile_armada/#');
+const seconds = parseInt(getArg('--seconds', '60'), 10) || 60;
+const outFile = getArg('--out', '');
+
+const out = { capturedAt: new Date().toISOString(), durationSec: seconds, topic, messages: 0, buses: {} };
+
+const client = mqtt.connect('wss://mqtt.tj.co.id:8084/mqtt', {
+  protocolVersion: 4, keepalive: 30, clean: true, connectTimeout: 15000, reconnectPeriod: 0
+});
+
+client.on('connect', () => client.subscribe(topic, { qos: 0 }));
+client.on('error', (e) => console.error('MQTT error:', e.message));
+
+client.on('message', (t, payload) => {
+  out.messages++;
+  try {
+    const d = JSON.parse(payload.toString());
+    if (!d.bus_body_no) return;
+    out.buses[d.bus_body_no] = {
+      lat: d.latitude, lng: d.longitude, bearing: d.bearing,
+      route: d.route_code, trip: d.trip_id, headsign: d.trip_headsign,
+      type: d.type, direction: d.direction,
+      currStop: d.curr_stops, nextStop: d.next_stops,
+      etaNextStopMin: d.estimated_time_next_stops,
+      lastUpdate: d.timestamp, mqttTopic: t
+    };
+  } catch { /* skip malformed */ }
+});
+
+setTimeout(() => {
+  out.busCount = Object.keys(out.buses).length;
+  console.log(`Captured ${out.messages} messages, ${out.busCount} unique buses in ${seconds}s`);
+  if (outFile) {
+    fs.writeFileSync(outFile, JSON.stringify(out));
+    console.log(`Snapshot written to ${outFile}`);
+  }
+  client.end(true);
+  process.exit(0);
+}, seconds * 1000);

transjakarta-live-data/sources.md

@@ -0,0 +1,30 @@
+# Sources
+
+★ = directly verified during this investigation (Oct 11–12, 2026). Unmarked items are community-documented leads, not yet re-verified.
+
+## TransJakarta services
+
+- ★ `https://tijeapi.transjakarta.co.id` — official app ("TJ:Transjakarta" / Tije) REST backend. Guest login works: `POST /v1/auth/login/guest`. (~105 endpoints total per open-tj; only the bus endpoints verified here.)
+- ★ `wss://mqtt.tj.co.id:8084/mqtt` — live bus feed; MQTT 3.1.1 over WebSocket Secure, anonymous access.
+- ★ `https://gtfs.transjakarta.co.id/files/file_gtfs.zip` — official GTFS zip (~2.4 MB). Portal UI is login-gated; the file URL itself is open. Version observed: 27 Jul 2026.
+- ★ `https://bustrack.transjakarta.co.id` — TJ's own tracker web app, built on Traccar (stock UI + API surface). `/api/session` returned 404; access looks restricted. Separate system from the Tije feed.
+- `https://chatbot.transjakarta.co.id/ws/` — "Tara" AI chatbot WebSocket (JWT) — documented by open-tj.
+- `https://tije-dashboard.transjakarta.co.id` — login page for the app's admin dashboard.
+
+## Third-party projects (how the stack became known)
+
+- `https://github.com/mugnimaestra/tj-live` — SvelteKit tracker. Source documents broker URL, guest JWT flow, `POST /v1/bus` topics, Bus schema.
+- `https://github.com/BroNils/open-tj` — Go SDK/CLI. Documents "MQTT anonymous", "~105 REST endpoints", Tara chatbot, Kong gateway.
+
+## Investigation target (origin of this research)
+
+- `https://transit.oyenza.com` — "Bus Jakarta Live", independent third-party tracker ("Tidak berafiliasi" — not affiliated). Serves TransJakarta's feed through its own backend:
+  - browser stream `wss://transit.oyenza.com/api/live` (custom protocol)
+  - REST: `/api/bus/<fleet-no>`, `/api/halte/<stop-id>`, `/api/stats`, `/api/meta`, `/api/places`
+  - `static.json` cites the official GTFS URL as its static source
+- Related domains: `transit.vantis.sh` (alias), `vantis.sh`, `oyenza.com` — same developer.
+
+## Context
+
+- App launch coverage: "Transjakarta launches new app with accurate bus tracking", The Jakarta Post, 2024-09-24 (headline as seen in search results).
+- Official app listings: Play Store `com.transjakmobile`, App Store "TJ:Transjakarta".