From 3a518be39fa766c109e98538d821e54dbbf204f1 Mon Sep 17 00:00:00 2001 From: hermes Date: Mon, 12 Oct 2026 09:34:23 +0000 Subject: [PATCH] Add transjakarta-live-data research topic Verified TransJakarta realtime stack: anonymous MQTT bus feed (wss://mqtt.tj.co.id:8084/mqtt), guest REST API (tijeapi.transjakarta.co.id), official GTFS. Cross-capture verification vs third-party tracker, connection recipes, snapshot tool. --- transjakarta-live-data/analysis.md | 49 +++++++++++++ transjakarta-live-data/index.md | 43 +++++++++++ transjakarta-live-data/methodology.md | 27 +++++++ transjakarta-live-data/next-steps.md | 23 ++++++ transjakarta-live-data/recipes.md | 71 +++++++++++++++++++ .../scripts/tj-snapshot.mjs | 63 ++++++++++++++++ transjakarta-live-data/sources.md | 30 ++++++++ 7 files changed, 306 insertions(+) create mode 100644 transjakarta-live-data/analysis.md create mode 100644 transjakarta-live-data/index.md create mode 100644 transjakarta-live-data/methodology.md create mode 100644 transjakarta-live-data/next-steps.md create mode 100644 transjakarta-live-data/recipes.md create mode 100644 transjakarta-live-data/scripts/tj-snapshot.mjs create mode 100644 transjakarta-live-data/sources.md diff --git a/transjakarta-live-data/analysis.md b/transjakarta-live-data/analysis.md new file mode 100644 index 0000000..cc1ccfa --- /dev/null +++ b/transjakarta-live-data/analysis.md @@ -0,0 +1,49 @@ +# Analysis — architecture and verification evidence + +## How the stack works + +1. Buses carry GPS devices; vehicles are identified by fleet body numbers (`MYS-…`, `TJ-…`, `KWK-…`, `DMR-…`, `PRM-…`; the prefix appears to be operator/depot). +2. TransJakarta's backend ingests positions and publishes each bus to its MQTT broker; REST state backs the official app. +3. Client flow: login (guest works) → `POST /v1/bus` with your location → you receive nearby buses **plus MQTT topic wildcards for the map cells around you** → subscribe those on the broker → live positions stream in. REST covers routes/schedules; GTFS covers static geometry. +4. The upstream path from vehicle hardware into the broker is inferred — it is not observable from outside. + +## The MQTT feed + +- Broker: `wss://mqtt.tj.co.id:8084/mqtt` — MQTT 3.1.1 over WebSocket Secure, **anonymous** (no username/password; verified with independent clients). +- Topic schema: `/mobile_armada////` + - Real captured example: `/mobile_armada/0021-0014/JAK.49/JAK.49-L03/KWK-1518` + - `` = `{lat-index}-{lng-index}`, a ~0.01° (~1.1 km) map cell. Verified by probing the discovery API with offset coordinates: ±0.1° of latitude/longitude shifts the matching index by ±10 (index grows southward and eastward). +- Retained messages: subscribing delivers ~2,000+ last-known states within seconds, then live updates continue. Per-bus refresh observed at ~8–75 s (some active vehicles faster; many ~25–37 s); delivery is bursty. +- Fleet size observed live: 3,300–3,722 buses. + +### Payload schema (complete key list, captured verbatim) + +`latitude`, `longitude`, `snap_latitude`, `snap_longitude`, `bearing`, `estimated_time_next_stops`, `estimated_distance_next_stops`, `curr_stops`, `next_stops`, `prev_stops`, `passengers_status`, `route_code`, `route_long_name`, `route_color`, `route_text_color`, `trip_short_name`, `trip_headsign`, `bus_type`, `type`, `trip_id`, `bus_body_no`, `stops[]`, `timestamp`, `last_update`, `eta_by_parent_stop_id`, `next_parent_gate`, `next_parent_gate_eta`, `next_gate`, `next_gate_eta`, `direction`, `eta`. + +- `stops[]` items: `name`, `stop_id`, `latitude`, `longitude`, `stop_sequence`, `time_from_prev`, `time_wait`, `headsign`, `parent_stop_id`, `parent_stop_name`, `eta`. +- `timestamp` is epoch milliseconds (e.g. `1791797549000`). Stop IDs (`B00436P`-style) match GTFS. +- Some fields are null for some vehicles (e.g. gate fields, `passengers_status`, `last_update`). + +## The REST API + +- `POST /v1/auth/login/guest` → `token` + `refresh_token` + `user{email: @guest.tije.com, is_guest: true}`. Token lifetime ~24 h. Headers used: `User-Agent: okhttp/4.12.0`, `X-App-OS: android`, `X-App-Version: 2.10.2`, `X-Device-ID: `. +- `POST /v1/bus` body `{latitude, longitude, radius, event: "change_loc"}` → `data { buses[], polyline, session_id, stops[], topics[] }`. `topics` = the MQTT wildcards for nearby cells (e.g. `/mobile_armada/0018-0022/#` …). This is the app's discovery mechanism. +- `GET /v1/bus?latitude=&longitude=&radius=` → array of nearby bus records (~15 at radius=1 near Monas; radius units unverified). + +## Verification case: oyenza serves this same data + +Simultaneous captures (node WS client on oyenza's stream + node MQTT client on TJ's broker): + +- 1,794 buses common to both (of 1,823 seen on the broker in the same window). +- 100% agreement on `trip_id` and `route_code`; 99.7% on headsign. +- Position deltas: median 8.9 m, p90 42 m, 1,783/1,794 ≤ 100 m — consistent with reading the same moving buses a few seconds apart; no other explanation fits. +- Fleet overlap 97.8% over a 75 s window (2,869/2,933); remainder = capture-window skew. +- Category taxonomy maps 1:1, including the edge case: the app's whole "other" bucket (2 buses) = the only two untyped buses in the feed. + +oyenza's own layer (what it adds): fleet-state backend; browser protocol `hello/feed/snap/delta` with positions packed as integers ×10⁵; deltas every 3.00 s (~1,000 position records each); full snapshot 3,722 buses; its own REST extras; a replay mode. Static layer from the official GTFS (240 routes, 698 shapes, 7,493 stops at capture). Its About panel states: "Posisi bus — feed langsung TransJakarta" and "Tidak berafiliasi". + +## Confidence + +- **Established (verified by direct capture):** guest token; both bus endpoints; anonymous MQTT access; topic schema + grid decoding; payload schema; retained behavior; GTFS file; Traccar-based bustrack; oyenza data identity. +- **Community-documented (not independently re-verified):** ~105 REST endpoint count; Tara chatbot WS; Kong gateway; refresh-token flow. +- **Inferred:** vehicle→broker ingest path; oyenza's exact subscription method (whole-tree vs per-cell — data identity holds either way); `radius` units. diff --git a/transjakarta-live-data/index.md b/transjakarta-live-data/index.md new file mode 100644 index 0000000..c497356 --- /dev/null +++ b/transjakarta-live-data/index.md @@ -0,0 +1,43 @@ +--- +title: "TransJakarta live data — verified stack: MQTT stream, app REST API, GTFS" +description: "How TransJakarta realtime bus data works and how to use it — anonymous MQTT broker, guest REST API, official GTFS. Verified by cross-capture, Oct 2026." +status: concluded +category: technical +tags: [transjakarta, mqtt, realtime, gtfs, reverse-engineering, transit-data] +draft: true +created: 2026-10-12 +updated: 2026-10-12 +--- + +# TransJakarta live data — verified stack + +**What this is:** a verified map of TransJakarta's public realtime data services — the MQTT live-bus stream, the official app REST API, and the GTFS static feed — plus ready-to-use connection recipes. + +**Origin:** reverse-engineering `transit.oyenza.com` (a third-party tracker) to determine what it uses. The trail led to TransJakarta's own services, which are reachable directly. Cross-capture verification proved the third-party tracker serves the same data as the official feed — see [[transjakarta-live-data/analysis|Analysis]]. + +## TL;DR — the three services + +1. **Live positions (MQTT):** `wss://mqtt.tj.co.id:8084/mqtt` — anonymous, no key needed. Topics `/mobile_armada////`; one JSON payload per bus. +2. **App REST API:** `https://tijeapi.transjakarta.co.id` — guest JWT via `POST /v1/auth/login/guest` (no account). `POST /v1/bus` returns nearby buses **and** the MQTT topics to subscribe. +3. **Static schedule:** `https://gtfs.transjakarta.co.id/files/file_gtfs.zip` (~2.4 MB, open download; version observed 27 Jul 2026). + +## Key verified numbers (Oct 12, 2026) + +- Live fleet: ~3,300–3,720 buses (BRT, Mikrotrans, Angkutan Umum Integrasi, Transjabodetabek, Royaltrans, Rusun). +- oyenza cross-capture: 1,794 common buses, 100% trip/route ID agreement, median position delta 8.9 m, 97.8% fleet overlap over 75 s. +- Broker: ~2,000+ retained messages delivered instantly on subscribe; per-bus refresh ~8–75 s in samples. +- Third-party tracker's own browser stream: full snapshot (3,722 buses) + deltas every 3.00 s. + +## Using it + +- Copy-paste recipes (curl / node): [[transjakarta-live-data/recipes|Recipes]] +- Ready tool: [[transjakarta-live-data/scripts/tj-snapshot.mjs|scripts/tj-snapshot.mjs]] — capture a live snapshot to JSON. +- Evidence archive: `workspace/transjakarta-live-captures/` (raw captures, scripts, bundles). + +## Files + +- [[transjakarta-live-data/sources|Sources]] — endpoints, repos, references +- [[transjakarta-live-data/analysis|Analysis]] — architecture + verification evidence +- [[transjakarta-live-data/methodology|Methodology]] — how it was verified (repeatable) +- [[transjakarta-live-data/recipes|Recipes]] — connection examples +- [[transjakarta-live-data/next-steps|Next steps]] — risks + future ideas diff --git a/transjakarta-live-data/methodology.md b/transjakarta-live-data/methodology.md new file mode 100644 index 0000000..d091b49 --- /dev/null +++ b/transjakarta-live-data/methodology.md @@ -0,0 +1,27 @@ +# Methodology — how this was verified + +Repeatable procedure (executed Oct 11–12, 2026). Tools: `curl`, `node` (packages `ws`, `mqtt`), `jq`. + +## Steps + +1. **Map the target:** fetch the site's HTML + JS bundles with curl; read the CSP header (`connect-src` names allowed WebSocket endpoints); grep bundles for protocol strings → found oyenza's own WS (`/api/live`) and its message types (`hello/feed/snap/delta`). +2. **Find static-data citations:** `static.json` referenced the official GTFS zip URL → static layer source identified. +3. **Find the upstream:** search GitHub for projects in the same ecosystem (`tj-live`, `open-tj`) → they document the REST + MQTT + GTFS stack. +4. **Simultaneous cross-capture:** + - `oyenza-ws.mjs` — WS client on `wss://transit.oyenza.com/api/live` + - `mqtt-cap.mjs` / `mqtt-long.mjs` — MQTT client on `wss://mqtt.tj.co.id:8084/mqtt`, subscribing `/mobile_armada/#` + - Analysis with `compare.mjs` / `compare2.mjs` (ID sets, field agreement, position deltas), `seg1.mjs` (topic segments), `mqtt-rate.mjs` (rate profile). +5. **Decode the topic grid:** four `POST /v1/bus` probes at ±0.1° offsets → index-shift mapping (≈0.01°/unit). +6. **Cadence check:** re-connect to oyenza's WS and record delta intervals (`oy2.mjs`) → exactly ~3.00 s. + +## Key results + +- 1,794 common buses; 100% trip/route ID match; median position delta 8.9 m; p90 42 m; 97.8% fleet overlap (75 s window). +- Broker: ~2,000+ retained messages instantly on subscribe; per-bus refresh ~8–75 s in samples. +- oyenza deltas: 3000–3013 ms gaps, ~1,000 position records each. + +## Artifacts + +- All scripts + raw captures archived at `workspace/transjakarta-live-captures/` (4.6 MB): oyenza page + bundles, oyenza WS captures, MQTT captures, comparison outputs, rate profiles, tije REST responses (guest token redacted). +- Scratch copies live under `~/.hermes/profiles/research/cache/scratch/oyenza/` (auto-pruned after ~24 h idle — use the archive instead). +- Reusable snapshot tool: [[transjakarta-live-data/scripts/tj-snapshot.mjs|scripts/tj-snapshot.mjs]] diff --git a/transjakarta-live-data/next-steps.md b/transjakarta-live-data/next-steps.md new file mode 100644 index 0000000..e99fe36 --- /dev/null +++ b/transjakarta-live-data/next-steps.md @@ -0,0 +1,23 @@ +# Next steps / open questions + +## Open questions + +- Exact ingest path from vehicle GPS hardware into the broker — not observable from outside. +- oyenza's exact subscription method (whole-tree subscribe vs per-cell sweep) — unresolved; the served data is identical to the broker's either way. +- Full REST surface: ~105 endpoints per open-tj (routes, stops, schedules, fares, facilities, Tara chatbot). Worth enumerating if a project needs them. +- bustrack.transjakarta.co.id auth model (Traccar) — session probe returned 404; likely internal, but could be explored further. +- Whether anonymous MQTT access is intentional or an oversight — affects long-term viability. + +## Risks + +- Access policy can change without notice; all third-party trackers depend on the same two entry points. +- Per-bus refresh observed ~8–75 s; apps showing 1–3 s updates re-batch/interpolate the same data — plan for the real cadence. +- No API versioning or deprecation contract exists for third-party consumers. + +## Ideas for future use + +- Personal realtime dashboard / alerts ("bus X is N stops from my stop"). +- Fleet analytics from archived captures: headway estimation, bus-bunching detection, corridor coverage over time. +- Continuous capture into a time-series store → historic delay/performance analysis. +- Combine feed positions with GTFS schedules for ETA modeling. +- Cross-validate against the bustrack/Traccar system if access ever opens. diff --git a/transjakarta-live-data/recipes.md b/transjakarta-live-data/recipes.md new file mode 100644 index 0000000..d204c19 --- /dev/null +++ b/transjakarta-live-data/recipes.md @@ -0,0 +1,71 @@ +# Recipes — using TransJakarta live data + +Prerequisites: `curl` + `jq` (REST), `node` + `npm install mqtt` (live stream). Any MQTT 3.1.1-over-WSS client works (e.g. MQTTX, paho with websockets transport). + +## 1. Live stream (MQTT, anonymous) + +```js +// save as live.mjs; run: npm i mqtt && node live.mjs +import mqtt from 'mqtt'; + +const client = mqtt.connect('wss://mqtt.tj.co.id:8084/mqtt', { + protocolVersion: 4, clean: true, connectTimeout: 15000 +}); + +// whole fleet: '/mobile_armada/#' — or narrow to map cells, e.g. '/mobile_armada/0018-0022/#' +client.on('connect', () => client.subscribe('/mobile_armada/#', { qos: 0 })); +client.on('message', (topic, payload) => { + const b = JSON.parse(payload.toString()); + console.log(b.bus_body_no, b.route_code, b.trip_id, b.latitude, b.longitude, b.bearing); +}); +``` + +Notes: +- Expect a retained burst (~2,000 messages) right after subscribing, then live updates. +- `clean: true` avoids stale queues; add simple reconnect handling for long runs. +- Subscribe to specific cells instead of `#` if you only care about one area. + +Ready-made snapshot tool: [[transjakarta-live-data/scripts/tj-snapshot.mjs|scripts/tj-snapshot.mjs]] +`node tj-snapshot.mjs --seconds 60 --out snapshot.json` + +## 2. REST — guest token + nearby buses + +```bash +# 1) guest token (no account) +TOKEN=$(curl -s -X POST 'https://tijeapi.transjakarta.co.id/v1/auth/login/guest' \ + -H 'Content-Type: application/json' -H 'Accept: application/json' \ + -H 'User-Agent: okhttp/4.12.0' -H 'X-App-OS: android' -H 'X-App-Version: 2.10.2' \ + -H 'X-Device-ID: tj-research-01' \ + -d '{"device_id":"tj-research-01"}' | jq -r .token) + +# 2) nearby buses (Monas-area example) +curl -s 'https://tijeapi.transjakarta.co.id/v1/bus?latitude=-6.2088&longitude=106.8456&radius=1' \ + -H "Authorization: Bearer $TOKEN" -H 'Accept: application/json' \ + -H 'User-Agent: okhttp/4.12.0' -H 'X-App-OS: android' -H 'X-App-Version: 2.10.2' \ + -H 'X-Device-ID: tj-research-01' | jq '.data[0]' + +# 3) discovery: which MQTT topics cover a location? +curl -s -X POST 'https://tijeapi.transjakarta.co.id/v1/bus' \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H 'Accept: application/json' \ + -H 'User-Agent: okhttp/4.12.0' -H 'X-App-OS: android' -H 'X-App-Version: 2.10.2' \ + -H 'X-Device-ID: tj-research-01' \ + -d '{"latitude":-6.2088,"longitude":106.8456,"radius":1,"event":"change_loc"}' | jq '.data.topics' +``` + +- Any stable `X-Device-ID` string appears accepted; tokens last ~24 h. +- `radius` units unverified (try 1–50). + +## 3. Static schedule (GTFS) + +```bash +curl -LO https://gtfs.transjakarta.co.id/files/file_gtfs.zip +unzip -d gtfs file_gtfs.zip # stops.txt, routes.txt, trips.txt, shapes.txt, stop_times.txt, ... +``` + +Join strategy: bus positions arrive with `trip_id` / `route_code`; use GTFS for stop names, shapes, and schedules. Feed stop IDs (`B00436P`-style) match GTFS `stop_id`. + +## Cautions + +- Undocumented infrastructure: endpoints can change or close without notice. Access is currently open and shared with the official app path — be gentle (one connection goes a long way; avoid aggressive polling). +- Do not open many parallel broker connections or reconnect in tight loops. +- Guest tokens expire; log in again rather than hammering the refresh endpoint. diff --git a/transjakarta-live-data/scripts/tj-snapshot.mjs b/transjakarta-live-data/scripts/tj-snapshot.mjs new file mode 100644 index 0000000..cb52812 --- /dev/null +++ b/transjakarta-live-data/scripts/tj-snapshot.mjs @@ -0,0 +1,63 @@ +#!/usr/bin/env node +// tj-snapshot.mjs — capture a live snapshot of the TransJakarta bus feed (MQTT over WSS). +// +// The broker (wss://mqtt.tj.co.id:8084/mqtt) is anonymous. This tool connects, +// subscribes, and records the last known state of every bus it sees. +// +// Setup: npm install mqtt (in this directory, or a parent dir) +// Usage: node tj-snapshot.mjs [--topic /mobile_armada/#] [--seconds 60] [--out snapshot.json] +// +// Output: summary on stdout; with --out, a JSON file: +// { capturedAt, durationSec, topic, messages, busCount, buses: { : {...} } } +// +// Notes: undocumented infrastructure — be gentle; one connection is enough. +// Whole fleet: '/mobile_armada/#'. One area: '/mobile_armada/0018-0022/#'. + +import mqtt from 'mqtt'; +import fs from 'fs'; + +const args = process.argv.slice(2); +const getArg = (name, fallback) => { + const i = args.indexOf(name); + return i >= 0 && args[i + 1] ? args[i + 1] : fallback; +}; + +const topic = getArg('--topic', '/mobile_armada/#'); +const seconds = parseInt(getArg('--seconds', '60'), 10) || 60; +const outFile = getArg('--out', ''); + +const out = { capturedAt: new Date().toISOString(), durationSec: seconds, topic, messages: 0, buses: {} }; + +const client = mqtt.connect('wss://mqtt.tj.co.id:8084/mqtt', { + protocolVersion: 4, keepalive: 30, clean: true, connectTimeout: 15000, reconnectPeriod: 0 +}); + +client.on('connect', () => client.subscribe(topic, { qos: 0 })); +client.on('error', (e) => console.error('MQTT error:', e.message)); + +client.on('message', (t, payload) => { + out.messages++; + try { + const d = JSON.parse(payload.toString()); + if (!d.bus_body_no) return; + out.buses[d.bus_body_no] = { + lat: d.latitude, lng: d.longitude, bearing: d.bearing, + route: d.route_code, trip: d.trip_id, headsign: d.trip_headsign, + type: d.type, direction: d.direction, + currStop: d.curr_stops, nextStop: d.next_stops, + etaNextStopMin: d.estimated_time_next_stops, + lastUpdate: d.timestamp, mqttTopic: t + }; + } catch { /* skip malformed */ } +}); + +setTimeout(() => { + out.busCount = Object.keys(out.buses).length; + console.log(`Captured ${out.messages} messages, ${out.busCount} unique buses in ${seconds}s`); + if (outFile) { + fs.writeFileSync(outFile, JSON.stringify(out)); + console.log(`Snapshot written to ${outFile}`); + } + client.end(true); + process.exit(0); +}, seconds * 1000); diff --git a/transjakarta-live-data/sources.md b/transjakarta-live-data/sources.md new file mode 100644 index 0000000..0b3a5b6 --- /dev/null +++ b/transjakarta-live-data/sources.md @@ -0,0 +1,30 @@ +# Sources + +★ = directly verified during this investigation (Oct 11–12, 2026). Unmarked items are community-documented leads, not yet re-verified. + +## TransJakarta services + +- ★ `https://tijeapi.transjakarta.co.id` — official app ("TJ:Transjakarta" / Tije) REST backend. Guest login works: `POST /v1/auth/login/guest`. (~105 endpoints total per open-tj; only the bus endpoints verified here.) +- ★ `wss://mqtt.tj.co.id:8084/mqtt` — live bus feed; MQTT 3.1.1 over WebSocket Secure, anonymous access. +- ★ `https://gtfs.transjakarta.co.id/files/file_gtfs.zip` — official GTFS zip (~2.4 MB). Portal UI is login-gated; the file URL itself is open. Version observed: 27 Jul 2026. +- ★ `https://bustrack.transjakarta.co.id` — TJ's own tracker web app, built on Traccar (stock UI + API surface). `/api/session` returned 404; access looks restricted. Separate system from the Tije feed. +- `https://chatbot.transjakarta.co.id/ws/` — "Tara" AI chatbot WebSocket (JWT) — documented by open-tj. +- `https://tije-dashboard.transjakarta.co.id` — login page for the app's admin dashboard. + +## Third-party projects (how the stack became known) + +- `https://github.com/mugnimaestra/tj-live` — SvelteKit tracker. Source documents broker URL, guest JWT flow, `POST /v1/bus` topics, Bus schema. +- `https://github.com/BroNils/open-tj` — Go SDK/CLI. Documents "MQTT anonymous", "~105 REST endpoints", Tara chatbot, Kong gateway. + +## Investigation target (origin of this research) + +- `https://transit.oyenza.com` — "Bus Jakarta Live", independent third-party tracker ("Tidak berafiliasi" — not affiliated). Serves TransJakarta's feed through its own backend: + - browser stream `wss://transit.oyenza.com/api/live` (custom protocol) + - REST: `/api/bus/`, `/api/halte/`, `/api/stats`, `/api/meta`, `/api/places` + - `static.json` cites the official GTFS URL as its static source +- Related domains: `transit.vantis.sh` (alias), `vantis.sh`, `oyenza.com` — same developer. + +## Context + +- App launch coverage: "Transjakarta launches new app with accurate bus tracking", The Jakarta Post, 2024-09-24 (headline as seen in search results). +- Official app listings: Play Store `com.transjakmobile`, App Store "TJ:Transjakarta".