From 77b0a55c4104d90de105591ae30a19f33ffcf16b Mon Sep 17 00:00:00 2001 From: hermes Date: Mon, 12 Oct 2026 12:17:01 +0000 Subject: [PATCH] Add comuline-krl-data topic + jakarta-public-transport-data umbrella Second audit under the Jakarta public transport umbrella: comuline.com replicates KAI Commuter's official partner API (api-partner.krl.co.id); its data has been frozen since ~2025-03-30. Adds the umbrella hub, the comuline topic (sources/analysis/recipes/next-steps), and links the existing transjakarta-live-data topic under the umbrella. --- comuline-krl-data/analysis.md | 36 ++++++++++++++++++++ comuline-krl-data/index.md | 42 +++++++++++++++++++++++ comuline-krl-data/next-steps.md | 20 +++++++++++ comuline-krl-data/recipes.md | 40 ++++++++++++++++++++++ comuline-krl-data/sources.md | 24 ++++++++++++++ jakarta-public-transport-data/index.md | 46 ++++++++++++++++++++++++++ transjakarta-live-data/index.md | 7 +++- 7 files changed, 214 insertions(+), 1 deletion(-) create mode 100644 comuline-krl-data/analysis.md create mode 100644 comuline-krl-data/index.md create mode 100644 comuline-krl-data/next-steps.md create mode 100644 comuline-krl-data/recipes.md create mode 100644 comuline-krl-data/sources.md create mode 100644 jakarta-public-transport-data/index.md diff --git a/comuline-krl-data/analysis.md b/comuline-krl-data/analysis.md new file mode 100644 index 0000000..28beb87 --- /dev/null +++ b/comuline-krl-data/analysis.md @@ -0,0 +1,36 @@ +# Analysis — how comuline works + +## Pipeline + +1. **Sync (daily cron, as designed):** `comuline/api/src/sync/station.ts` fetches `{KRL_ENDPOINT_BASE_URL}/krl-station`; `schedule.ts` then iterates all stations in batches of 5 (5 s sleep between) — CORS `OPTIONS` preflight, then `GET /schedule?stationid={id}&timefrom=00:00&timeto=23:00`. Requests carry browser-like headers plus a hardcoded bearer token, because the upstream is an authenticated partner API. +2. **Transform:** stations → `st_krl_` records (upstream `WIL` entries filtered; airport station BST manually added); schedules → `sc_krl__` rows with origin/destination station IDs resolved by name, `departs_at`/`arrives_at` timestamps, and line color; upserted into Postgres. +3. **Serve:** Hono API on Cloudflare Workers (Neon Postgres + Upstash Redis per-station read cache) exposes clean JSON with an OpenAPI spec; web/native apps consume it. + +`KRL_ENDPOINT_BASE_URL` = `https://api-partner.krl.co.id/krl-webs/v1` (from the repo's `.dev.example.vars`). + +## Data model (public API) + +- **Station:** `{ uid, id (e.g. MRI), name, type (KRL|LOCAL), metadata.origin.{daop,fg_enable}, active }` +- **Schedule row:** `{ id, station_id, station_origin_id, station_destination_id, train_id (e.g. 1448B), line ("COMMUTER LINE BOGOR"), route ("JAKARTAKOTA-DEPOK"), departs_at, arrives_at, metadata.origin.color }` +- **Route view** (`/v1/route/{train_id}`): ordered stop sequence (station name + departure time) for one train. + +## Observed state (2026-10-12) + +- API live; 113 stations served. Row counts sampled: MRI 1,045 · DP 730 · CLT 683 · JAKK 367 · BOO 341 · TNG 84 · BEK 0 (station present, no schedule rows). +- **Freshness:** the newest `updated_at` values are bulk 2024-11-13 and partial re-sync 2025-03-30; nothing newer. Repo activity stops the same month (`api` last push 2025-03-30). Conclusion: the daily cron is not currently running; the data is a frozen snapshot (~18 months old as of this audit). +- **No realtime tracking:** no position/GPS data in the bundle or API. The product is schedule lookup with locally saved stations. + +## Boundaries respected (deliberate limits) + +- Upstream `api-partner.krl.co.id` was probed once **without credentials** (returned 404) to characterize the auth wall. +- No calls were made using the token published in the public repo. The token is not reproduced in this vault. +- No attempts to bypass auth, rate limits, or access controls. + +## Confidence + +- **Established:** pipeline structure and architecture (public source + live API); coverage counts; staleness timestamps; absence of live tracking; upstream URL and auth wall. +- **Inferred:** why the sync stopped (project dormancy vs token expiry vs upstream change — not observable from outside); whether the upstream data model changed since. + +## Artifacts + +- Evidence archive: `workspace/comuline-captures/` — page HTML + headers, web bundle, live API samples (`stations.json`, `sched_MRI.json`, `openapi.json`), sync sources (bearer token REDACTED in the archive copy), `dev.example.vars`. diff --git a/comuline-krl-data/index.md b/comuline-krl-data/index.md new file mode 100644 index 0000000..fe29489 --- /dev/null +++ b/comuline-krl-data/index.md @@ -0,0 +1,42 @@ +--- +title: "Comuline — KRL Commuter Line Data (schedule API replication)" +description: "Reverse-engineering comuline.com: an open-source KRL schedule app whose public API replicates KAI Commuter's official partner API (api-partner.krl.co.id). Covers the pipeline, current staleness (data frozen ~Mar 2025), and usable endpoints." +status: concluded +category: technical +tags: [comuline, krl, commuter-line, kai-commuter, kci, schedule, api, jabodetabek, yogyakarta, jakarta-public-transport-data] +draft: true +created: 2026-10-12 +updated: 2026-10-12 +--- + +# Comuline — KRL Commuter Line Data + +## Summary + +[comuline.com](https://comuline.com) is an open-source KRL commuter-line schedule app (web + iOS + Android) by Abiel Zulio M (`github.com/comuline`). Its "own public API replication" is a **daily ETL copy of KAI Commuter's official partner API** (`api-partner.krl.co.id/krl-webs/v1` — an official, token-gated API on the krl.co.id domain), re-served publicly at `api.comuline.com`. + +**Status twist (verified 2026-10-12):** the sync last ran ~2025-03-30. The public API still serves that frozen snapshot, and the project has been dormant since roughly the same month. + +## Key Findings + +- **Upstream is official, token-gated:** the sync fetches `/krl-station` and `/schedule?stationid=…&timefrom=00:00&timeto=23:00` from `api-partner.krl.co.id` with a bearer token hardcoded in the public repo (value not reproduced here). Unauthenticated probe (no token) returned HTTP 404; no credentialed upstream calls were made during this audit. +- **Architecture:** daily cron → Cloudflare Workers (Hono + Bun) → Neon Postgres (Drizzle ORM) + Upstash Redis read cache → public REST API with OpenAPI spec → Vite/React web app (plus native apps) consuming it via a typed `openapi-fetch` client. +- **Coverage (from live API):** 113 stations — 95 daop-1 (Jabodetabek), 17 daop-6 (Yogyakarta area), 1 daop-2 (Purwakarta); 111 typed KRL + 2 LOCAL (Cikampek, Purwakarta). Sample row counts: MRI = 1,045 departures; DP = 730; CLT = 683; JAKK = 367; TNG = 84; BEK = 0. +- **No live tracking:** schedules and departures only — no GPS/position data anywhere (bundle + API verified). "Real-time" in their copy means "current schedule lookup", not live trains. +- **Data frozen:** schedule rows show last updates 2024-11-13 (bulk) + 2025-03-30 (partial passes); nothing newer. Repo activity stops 2025-03-30 (api) / 2025-04-03 (web). + +## Files + +- [[comuline-krl-data/sources|Sources]] — repos, endpoints, upstream, ecosystem +- [[comuline-krl-data/analysis|Analysis]] — pipeline, data model, observed state, boundaries +- [[comuline-krl-data/recipes|Recipes]] — using the public API (curl) +- [[comuline-krl-data/next-steps|Next steps]] — open questions, risks, ideas + +## Related Research + +- [[jakarta-public-transport-data/index|Jakarta Public Transport Data]] — umbrella topic +- [[transjakarta-live-data/index|TransJakarta Live Data]] — sibling audit (realtime bus feed) + +## Discussion + +Triggered 2026-10-12 from a Telegram request ("comuline.com which is including krl/jabotabek train data here — can we reverse engineer this as well?"). Documented as the second branch under the Jakarta public transport data umbrella. Method: artifact inspection (site bundle, response headers), public source reading (GitHub), live API sampling, and freshness checks via `updated_at` fields. diff --git a/comuline-krl-data/next-steps.md b/comuline-krl-data/next-steps.md new file mode 100644 index 0000000..9d94ea5 --- /dev/null +++ b/comuline-krl-data/next-steps.md @@ -0,0 +1,20 @@ +# Next steps / open questions + +## Open questions + +- Why did comuline's sync stop (~2025-03-30)? Project dormancy, token expiry, or upstream change — not observable from outside. (Author is reachable via the links in [[comuline-krl-data/sources|Sources]].) +- Current shape of the upstream partner API's data model (`krl-station`, `schedule`) — unverified; no credentialed call was made. +- **Live KRL tracking sources:** community trackers claim ~30 s updates (trackmycommuter, krljakarta.com, botkita). Where do their positions come from? A dedicated audit would be its own subtopic (KRL realtime — likely a different upstream than schedules). +- MRT Jakarta / LRT Jabodebek data sources (`jakartamrt.co.id` API was referenced by one community project) — candidates for umbrella subtopics. + +## Risks + +- **Comuline API:** frozen data; treat as reference only. Add freshness checks; do not build a "current timetable" product on it without another source. +- **Upstream partner API:** authenticated; using the repo-published token is not acceptable use. Legitimate access = partner agreement. +- Both are informal dependencies that can change or disappear without notice. + +## Ideas + +- Freshness monitor for the comuline API (alert when data updates — would signal the project waking up or a re-sync). +- Multi-modal view: join [[transjakarta-live-data/index|TransJakarta live feed]] + KRL schedules to prototype Jabodetabek trip planning. +- If current KRL timetables are needed long-term: evaluate official channels (KAI Commuter app/website) or pursue partner access properly. diff --git a/comuline-krl-data/recipes.md b/comuline-krl-data/recipes.md new file mode 100644 index 0000000..8adcab8 --- /dev/null +++ b/comuline-krl-data/recipes.md @@ -0,0 +1,40 @@ +# Recipes — Comuline public API + +Base: `https://api.comuline.com` — no auth, documented. Docs: `https://api.comuline.com/docs`; OpenAPI spec: `https://api.comuline.com/openapi`. + +**Caveat:** data is a frozen snapshot (last sync ~2025-03-30). Use it for station/route reference and historical schedule structure — not for current timetables. + +## Stations + +```bash +# all stations +curl -s https://api.comuline.com/v1/station | jq '.data[] | {id, name, daop: .metadata.origin.daop}' + +# one station (MRI = Manggarai) +curl -s https://api.comuline.com/v1/station/MRI | jq '.data' +``` + +## Departures at a station + +```bash +curl -s https://api.comuline.com/v1/schedule/MRI | jq '.data[0:5]' +# rows: { train_id, line, route, station_origin_id, station_destination_id, departs_at, arrives_at, metadata.origin.color } +``` + +## One train's full stop sequence + +```bash +curl -s https://api.comuline.com/v1/route/1448B | jq '.data.routes[] | {station_name, departs_at}' +``` + +## Status + +```bash +curl -s https://api.comuline.com/status +``` + +## If you need CURRENT timetables + +- The upstream partner API (`api-partner.krl.co.id`) is token-gated. Do **not** reuse the token published in a public repo — obtain legitimate partner access if you need it. +- Community alternatives (not audited here): `tarif-krl` (fares/schedules), `Fannandya/api-krl` (network-data SaaS), plus the live trackers listed in [[comuline-krl-data/sources|Sources]]. +- Always add a freshness check when consuming third-party transit APIs — compare `updated_at`/timestamps against today. diff --git a/comuline-krl-data/sources.md b/comuline-krl-data/sources.md new file mode 100644 index 0000000..1e0045e --- /dev/null +++ b/comuline-krl-data/sources.md @@ -0,0 +1,24 @@ +# Sources + +★ = directly verified during this audit (Oct 12, 2026). Unmarked items are community context, not re-verified. + +## Comuline (the audited project) + +- ★ `https://comuline.com` — web app ("Cari & Simpan Jadwal KRL 2024"). Vite SPA on Cloudflare; analytics via `analytics.zulio.me`. +- ★ `https://api.comuline.com` — public API (no auth). Docs at `/docs` (Scalar), OpenAPI spec at `/openapi`. Endpoints: `/v1/station`, `/v1/station/{id}`, `/v1/schedule/{station_id}`, `/v1/route/{train_id}`, `/status`. +- ★ Source repos: `github.com/comuline/web` (Vite/React; openapi-fetch + swr-openapi; i18n default `id`), `github.com/comuline/api` (Hono + Bun; Drizzle + Neon Postgres + Upstash Redis; sync scripts under `src/sync/`), plus `android` (Kotlin) and `ios` (Swift). +- ★ Author: Abiel Zulio M — `github.com/abielzulio`; prior project `abielzulio/jadwal-krl`. +- ★ Repo activity: `api` last push 2025-03-30 (116★), `web` 2025-04-03 (95★), `android` 2025-08-22, `ios` 2024-09-02. None archived; all dormant on the data side. + +## Upstream (official KAI Commuter) + +- ★ `https://api-partner.krl.co.id/krl-webs/v1` — official partner API of KRL (krl.co.id = KAI Commuter domain). Endpoints used by comuline's sync: `GET /krl-station`, `GET /schedule?stationid={id}&timefrom=00:00&timeto=23:00`. + - Auth: Bearer JWT. A long-lived token is hardcoded in the public repo (`comuline/api/src/sync/headers.ts`); its value is [REDACTED] here and was not used in this audit. Unauthenticated probe returned HTTP 404. + - Sync also sends browser-like headers and does a CORS `OPTIONS` preflight before each GET. +- Context: KRL (Commuter Line) is operated by PT Kereta Commuter Indonesia (KAI Commuter), subsidiary of PT KAI. "Daop" = KAI operational region (daop 1 = Jakarta, daop 2 = Bandung, daop 6 = Yogyakarta). + +## Related ecosystem (community, for context) + +- `github.com/vyonizr/tarif-krl` — KRL fare calculator "based on the official APIs: `api-partner.krl.co.id` and `jakartamrt.co.id`" — independent confirmation that the partner API is the de-facto official source. +- `github.com/Fannandya/api-krl` — KRL network-data SaaS (registration, API keys, daily quotas; curated reference data). +- Live-ish KRL trackers (different approach, NOT audited): `trackmycommuter.com`, `krljakarta.com` ("updates every 30 seconds"), `botkita.com/krl-tracker`. diff --git a/jakarta-public-transport-data/index.md b/jakarta-public-transport-data/index.md new file mode 100644 index 0000000..b79b7da --- /dev/null +++ b/jakarta-public-transport-data/index.md @@ -0,0 +1,46 @@ +--- +title: "Jakarta Public Transport Data" +description: "Umbrella topic for Jakarta-area public transport data — how each operator's realtime/schedule systems actually work, what is publicly reachable, and what is usable for future projects. Subtopics: TransJakarta live bus feed; KRL/Comuline schedules." +status: active +category: technical +tags: [jakarta-public-transport-data, transit-data, jakarta, jabodetabek, transjakarta, krl, commuter-line, open-data, reverse-engineering] +draft: true +created: 2026-10-12 +updated: 2026-10-12 +--- + +# Jakarta Public Transport Data + +## Summary + +Umbrella topic for **Jakarta-area public transport data**: what data each operator exposes, how third-party apps actually get it, and what is usable for future projects. Each system lives in a subtopic below; the shared thread is reverse-engineering public transport data pipelines and documenting them for reuse. + +## Subtopic Map + +- [[transjakarta-live-data/index|TransJakarta Live Data]] — bus lens: anonymous MQTT fleet stream + guest REST API + official GTFS; verified via cross-capture (audited Oct 2026) +- [[comuline-krl-data/index|Comuline / KRL Schedule Data]] — rail lens: KRL Commuter Line schedules replicated from KAI Commuter's official partner API; currently a frozen snapshot (audited Oct 2026) + +## Research Question + +Which Jakarta public transport data sources are publicly reachable today, how do they work end-to-end, and which are stable enough to build on? + +## Key Findings + +- **TransJakarta buses: live and open (as of Oct 2026).** Anonymous MQTT broker (`wss://mqtt.tj.co.id:8084/mqtt`) + guest-token REST API + official GTFS give the full realtime fleet (~3.3–3.7k buses). Usable today. +- **KRL commuter line: schedules only, and stale.** Comuline replicates KAI Commuter's partner API, but its sync stopped ~Mar 2025; its public API still serves that snapshot. No live train positions there. +- **Pattern across both:** operators rarely publish developer-facing APIs; practical access comes from app backends, partner endpoints, and open replication projects that document the pipelines. Access is undocumented and can change without notice. + +## Related Research + +- [[transjakarta-live-data/index|TransJakarta Live Data]] — first audit (origin of this umbrella) +- [[comuline-krl-data/index|Comuline / KRL Schedule Data]] — second audit + +## Discussion + +Branched 2026-10-12 per user request after two audits: TransJakarta (via `transit.oyenza.com`) and KRL schedules (via `comuline.com`). Both were already documented as separate topics; this umbrella gathers them under one home and is the place to add future systems (MRT Jakarta, LRT Jabodebek, TransJabodetabek, etc.). + +## Next Steps + +- [ ] Add subtopics for other Jabodetabek systems as they get audited (MRT Jakarta, LRT Jakarta/Jabodebek, TransJabodetabek) +- [ ] Re-verify access status of both existing feeds periodically (undocumented; can change) +- [ ] Keep per-system "verified on" dates current in subtopics diff --git a/transjakarta-live-data/index.md b/transjakarta-live-data/index.md index c497356..f2e9fa7 100644 --- a/transjakarta-live-data/index.md +++ b/transjakarta-live-data/index.md @@ -3,7 +3,7 @@ title: "TransJakarta live data — verified stack: MQTT stream, app REST API, GT description: "How TransJakarta realtime bus data works and how to use it — anonymous MQTT broker, guest REST API, official GTFS. Verified by cross-capture, Oct 2026." status: concluded category: technical -tags: [transjakarta, mqtt, realtime, gtfs, reverse-engineering, transit-data] +tags: [transjakarta, mqtt, realtime, gtfs, reverse-engineering, transit-data, jakarta-public-transport-data] draft: true created: 2026-10-12 updated: 2026-10-12 @@ -41,3 +41,8 @@ updated: 2026-10-12 - [[transjakarta-live-data/methodology|Methodology]] — how it was verified (repeatable) - [[transjakarta-live-data/recipes|Recipes]] — connection examples - [[transjakarta-live-data/next-steps|Next steps]] — risks + future ideas + +## Related Research + +- [[jakarta-public-transport-data/index|Jakarta Public Transport Data]] — umbrella topic +- [[comuline-krl-data/index|Comuline / KRL Schedule Data]] — sibling audit (rail schedules)