Add jabodetabek-cctv-live-data topic (DKI/BPJT/Bekasi/Bogor CCTV sources, stream stacks); link from umbrella

jabodetabek-cctv-live-data/analysis.md

@@ -0,0 +1,56 @@
+# Analysis — the chain, overlaps, stacks
+
+## The chain
+
+```
+camera owners (Dishub/bank/operator cameras)
+  → operator/government stream servers (HLS: Ant Media, MediaMTX, datarhei, custom)
+    → government portals publish catalogs (HTML/JSON; some embed player iframes)
+      → transit.oyenza.com harvests + verifies + curates → static catalog in its bundle
+        → visitor's browser plays the owner's HLS URL directly (hotlink, no proxy)
+```
+
+## Evidence per source (cross-match against oyenza's catalog)
+
+Method: extract oyenza's view URLs from `asset-cctv.json`; extract the portal's camera identifiers from its page/JSON; compare sets exactly (case-insensitively where the portal is inconsistent).
+
+- **DKI — 46/46.** Portal `cards` iframes carry `<camera-id>/embed.html`; oyenza lists `<camera-id>/index.fmp4.m3u8` + the same `embed.html` as "owner page". Every oyenza DKI path appears in the portal set (52 cameras at capture). Example id: `501844_JKP_DISHUB_JL.-MH.-THAMRIN-C01_CCTV-01`.
+- **Toll — 190/190.** BPJT's inline JSON carries `streamhls` values; after unescaping (`\/`) and cleaning a few literal `\r\n` artifacts, **all 190** oyenza toll view URLs appear verbatim. BPJT page at capture: 1,530 records → 1,415 unique URLs across ~39 hosts.
+- **Bekasi — 23/23.** oyenza's 23 stream filenames all match `list.json` `name` values exactly.
+- **Bogor — 51/51.** All 51 unique detail-link IDs (`/cctv/<id>/detail`) used by oyenza appear on the BSW page (96 IDs total). Strongest link: the Alun-Alun detail page (id 82) embeds exactly `https://restreamer2.kotabogor.go.id/memfs/c07c1926-288c-46e4-a19c-9f51022edc5d.m3u8` — the URL oyenza lists for that camera.
+
+Interpretation: oyenza is a **curated mirror**, not a re-publisher with its own pipeline to the cameras. The catalog is rebuilt and re-verified (footer "Daftar diperiksa 13 Oktober 2026"; `checkedAt` 2026-10-12T22:07:39Z).
+
+## Selection logic (the 154 toll cams)
+
+oyenza's toll subset = 190 views / 154 cams from 9 operators: Hutama Karya 58, Trans Bumi Serbaraja 29, Cimanggis-Cibitung 19, Cibitung-Tanjung Priok 19, JTD 8, Marga Mandalasakti 7, KKDM 6, Jakarta Lingkar Baratsatu 5, CMNP 3.
+
+Not simple geography: 589 BPJT cams fall inside a Jabodetabek bounding box, but only ~190 views were taken. Conspicuous exclusions, with live checks:
+
+- **Jasa Marga (~605 cams) — absent.** `jmlive.jasamarga.com` returned HTTP 200 with an **empty body** (manifest not publicly served to us). Exclusion is consistent with "dead unless probed from a permitted vantage".
+- **BSD Toll — absent.** `stream.bsdtol.com` → nginx **502** at capture.
+- **Waskita (`wbw`, 55) — absent** even though its streams work (playlist verified) → curation beyond pure liveness (coverage-area choice).
+
+**Confidence:** Stream-liveness exclusions (JM, BSD) = established via probes. "Verify-then-pick + Jabodetabek coverage" selection rule = strong inference; the exact rule isn't observable from outside.
+
+## Stream-stack fingerprints (observed live)
+
+- `/LiveApp/streams/<id>.m3u8` → **Ant Media Server** — HK (Cloudflare-fronted), Serbaraja, CCT. 2 s TS segments.
+- `server: mediamtx` → **MediaMTX** — CTP. Master playlist → `stream.m3u8`.
+- `/memfs/<uuid>.m3u8` (+ `?session=` on sub-playlists) → **datarhei Restreamer** — all Bogor servers (also bsdtol, PPTR).
+- `index.fmp4.m3u8` + `tracks-v1/` + `X-Sid`/`X-Run-Time` headers, `server: Streamer 26.04.1` → **Bali Tower custom** — DKI.
+- nginx + `X-Powered-By: Express` packagers → JTD, KKDM. Plain nginx/Apache HLS → MMS, JORRW1, Citramarga, Bekasi.
+
+## Playback path (from the site's own code)
+
+- Catalog fetched at runtime: `fetch("/assets/asset-cctv-<hash>.json")`.
+- Views mapped to `{label, hls, page}`; on tap: `import("/assets/lib-E4YH4VVH.js")` (hls.js) → `hls.loadSource(view.hls); hls.attachMedia(video)`; Safari falls back to native `canPlayType("application/vnd.apple.mpegurl")` → `video.src`.
+- "Buka Layar penuh" opens `view.page` — the owner's page (portal or `embed.html`).
+- No proxy: streams load from the owner host in the visitor's browser. Probed hosts sent `Access-Control-Allow-Origin: *`, and returned 200 even without `Origin`/`Referer` — no hotlink protection at capture.
+- The site's own claim ("Video diputar langsung dari server pemilik kamera dan tidak kami simpan") matches the code path.
+
+## Anti-overclaim notes
+
+- The static catalog proves what the client understands (URLs, labels, owner pages). It does not prove the operator's exact checker tooling — no public repo was found; `checkedAt` + redeploy hashes = inference of a periodic probe + rebuild.
+- Uptime varies: one host timed out once and answered on retry. Treat single probes as readings, not facts — timestamp them (done: Oct 13, 2026).
+- Access policy can change at any time; CORS `*` today ≠ CORS `*` forever.

jabodetabek-cctv-live-data/index.md

@@ -0,0 +1,47 @@
+---
+title: "Jabodetabek CCTV live data — four public sources, hotlinked HLS"
+description: "How the Jabodetabek public CCTV layer works: DKI Jakarta, BPJT toll-road, Bekasi, and Bogor camera catalogs, and the owner-hosted HLS streams that apps hotlink. Audited Oct 2026 via transit.oyenza.com/cctv."
+status: concluded
+category: technical
+tags: [jabodetabek, cctv, video-streaming, hls, reverse-engineering, transit-data, jakarta-public-transport-data, balitower, bpjt]
+draft: true
+created: 2026-10-13
+updated: 2026-10-13
+---
+
+# Jabodetabek CCTV live data — verified upstreams
+
+**What this is:** a verified map of the public CCTV layer across Jabodetabek — which government portals publish camera catalogs, what stream servers actually serve the video, and how third-party apps assemble and play them.
+
+**Origin:** reverse-engineering the `/cctv` page of `transit.oyenza.com` (the same tracker from [[transjakarta-live-data/index|TransJakarta Live Data]]) to determine where its camera list and live video come from.
+
+## TL;DR
+
+- **Four upstreams:** DKI Jakarta (`jakcctv.jakarta.go.id`), BPJT toll roads (`bpjt.pu.go.id/cctv`), Kota Bekasi, Kota Bogor (BSW). All four cross-matched to oyenza's list with ~100% URL/ID agreement — see [[jabodetabek-cctv-live-data/analysis|Analysis]].
+- **No aggregator API:** oyenza ships a static catalog (214 cams / 310 views) inside its own bundle and hotlinks each camera's HLS URL directly from the owner's server (hls.js; CORS `*`; no proxy, no tokens at capture).
+- **Toll layer = one page scrape:** BPJT's page embeds an inline JSON array of ~1,530 cameras (stream URL + operator + coordinates + status); the app's 154 toll cams are a curated, working Jabodetabek subset (all 190 of its toll view URLs appear verbatim).
+- **Stream stacks fingerprinted:** Ant Media Server (`/LiveApp/streams/`), MediaMTX, datarhei Restreamer (`memfs/`), Bali Tower custom fMP4 HLS, and plain nginx/Apache HLS.
+
+## Key verified numbers (Oct 13, 2026)
+
+- oyenza catalog: 214 cameras / 310 views — DKI 14/46, toll 154/190, Bekasi 23/23, Bogor 23/51.
+- Cross-match: DKI 46/46 views, toll 190/190 views, Bekasi 23/23 names, Bogor 51/51 detail IDs (the Bogor Alun-Alun detail page embeds the exact `restreamer2.kotabogor.go.id/memfs/…` URL oyenza lists).
+- Stream hosts probed live: **12/12** answered with valid HLS playlists (2–8 s segments).
+- Catalog `checkedAt`: `2026-10-12T22:07:39Z` (page footer: "Daftar diperiksa 13 Oktober 2026").
+
+## Using it
+
+- Per-source endpoints and stream hosts: [[jabodetabek-cctv-live-data/sources|Sources]]
+- Evidence archive: `workspace/cctv-jabodetabek-captures/` (catalog JSON, portal dumps, probe logs)
+
+## Files
+
+- [[jabodetabek-cctv-live-data/sources|Sources]] — portals, stream hosts per operator, context
+- [[jabodetabek-cctv-live-data/analysis|Analysis]] — the chain, overlap evidence, stack fingerprints
+- [[jabodetabek-cctv-live-data/methodology|Methodology]] — how it was verified (repeatable)
+- [[jabodetabek-cctv-live-data/next-steps|Next steps]] — risks, gaps, ideas
+
+## Related Research
+
+- [[jakarta-public-transport-data/index|Jakarta Public Transport Data]] — umbrella topic
+- [[transjakarta-live-data/index|TransJakarta Live Data]] — same site, bus layer

jabodetabek-cctv-live-data/methodology.md

@@ -0,0 +1,32 @@
+# Methodology — how this was verified
+
+Repeatable procedure (executed Oct 13, 2026). Tools: `curl`, a bash probe script, Python parsing (via Hermes `execute_code`).
+
+## Steps
+
+1. **Map the target:** fetch `https://transit.oyenza.com/cctv` + its JS bundles with curl (Chrome UA). Identify: catalog asset (`/assets/asset-cctv-<hash>.json`, fetched at runtime), player (hls.js chunk), map (MapLibre, `/tiles`).
+2. **Extract the catalog:** download the JSON — 214 cams / 310 views with `[label, hls, ownerPage]` per view and a `checkedAt` timestamp. Group views by `src` (dki/tol/bks/bgr).
+3. **Fetch each portal and extract its camera identifiers:**
+   - DKI: `jakcctv.jakarta.go.id/publik/data?q=&page=N` for pages 1–7; regex the `cards` HTML for iframe `src`s.
+   - Toll: `bpjt.pu.go.id/cctv/`; unescape `\/`; extract the inline JSON records (`streamhls`, `bujt`, lat/lon, status) by position-pairing fields; also raw-regex all `https://….m3u8` matches and strip `\r\n` artifacts.
+   - Bekasi: `eofficev2.bekasikota.go.id/backupcctv/m3/list.json`.
+   - Bogor: BSW `/cctv` list + `/cctv/<id>/detail`. **Geo-blocked from our network** → fetched through a public reader proxy (`https://r.jina.ai/<url>`); the vantage change is noted in findings. A sibling host (`restreamer2.kotabogor.go.id`) was reachable directly and fingerprinted.
+4. **Cross-match programmatically:** set comparisons of view URLs / camera IDs / filenames; compute overlaps (46/46, 190/190, 23/23, 51/51). Where a comparison failed initially, re-check for parsing artifacts before concluding (two "missing" DKI cams and one "missing" toll URL turned out to be regex/escaping artifacts, not real gaps).
+5. **Probe the streams:** one request per distinct stream host (12 hosts) with browser-like headers, `Origin: https://transit.oyenza.com`; save status + full headers + playlist bytes (`probes/`). Then hotlink test: repeat a request with no `Origin`/`Referer` → still 200. Identify server software from headers (`mediamtx`, `Streamer`, `cloudflare`, `nginx`) and URL shapes (`LiveApp/`, `memfs/`, `tracks-v1/`).
+6. **Read the player path in the bundle:** locate `fetch(<catalog>)`, the `views` mapping, the hls.js dynamic import, `loadSource`, the Safari fallback, and the owner-page fullscreen handler.
+
+## Boundaries
+
+- Public, unauthenticated content only; single targeted fetches; no crawling (the site's robots.txt says `Disallow: /`), no account creation, no auth/payment bypass, nothing beyond what a visitor's browser does.
+- Geo-blocked portals were read through a public reader proxy for verification only — labeled as such.
+
+## Key results
+
+- 100% catalog↔portal agreement on all four sources (DKI 46/46 views, Toll 190/190 views, Bekasi 23/23, Bogor 51/51 detail IDs).
+- 12/12 stream hosts live at capture; CORS `*`; no Origin/Referer checks.
+- BPJT page: 1,530 records / 1,415 unique URLs; oyenza's toll subset = 9 operators, 154 cams.
+
+## Artifacts
+
+- Archive: `workspace/cctv-jabodetabek-captures/` (3.3 MB, 55 files): catalog JSON, page+bundle dumps, portal dumps (`portals/`), probe headers/bodies (`probes/`), `probe.sh`.
+- Scratch copies live under `~/.hermes/profiles/research/cache/scratch/cctv-audit/` (auto-pruned after ~24 h idle — use the archive instead).

jabodetabek-cctv-live-data/next-steps.md

@@ -0,0 +1,23 @@
+# Next steps
+
+## Re-verify (access is undocumented and can change)
+
+- [ ] Re-run the per-host probe (script pattern in `methodology.md`; archive has `probe.sh`) — catalogue the current live/dead state with a fresh timestamp.
+- [ ] Watch for CORS/Referer/token introduction by any operator (any host switching from `Access-Control-Allow-Origin: *` breaks the hotlink pattern).
+- [ ] Track oyenza's catalog hash (`/assets/asset-cctv-<hash>.json`) — a change signals a re-check/redeploy; the `checkedAt` value gives the cadence.
+
+## Gaps to close
+
+- [ ] **Verify from an Indonesian vantage:** `bsw.kotabogor.go.id` and `cctv.balitower.co.id` TCP-timeout from our network (likely geo/IP filtering). Confirm both are alive and that BSW's detail pages are the only listing for Bogor cams (we verified via reader proxy).
+- [ ] Confirm the DKI portal's full camera count trend (52 at capture) and whether oyenza adds future DKI cams beyond its current 14 roads.
+- [ ] Jasa Marga exclusion: check `jmlive.jasamarga.com` from an Indonesian vantage — if the manifest serves there, its absence from oyenza implies a deliberate policy/coverage choice, not a dead feed.
+
+## Candidate sources to verify next (leads, not yet re-verified)
+
+- `atcs-jakarta.com` (Dishub ATCS map), `atcs-bptj.dephub.go.id` (BPTJ/BKJT streaming), `smartcity.jakarta.go.id` CCTV layer, `lewatmana.com/cam` (DKI re-embedder), `cctv.balitower.co.id` (Balitower public portal).
+- Non-Jabodetabek: BPJT page carries cams nationwide (Makassar, Sumatra) — same scrape pattern applies if ever needed.
+
+## Ideas
+
+- [ ] Small prober tool: check all 310 catalog URLs (or any list of HLS URLs), report live/dead + segment cadence — reusable for any future "is this feed up?" question.
+- [ ] Snapshot tool for one camera (grab a few segments' worth of playlist + segment list) mirroring `transjakarta-live-data/scripts/tj-snapshot.mjs` style.

jabodetabek-cctv-live-data/sources.md

@@ -0,0 +1,42 @@
+# Sources
+
+★ = directly verified during this investigation (Oct 13, 2026). Unmarked items are leads not yet re-verified.
+
+## Government portals (catalog sources)
+
+- ★ `https://jakcctv.jakarta.go.id/publik` — DKI Jakarta public CCTV portal. The page is a server-rendered paginator; the underlying data endpoint is `https://jakcctv.jakarta.go.id/publik/data?q=&page=N`, returning JSON `{cards, pagination, count, total}`. Each card embeds an iframe to a Bali Tower player (`embed.html`). 52 cameras listed at capture.
+- ★ `https://bpjt.pu.go.id/cctv/` — BPJT (Kementerian PU) toll-road page. Renders an **inline JSON array of ~1,530 camera records** directly in the page HTML: per-cam `streamhls` (HLS URL), `bujt` (operator tag), `nama_km`, `lat`, `lon`, `status`, `updated_at`. This is the single source for the toll layer.
+- ★ `https://bekasikota.go.id/cctv` — Kota Bekasi page. Consumes `https://eofficev2.bekasikota.go.id/backupcctv/m3/list.json` (24 entries: `path`, `name`, `pic`). Thumbnails hosted on `pemkotbekasi.github.io/cctv_pic/` (fronted by i0.wp.com).
+- ★ `https://bsw.kotabogor.go.id/cctv` — Kota Bogor "Bogor Single Window" page; per-camera detail pages at `/cctv/<id>/detail` (96 camera IDs observed). Detail pages embed datarhei Restreamer HLS. **TCP-timeout from our network** (likely geo/IP filtering); read via a reader proxy — see [[jabodetabek-cctv-live-data/methodology|Methodology]].
+
+## Stream servers by operator (what actually serves video)
+
+- ★ **DKI Jakarta** — `dki-jkt.balitower.co.id:7028` — Bali Tower (PT Bali Towerindo Sentra) custom stack, `server: Streamer 26.04.1`. fMP4 HLS (`index.fmp4.m3u8`, multi-track under `tracks-v1/`), ~7.5 s segments, separate `embed.html` "Live Player" page per camera. Camera paths encode the agency, e.g. `501844_JKP_DISHUB_JL.-MH.-THAMRIN-C01_CCTV-01`.
+- ★ **Hutama Karya (HK)** — `pub2.hk-opt.com` (+ `pub.hk-opt.com`) — Ant Media Server (`/LiveApp/streams/<id>.m3u8`), behind Cloudflare; 2.07 s segments. The operator's own CCTV portal is `extern.hk-opt.com` ("CCTV External — Divisi OPT — PT Hutama Karya (Persero)"), plus `traffic.hk-opt.com` ("SMARTCCTV").
+- ★ **Trans Bumi Serbaraja** — `info.serbaraja-tollroad.co.id:20443` — Ant Media Server; ~2 s segments.
+- ★ **Cimanggis–Cibitung Tollways (CCT)** — `streaming-cct.co.id` — Ant Media Server.
+- ★ **Cibitung–Tanjung Priok (CTP)** — `cctv.ctptollways.com:2254` — **MediaMTX** (`server: mediamtx`); master playlist → `stream.m3u8`.
+- ★ **Jakarta Tollroad Development (JTD)** — `camera.jtd.co.id` — nginx + Express packager (`X-Powered-By: Express`); `camera/share/tios/<a>/<b>/index.m3u8`.
+- ★ **Marga Mandalasakti (MMS)** — `cctv.mms-corp.id` — nginx HLS (`<cam>/stream.m3u8`, `_seg<N>.ts`).
+- ★ **Kresna Kusuma Dyandra Marga (KKDM)** — `cctv.kkdm.co.id` — nginx + Express (`api/cctv/<a>/<b>/index.m3u8`, `seg_<N>.ts`).
+- ★ **Jakarta Lingkar Baratsatu (JLB)** — `cctvjorrw1.com` — nginx HLS (e.g. `KM/7+950B.m3u8`), ~5 s segments.
+- ★ **CMNP** — `cctv.citramarga.com` — Apache HLS (`hls/cam3-…/stream.m3u8`), ~1 s segments.
+- ★ **Pemkot Bekasi** — `eofficev2.bekasikota.go.id` — nginx serving `backupcctv/m3/…` (city e-office host).
+- ★ **Pemkot Bogor** — `restreamer{1,2,3,4,5,6}.kotabogor.go.id` — **datarhei Restreamer** (`memfs/<uuid>.m3u8`; sub-playlists append a `?session=` token).
+
+Not selected by the audited app but present on BPJT's page: Jasa Marga (`jmlive.jasamarga.com`, ~605 cams — manifest returns empty 200 from outside), Waskita (`cctv.waskitabumiwira.com`), BSD Toll (`stream.bsdtol.com` — datarhei, 502 at capture), PPTR (`pptrlive.co.id` — datarhei), and ~25 other hosts.
+
+## Site under study
+
+- ★ `https://transit.oyenza.com/cctv` — "CCTV Jakarta Live" third-party map. Key surfaces:
+  - static catalog: `/assets/asset-cctv-<hash>.json` (fetched at runtime by `cctv-<hash>.js`)
+  - playback: hls.js (lazy-loaded chunk) → directly loads each view's owner-hosted URL
+  - map: MapLibre with self-hosted tiles (`/tiles`)
+  - analytics: `/_e` (POST events)
+  - robots.txt: `Disallow: /`
+- Related domains: `transit.vantis.sh` (alias), `vantis.sh` — same developer, per the TransJakarta audit.
+
+## Context (third-party / first-party-adjacent)
+
+- `https://smartcity.jakarta.go.id/en/blog/pantau-lalu-lintas-dan-keamanan-dengan-cctv-jakarta/` — Jakarta Smart City blog listing Balitower `embed.html` links for city cameras (first-party-adjacent evidence that Balitower hosts DKI public camera streams).
+- `https://iameberhard.com/jakarta-cams` and its blog post — independent dashboard over Balitower / LewatMana / ATCS feeds; names further aggregators (LewatMana, `atcs-jakarta.com`, `atcs-bptj.dephub.go.id`). Third-party; not re-verified here.