Add jabodetabek-cctv-live-data topic (DKI/BPJT/Bekasi/Bogor CCTV sources, stream stacks); link from umbrella

jabodetabek-cctv-live-data/methodology.md

@@ -0,0 +1,32 @@
+# Methodology — how this was verified
+
+Repeatable procedure (executed Oct 13, 2026). Tools: `curl`, a bash probe script, Python parsing (via Hermes `execute_code`).
+
+## Steps
+
+1. **Map the target:** fetch `https://transit.oyenza.com/cctv` + its JS bundles with curl (Chrome UA). Identify: catalog asset (`/assets/asset-cctv-<hash>.json`, fetched at runtime), player (hls.js chunk), map (MapLibre, `/tiles`).
+2. **Extract the catalog:** download the JSON — 214 cams / 310 views with `[label, hls, ownerPage]` per view and a `checkedAt` timestamp. Group views by `src` (dki/tol/bks/bgr).
+3. **Fetch each portal and extract its camera identifiers:**
+   - DKI: `jakcctv.jakarta.go.id/publik/data?q=&page=N` for pages 1–7; regex the `cards` HTML for iframe `src`s.
+   - Toll: `bpjt.pu.go.id/cctv/`; unescape `\/`; extract the inline JSON records (`streamhls`, `bujt`, lat/lon, status) by position-pairing fields; also raw-regex all `https://….m3u8` matches and strip `\r\n` artifacts.
+   - Bekasi: `eofficev2.bekasikota.go.id/backupcctv/m3/list.json`.
+   - Bogor: BSW `/cctv` list + `/cctv/<id>/detail`. **Geo-blocked from our network** → fetched through a public reader proxy (`https://r.jina.ai/<url>`); the vantage change is noted in findings. A sibling host (`restreamer2.kotabogor.go.id`) was reachable directly and fingerprinted.
+4. **Cross-match programmatically:** set comparisons of view URLs / camera IDs / filenames; compute overlaps (46/46, 190/190, 23/23, 51/51). Where a comparison failed initially, re-check for parsing artifacts before concluding (two "missing" DKI cams and one "missing" toll URL turned out to be regex/escaping artifacts, not real gaps).
+5. **Probe the streams:** one request per distinct stream host (12 hosts) with browser-like headers, `Origin: https://transit.oyenza.com`; save status + full headers + playlist bytes (`probes/`). Then hotlink test: repeat a request with no `Origin`/`Referer` → still 200. Identify server software from headers (`mediamtx`, `Streamer`, `cloudflare`, `nginx`) and URL shapes (`LiveApp/`, `memfs/`, `tracks-v1/`).
+6. **Read the player path in the bundle:** locate `fetch(<catalog>)`, the `views` mapping, the hls.js dynamic import, `loadSource`, the Safari fallback, and the owner-page fullscreen handler.
+
+## Boundaries
+
+- Public, unauthenticated content only; single targeted fetches; no crawling (the site's robots.txt says `Disallow: /`), no account creation, no auth/payment bypass, nothing beyond what a visitor's browser does.
+- Geo-blocked portals were read through a public reader proxy for verification only — labeled as such.
+
+## Key results
+
+- 100% catalog↔portal agreement on all four sources (DKI 46/46 views, Toll 190/190 views, Bekasi 23/23, Bogor 51/51 detail IDs).
+- 12/12 stream hosts live at capture; CORS `*`; no Origin/Referer checks.
+- BPJT page: 1,530 records / 1,415 unique URLs; oyenza's toll subset = 9 operators, 154 cams.
+
+## Artifacts
+
+- Archive: `workspace/cctv-jabodetabek-captures/` (3.3 MB, 55 files): catalog JSON, page+bundle dumps, portal dumps (`portals/`), probe headers/bodies (`probes/`), `probe.sh`.
+- Scratch copies live under `~/.hermes/profiles/research/cache/scratch/cctv-audit/` (auto-pruned after ~24 h idle — use the archive instead).