0xrsydn/research
1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
research/jabodetabek-cctv-live-data/methodology.md

3.2 KiB
Raw Permalink Blame History

Methodology — how this was verified

Repeatable procedure (executed Oct 13, 2026). Tools: curl, a bash probe script, Python parsing (via Hermes execute_code).

Steps

  1. Map the target: fetch https://transit.oyenza.com/cctv + its JS bundles with curl (Chrome UA). Identify: catalog asset (/assets/asset-cctv-<hash>.json, fetched at runtime), player (hls.js chunk), map (MapLibre, /tiles).
  2. Extract the catalog: download the JSON — 214 cams / 310 views with [label, hls, ownerPage] per view and a checkedAt timestamp. Group views by src (dki/tol/bks/bgr).
  3. Fetch each portal and extract its camera identifiers:
    • DKI: jakcctv.jakarta.go.id/publik/data?q=&page=N for pages 1–7; regex the cards HTML for iframe srcs.
    • Toll: bpjt.pu.go.id/cctv/; unescape \/; extract the inline JSON records (streamhls, bujt, lat/lon, status) by position-pairing fields; also raw-regex all https://….m3u8 matches and strip \r\n artifacts.
    • Bekasi: eofficev2.bekasikota.go.id/backupcctv/m3/list.json.
    • Bogor: BSW /cctv list + /cctv/<id>/detail. Geo-blocked from our network → fetched through a public reader proxy (https://r.jina.ai/<url>); the vantage change is noted in findings. A sibling host (restreamer2.kotabogor.go.id) was reachable directly and fingerprinted.
  4. Cross-match programmatically: set comparisons of view URLs / camera IDs / filenames; compute overlaps (46/46, 190/190, 23/23, 51/51). Where a comparison failed initially, re-check for parsing artifacts before concluding (two "missing" DKI cams and one "missing" toll URL turned out to be regex/escaping artifacts, not real gaps).
  5. Probe the streams: one request per distinct stream host (12 hosts) with browser-like headers, Origin: https://transit.oyenza.com; save status + full headers + playlist bytes (probes/). Then hotlink test: repeat a request with no Origin/Referer → still 200. Identify server software from headers (mediamtx, Streamer, cloudflare, nginx) and URL shapes (LiveApp/, memfs/, tracks-v1/).
  6. Read the player path in the bundle: locate fetch(<catalog>), the views mapping, the hls.js dynamic import, loadSource, the Safari fallback, and the owner-page fullscreen handler.

Boundaries

  • Public, unauthenticated content only; single targeted fetches; no crawling (the site's robots.txt says Disallow: /), no account creation, no auth/payment bypass, nothing beyond what a visitor's browser does.
  • Geo-blocked portals were read through a public reader proxy for verification only — labeled as such.

Key results

  • 100% catalog↔portal agreement on all four sources (DKI 46/46 views, Toll 190/190 views, Bekasi 23/23, Bogor 51/51 detail IDs).
  • 12/12 stream hosts live at capture; CORS *; no Origin/Referer checks.
  • BPJT page: 1,530 records / 1,415 unique URLs; oyenza's toll subset = 9 operators, 154 cams.

Artifacts

  • Archive: workspace/cctv-jabodetabek-captures/ (3.3 MB, 55 files): catalog JSON, page+bundle dumps, portal dumps (portals/), probe headers/bodies (probes/), probe.sh.
  • Scratch copies live under ~/.hermes/profiles/research/cache/scratch/cctv-audit/ (auto-pruned after ~24 h idle — use the archive instead).