feat(rubick): rename project and add production release bundle

2026-08-07 01:33:52 +00:00 · 2026-03-06 03:12:28 +07:00 · 2026-03-06 03:12:28 +07:00 · fab6cf26eb
commit fab6cf26eb
33 changed files with 11554 additions and 0 deletions
--- a/README.md
+++ b/README.md
@ -0,0 +1,784 @@
+# Rubick
+
+Enterprise-ready, multi-language CLI platform for Indonesian market intelligence.
+
+This repository combines:
+- high-throughput data collection from MSN Finance (Go),
+- news discovery from Brave Search (Go),
+- article-body extraction using `newspaper` (Python),
+- structured export to JSON/CSV/XLSX (Python),
+- SQLite persistence for repeatable historical analysis.
+
+The system is built as one unified CLI with clear subcommands and strongly typed runtime boundaries so you can extend it safely without scattering code.
+
+## Why This Project Exists
+
+The goal is to solve a practical workflow problem:
+- collect Indonesian stock fundamentals and snapshots,
+- enrich stock context with real article text instead of headline-only snippets,
+- persist data in a durable local store,
+- export clean artifacts for analysis, reporting, and downstream automation.
+
+Common use cases:
+- analyst daily market snapshots,
+- watchlist intelligence pipelines,
+- historical trend extraction and spreadsheet exports,
+- scheduled ETL feeding BI/ML workloads.
+
+## Core Capabilities
+
+- Unified CLI (`rubick`) with domain commands: `msn`, `news`, `export`, `extractor`.
+- Live stock metadata and multi-endpoint stock fetch pipelines.
+- Controlled concurrency and rate limiting for network-bound tasks.
+- Python extractor process connected over Unix socket for fast IPC and language isolation.
+- SQLite-based persistence for resumable scraping runs.
+- Multiple export formats for both machine and human consumption.
+- Regression tests and live API e2e test coverage.
+
+## High-Level Architecture
+
+```mermaid
+flowchart LR
+    U[User / Scheduler] --> C[Unified CLI<br/>Go]
+
+    C --> M[MSN Workflows<br/>Go]
+    C --> N[News Workflow<br/>Go]
+    C --> E[Export Workflow<br/>Python]
+
+    M --> MSNAPI[MSN Finance APIs]
+    M --> DB[(SQLite)]
+
+    N --> BRAVE[Brave News API]
+    N --> FETCH[HTML Fetcher]
+    FETCH --> IPC[Unix Socket IPC]
+    IPC --> PX[Python Extractor<br/>newspaper]
+    N --> OUTJSON[News JSON Output]
+
+    E --> DB
+    E --> XLSX[XLSX / CSV / JSON Export Files]
+```
+
+## Runtime Data Flow
+
+### 1. MSN bulk ingestion (`msn fetch-all`)
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant CLI as Go CLI
+    participant DB as SQLite
+    participant MSN as MSN API
+
+    User->>CLI: msn fetch-all --index idx30 --db output/run/stocks.db
+    CLI->>DB: start scrape_run + init progress
+    loop each stock (concurrency workers)
+        CLI->>MSN: fetch stock endpoints
+        MSN-->>CLI: payloads + statuses
+        CLI->>DB: save stock + history + news + ratios
+        CLI->>DB: update progress row
+    end
+    CLI->>DB: complete scrape_run
+    CLI-->>User: progress + summary
+```
+
+### 2. News enrichment (`news`)
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant CLI as Go CLI
+    participant Brave as Brave API
+    participant Web as Article URL
+    participant Py as Python Extractor
+    participant Out as JSON File
+
+    User->>CLI: news "IHSG" --count 20
+    CLI->>Brave: search news
+    Brave-->>CLI: URL list
+    CLI->>Py: start extractor process + socket pool
+    loop each URL (workers)
+        CLI->>Web: GET page HTML
+        CLI->>Py: extract(url, html)
+        Py-->>CLI: text/status
+    end
+    CLI->>Out: write enriched JSON
+    CLI-->>User: extraction summary
+```
+
+## Repository Structure and Folder Contracts
+
+```text
+.
+├── cmd/
+│   └── rubick/      # canonical binary entrypoint
+├── internal/
+│   └── cli/                     # root command router + command implementations
+├── msn/                         # MSN domain library: client, db, types, limiter, stock universe
+├── scripts/                     # Python export scripts
+├── tests/
+│   ├── go/                      # Go CLI regression + live network e2e tests
+│   └── test_export_simple.py    # Python exporter tests
+├── extractor.py                 # Python unix-socket extraction service
+├── main.go                      # compatibility entrypoint for local dev only
+├── pyproject.toml               # Python deps and tooling
+├── go.mod / go.sum              # Go module deps
+└── README.md
+```
+
+Folder responsibilities:
+
+| Path | Responsibility | Extension Rule |
+|---|---|---|
+| `cmd/rubick` | binary entrypoint only | keep thin; no business logic |
+| `internal/cli` | command parsing, orchestration, runtime coordination | add new command handlers here |
+| `msn` | data-source domain package and persistence internals | keep API/service-specific logic here |
+| `scripts` | Python export implementations | each exporter should be independent CLI |
+| `tests/go` | end-to-end and CLI behavior tests | test via public CLI behavior, not internals |
+| `tests` | Python unit tests | one test module per script/component |
+
+## Language Boundary (Go + Python)
+
+Go is used for:
+- CLI UX,
+- concurrency and throughput,
+- networking and orchestration,
+- SQLite ingestion flows.
+
+Python is used for:
+- article extraction where ecosystem libraries are stronger,
+- export formatting (especially XLSX convenience).
+
+IPC contract:
+- transport: Unix domain socket,
+- framing: 4-byte big-endian length + JSON payload,
+- request shape: `{ "url": string, "html": string }`,
+- response shape: `{ "text": string, "status": "ok|failed", "error"?: string }`.
+
+Reliability safeguards already implemented:
+- per-run unique socket path to prevent collision,
+- deadline-aware socket calls,
+- full-frame read/write (`ReadFull` semantics),
+- unhealthy pooled connections are closed/replaced,
+- extractor process group termination and socket file cleanup.
+
+## Getting Started
+
+## Prerequisites
+
+- Go `1.25+`
+- Python `3.12+`
+- `uv` for Python environment/dependency management
+
+## Install
+
+```bash
+make setup
+make build
+```
+
+Manual equivalent:
+
+```bash
+go mod download
+uv sync
+mkdir -p bin
+go build -o bin/rubick ./cmd/rubick
+```
+
+## Environment
+
+Create `.env` for Brave-powered news commands:
+
+```bash
+cp .env.example .env
+# edit .env
+# BRAVE_API_KEY=your_key_here
+```
+
+## Running the CLI
+
+```bash
+# production/dev standard
+./bin/rubick <command> [options]
+```
+
+Developer-only shortcut (not recommended for operational runbooks):
+
+```bash
+go run ./cmd/rubick <command> [options]
+```
+
+## Command Catalog
+
+Root commands:
+
+| Command | Purpose | Output |
+|---|---|---|
+| `msn` | MSN finance workflows (`screener`, `fetch`, `fetch-all`, `lookup`) | JSON + SQLite |
+| `news` | Brave search + article extraction | JSON |
+| `export` | Python exports (`dashboard`, `history`, `simple`) | XLSX/CSV/JSON |
+| `extractor` | run extractor service directly (advanced/debug) | socket server |
+
+## Command Reference (Detailed)
+
+### `msn screener`
+
+Find stocks using preset screener filters.
+
+```bash
+./bin/rubick msn screener --region id --filter top-performers --limit 20 --output output/screener.json
+```
+
+Arguments:
+
+| Flag | Expected Value | Default | Validation | What It Does |
+|---|---|---|---|---|
+| `--region` | region code (`id`) | `id` | non-empty | target market region for screener query |
+| `--filter` | one preset from list below | `large-cap` | must map to known filter | selects screener criteria |
+| `--limit` | integer `>= 1` | `50` | strict integer and min bound | max rows returned |
+| `--output`, `-o` | file path | `screener_YYYYMMDD.json` | writable path | output JSON location |
+
+Supported filter presets:
+- `top-performers`
+- `worst-performers`
+- `high-dividend`
+- `low-pe`
+- `52w-high`
+- `52w-low`
+- `high-volume`
+- `large-cap`
+
+Expected output example:
+
+```json
+{
+  "filter": "top-performers",
+  "region": "id",
+  "generated_at": "2026-03-06T02:39:47Z",
+  "total": 20,
+  "stocks": [
+    {
+      "id": "bn91jc",
+      "symbol": "BBCA",
+      "name": "Bank Cntrl Asia",
+      "price": 9000.0,
+      "price_change_pct": 1.2
+    }
+  ]
+}
+```
+
+Variations:
+
+```bash
+# Small deterministic sample
+./bin/rubick msn screener --region id --filter large-cap --limit 3 -o output/run/screener_top3.json
+
+# Alternate filter
+./bin/rubick msn screener --filter low-pe --limit 50
+```
+
+### `msn lookup`
+
+Resolve ticker symbols to internal MSN IDs.
+
+```bash
+./bin/rubick msn lookup BBCA BBRI TLKM
+```
+
+Arguments:
+
+| Input | Expected Value | What It Does | Output |
+|---|---|---|---|
+| positional tickers | uppercase ticker symbols | maps tickers to static IDX dictionary | table printed to stdout |
+
+Example output:
+
+```text
+Ticker   MSN ID     Company Name
+--------------------------------------------------
+BBCA     bn91jc     Bank Cntrl Asia
+BBRI     bn6wly     Bank Rakyat Indonesia
+TLKM     bn4k6h     Telkom Indonesia
+--------------------------------------------------
+Found 3/3 tickers
+```
+
+### `msn fetch`
+
+Fetch comprehensive stock data for specific IDs/tickers or screener input.
+
+```bash
+./bin/rubick msn fetch --tickers BBCA,BBRI,TLKM --concurrency 5 --output output/fetch.json
+```
+
+Arguments:
+
+| Flag | Expected Value | Default | Validation | What It Does |
+|---|---|---|---|---|
+| `--input` | path to screener JSON | none | file must exist and parse | imports stock IDs from screener output |
+| `--ids` | comma-separated IDs | none | non-empty entries | fetch by explicit MSN IDs |
+| `--tickers` | comma-separated tickers | none | unknown tickers skipped with warning | resolves ticker to MSN ID |
+| `--concurrency` | integer `>= 1` | `5` | strict integer and min bound | worker parallelism |
+| `--output`, `-o` | file path | `stocks_YYYYMMDD.json` | writable path | output JSON |
+
+Behavior notes:
+- you must provide at least one source of IDs (`--input`, `--ids`, `--tickers`),
+- duplicate IDs are deduplicated before fetch,
+- fetch status is tracked per API subsection.
+
+Expected output (truncated):
+
+```json
+{
+  "generated_at": "2026-03-06T02:40:00Z",
+  "total": 2,
+  "stocks": [
+    {
+      "id": "bn91jc",
+      "symbol": "BBCA",
+      "fetch_status": {
+        "quote": "ok",
+        "profile": "ok",
+        "financials": "ok"
+      }
+    }
+  ]
+}
+```
+
+Variations:
+
+```bash
+# By explicit IDs
+./bin/rubick msn fetch --ids bn91jc,bn6wly -o output/run/fetch_ids.json
+
+# From screener output
+./bin/rubick msn fetch --input output/run/screener_top3.json --concurrency 2 -o output/run/fetch_from_screener.json
+```
+
+### `msn fetch-all`
+
+Bulk ingest index constituents into SQLite with progress tracking.
+
+```bash
+./bin/rubick msn fetch-all --index idx30 --db output/stocks.db --rps 20 --delay 100-500 --concurrency 3
+```
+
+Arguments:
+
+| Flag | Expected Value | Default | Validation | What It Does |
+|---|---|---|---|---|
+| `--db` | sqlite file path | `output/stocks.db` | writable path | target DB |
+| `--index` | `all` / `lq45` / `idx30` / `idx80` | `all` | must be known, unknown falls back to all | stock universe scope |
+| `--proxy` | proxy URL | empty | URL format checked by client path | route requests through proxy |
+| `--concurrency` | integer `>= 1` | `5` | strict integer | worker count |
+| `--rps` | float `> 0` | `25` | strict positive | global request throttling |
+| `--delay` | `min-max` milliseconds | `100-500` | `0 <= min <= max` | jitter between requests |
+| `--retry` | integer `>= 0` | `2` | strict integer | retry attempts per stock |
+| `--limit` | integer `>= 0` | `0` (all) | strict integer | process first N stocks |
+| `--resume` | no value | off | flag | continue incomplete run |
+
+What it writes:
+- `stocks`
+- `price_history`
+- `ratios_history`
+- `news`
+- `sentiment_history`
+- `scrape_runs`
+- `scrape_progress`
+
+Expected terminal progress:
+
+```text
+Fetch-All Configuration:
+  Database: output/run/stocks.db
+  Index: idx30
+  Concurrency: 2 workers
+  Rate limit: 10.0 req/sec
+  Delay: 100-150 ms
+Started run #1
+Pending: 3 stocks to process
+[1/3] ADRO - 8 APIs succeeded
+[2/3] ASII - 8 APIs succeeded
+[3/3] GOTO - 8 APIs succeeded
+=== Run #1 Completed ===
+```
+
+Variations:
+
+```bash
+# Deterministic mini run for test
+./bin/rubick msn fetch-all --index idx30 --limit 3 --db output/run/stocks.db --rps 10 --delay 100-150 --concurrency 2
+
+# Resume interrupted batch
+./bin/rubick msn fetch-all --index idx80 --db output/prod/stocks.db --resume
+```
+
+### `news`
+
+Search Brave News and extract full article text from each result.
+
+```bash
+./bin/rubick news "IHSG stock market" --from 2026-03-01 --to 2026-03-05 --count 20 --concurrency 10 --output output/news.json
+```
+
+Arguments:
+
+| Flag | Expected Value | Default | Validation | What It Does |
+|---|---|---|---|---|
+| positional `<query>` | free-text query | required | non-empty | base search query |
+| `--from` | `YYYY-MM-DD` | now - 7d | valid date | start date |
+| `--to` | `YYYY-MM-DD` | today | valid date and `from <= to` | end date |
+| `--count` | integer `>= 1` | `20` | strict integer | result count requested |
+| `--concurrency` | integer `>= 1` | `10` | strict integer | worker count for fetch/extract |
+| `--output`, `-o` | json path | `output_YYYYMMDD.json` | writable path | final enriched output |
+| `--stock` | no value | off | flag | transforms comma terms into IDX-centric boolean query |
+
+Environment:
+
+| Variable | Required | Used By | Purpose |
+|---|---|---|---|
+| `BRAVE_API_KEY` | yes for `news` | Go brave client | authenticate Brave Search API calls |
+
+Expected output sample:
+
+```json
+{
+  "query": "IHSG",
+  "generated_at": "2026-03-06T02:39:47Z",
+  "results": [
+    {
+      "title": "...",
+      "url": "https://...",
+      "description": "...",
+      "page_age": "1d",
+      "text": "full extracted body text ...",
+      "fetch_status": "ok",
+      "extract_status": "ok"
+    }
+  ]
+}
+```
+
+Variations:
+
+```bash
+# Plain query
+./bin/rubick news IHSG --count 5 -o output/run/news_plain.json
+
+# Stock-mode query builder
+./bin/rubick news "BBCA,Bank Central Asia" --stock --from 2026-03-01 --to 2026-03-05 --count 10 -o output/run/news_stock.json
+
+# Lower concurrency for constrained hosts
+./bin/rubick news IHSG --count 10 --concurrency 2
+```
+
+### `export dashboard`
+
+Create dashboard-oriented workbook from SQLite.
+
+```bash
+./bin/rubick export dashboard --db output/stocks.db --output output/dashboard.xlsx
+```
+
+Arguments:
+
+| Flag | Expected Value | Required | Purpose |
+|---|---|---|---|
+| `--db` | SQLite path | yes | source database |
+| `--output` | `.xlsx` path | yes | generated workbook |
+
+### `export history`
+
+Create history-oriented workbook from SQLite.
+
+```bash
+./bin/rubick export history --db output/stocks.db --output output/history.xlsx
+```
+
+Arguments:
+
+| Flag | Expected Value | Required | Purpose |
+|---|---|---|---|
+| `--db` | SQLite path | yes | source database |
+| `--output` | `.xlsx` path | yes | generated workbook |
+
+### `export simple`
+
+Lightweight table export for automation and quick inspection.
+
+```bash
+./bin/rubick export simple --db output/stocks.db --format csv --output output/simple_csv
+```
+
+Arguments:
+
+| Flag | Expected Value | Required | What It Does |
+|---|---|---|---|
+| `--db` | SQLite path | yes | source DB |
+| `--format` | `json` / `csv` / `xlsx` | yes | output encoding |
+| `--output`, `-o` | directory (json/csv) or file (xlsx) | yes | destination |
+| `--tables` | comma-separated table list | no | export subset |
+
+Default table set:
+- `stocks`
+- `price_history`
+- `ratios_history`
+- `news`
+- `sentiment_history`
+- `scrape_runs`
+- `scrape_progress`
+
+Variations:
+
+```bash
+# JSON folder export
+./bin/rubick export simple --db output/run/stocks.db --format json --output output/run/simple_json
+
+# XLSX single workbook
+./bin/rubick export simple --db output/run/stocks.db --format xlsx --output output/run/simple.xlsx
+
+# Table subset
+./bin/rubick export simple --db output/run/stocks.db --format csv --tables stocks,news --output output/run/simple_subset
+```
+
+### `extractor` (advanced)
+
+Run Python extractor server directly for debugging/local integration.
+
+```bash
+./bin/rubick extractor --socket /tmp/extractor.sock
+```
+
+Arguments:
+
+| Flag | Expected Value | Required | Purpose |
+|---|---|---|---|
+| `--socket` | unix socket path | yes | bind location |
+
+## Deterministic End-to-End Run (Timestamped)
+
+Use this for repeatable smoke validation and artifact capture:
+
+```bash
+TS=$(date +%Y%m%d-%H%M%S)
+mkdir -p output/$TS
+
+# 1) Small stock ingestion
+./bin/rubick msn fetch-all --index idx30 --limit 3 --db output/$TS/stocks.db --rps 10 --delay 100-150 --concurrency 2
+
+# 2) News extraction
+./bin/rubick news IHSG --from 2026-03-01 --to 2026-03-05 --count 2 --concurrency 2 --output output/$TS/news.json
+
+# 3) Exports
+./bin/rubick export simple --db output/$TS/stocks.db --format json --output output/$TS/simple_json
+./bin/rubick export simple --db output/$TS/stocks.db --format csv --output output/$TS/simple_csv
+./bin/rubick export simple --db output/$TS/stocks.db --format xlsx --output output/$TS/simple.xlsx
+./bin/rubick export dashboard --db output/$TS/stocks.db --output output/$TS/dashboard.xlsx
+./bin/rubick export history --db output/$TS/stocks.db --output output/$TS/history.xlsx
+```
+
+## Testing Strategy
+
+### Go tests
+
+```bash
+# Full suite (includes tests/go)
+go test -v ./...
+
+# Focused CLI regression + live tests
+go test -v ./tests/go
+```
+
+For CI and local repeatability, prefer:
+
+```bash
+make test
+```
+
+### Live network e2e tests
+
+```bash
+set -a; source .env; set +a
+RUN_LIVE_E2E=1 go test -v ./tests/go -run TestLive
+```
+
+Behavior note:
+- tests treat transient network errors (DNS timeout, temporary connectivity, 429) as skippable for live-only coverage.
+
+### Python tests
+
+```bash
+uv run python -m unittest discover -s tests -p 'test_*.py'
+```
+
+
+## Release Bundle
+
+Create a distributable artifact that includes:
+- compiled `rubick` binary,
+- `extractor.py`,
+- Python export scripts (`scripts/export_*.py`),
+- `pyproject.toml` and `uv.lock`,
+- `.env.example`, `README.md`, and install instructions.
+
+```bash
+# auto version from git tag/commit
+make release
+
+# explicit version
+VERSION=v1.0.0 make release
+```
+
+Output artifacts:
+- `dist/rubick_<version>_<os>_<arch>/`
+- `dist/rubick_<version>_<os>_<arch>.tar.gz`
+- `dist/rubick_<version>_<os>_<arch>.zip`
+
+Bundle runtime setup:
+
+```bash
+cd dist/rubick_<version>_<os>_<arch>
+uv sync --frozen
+./bin/rubick --help
+```
+
+## Build and Run Profiles
+
+| Profile | Command | When To Use |
+|---|---|---|
+| Local developer iteration | `go run ./cmd/rubick ...` | rapid code changes before rebuilding |
+| Normal local/CI usage | `./bin/rubick ...` | default path for scripts and tests |
+| Release artifact | `go build -o bin/rubick ./cmd/rubick` | reproducible deployable binary |
+
+## Automation Targets
+
+`Makefile` commands:
+
+| Target | Action |
+|---|---|
+| `make setup` | install Go + Python dependencies |
+| `make build` | compile `bin/rubick` |
+| `make run ARGS='...'` | run compiled binary with arguments |
+| `make test` | run Go + Python tests |
+| `make test-live` | run live e2e tests with `.env` |
+| `make e2e` | run deterministic timestamped end-to-end workflow |
+
+## Operational Characteristics
+
+### Performance
+
+- `msn fetch-all` throughput controlled by:
+  - worker count (`--concurrency`),
+  - global RPS (`--rps`),
+  - jitter (`--delay`),
+  - retries (`--retry`).
+- `news` throughput controlled by:
+  - Brave result count (`--count`),
+  - concurrent fetch/extract workers (`--concurrency`).
+
+Tuning guidance:
+- start conservative (`--concurrency 2`, `--rps 10`) and increase gradually,
+- use lower concurrency on unstable networks,
+- avoid high parallelism if extractor host is resource-constrained.
+
+### Reliability
+
+- resumable runs via `scrape_runs` and `scrape_progress`,
+- no socket-path collision due to per-process unique socket names,
+- pooled socket self-healing on I/O error,
+- explicit process-group shutdown for extractor.
+
+### Failure Modes and Recovery
+
+| Symptom | Likely Cause | Recovery |
+|---|---|---|
+| `failed to search` in `news` | missing/invalid `BRAVE_API_KEY` or API/network issue | check `.env`, retry with smaller `--count` |
+| extractor startup timeout | Python env/deps not ready | run `uv sync`, retry command |
+| `failed to open database` | invalid DB path/permissions | use writable path under `output/` |
+| high fail count in `fetch-all` | API throttling/network instability | reduce `--concurrency`, reduce `--rps`, increase retries |
+
+## Technical Implementation Notes
+
+### Internal command model
+
+- `cmd/rubick/main.go` delegates to `internal/cli.Run(args)`.
+- each top-level command has dedicated handler logic.
+- help-path exit code is `0`; invalid usage and command errors return non-zero.
+
+### Data model and storage
+
+SQLite tables persist both point-in-time and historical views. Export scripts consume the same DB, which makes the workflow reproducible and scriptable.
+
+### Security and secret handling
+
+- never hardcode API keys,
+- keep `.env` out of source control,
+- rotate `BRAVE_API_KEY` periodically,
+- prefer environment injection in CI/CD rather than plaintext files.
+
+## Extending the Codebase
+
+### Add a new data source command
+
+1. Add a new handler in `internal/cli`.
+2. Register command routing in `Run(args)`.
+3. Keep source-specific logic in a dedicated package (similar to `msn/`).
+4. Add CLI regression tests in `tests/go`.
+5. Add docs + deterministic sample in README.
+
+### Add a new Python-assisted feature
+
+1. Put Python implementation in `scripts/` or standalone server file.
+2. Keep wire contract small and explicit if IPC is needed.
+3. Add input/output schema tests in `tests/`.
+4. Expose feature through one unified CLI command, not ad-hoc scripts.
+
+### Go-only vs Python-only decisions
+
+Use Go when:
+- you need high-concurrency network orchestration,
+- strong type-safety and binary distribution matter.
+
+Use Python when:
+- library ecosystem is materially better for the task,
+- rapid iteration of parsing/formatting logic is needed.
+
+Hybrid rule:
+- keep orchestration in Go,
+- isolate Python to specialized components with strict IPC contracts,
+- document protocol and lifecycle clearly.
+
+## Example Production-Like Workflow
+
+```bash
+TS=$(date +%Y%m%d-%H%M%S)
+BASE=output/$TS
+mkdir -p $BASE
+
+# Collect core market dataset
+./bin/rubick msn fetch-all --index idx80 --db $BASE/stocks.db --rps 15 --delay 150-400 --concurrency 4
+
+# Enrich with news for macro keyword
+./bin/rubick news "IHSG OR Jakarta Composite Index" --from 2026-03-01 --to 2026-03-06 --count 30 --concurrency 6 --output $BASE/news_macro.json
+
+# Export for analyst consumption
+./bin/rubick export dashboard --db $BASE/stocks.db --output $BASE/dashboard.xlsx
+./bin/rubick export history --db $BASE/stocks.db --output $BASE/history.xlsx
+./bin/rubick export simple --db $BASE/stocks.db --format csv --output $BASE/csv
+```
+
+## Glossary
+
+| Term | Meaning |
+|---|---|
+| MSN ID | internal identifier used by MSN endpoints |
+| Screener | preset query to select stocks by criteria |
+| Fetch-all run | bulk ingestion execution tracked in DB |
+| Extractor | Python process that converts raw HTML to article text |
+| Stock mode query | boolean query generated from comma-separated stock terms |
+
+## License / Internal Policy
+
+Add your repository license and internal data-usage policy here if this project is used in production or shared environments.