# Data Catalog — UI_INDEX

Last updated: 2026-06-15

## Portfolio Pages (Static Site, GitHub Pages)

| File | Purpose |
|------|---------|
| `index.html` | The audit — 5 chapters, charts, investigative takes, cross-index severity banner, lawmaker salary contrast |
| `political.html` | Political layer — four-view FEC comparison matrix, Trone profile, lawmaker deep-dive, methodology disclosure |
| `methodology.html` | Data transparency — live-source validation table, formulas, housing cost methodology, limitations |
| `about.html` | Author bio, fractional CDO offer, hire-me CTAs |
| `OPTIMIZATIONS.md` | Running correction/optimization log (single source of truth for what changed) |

**Figure numbering convention:** Slots `01–07` = index charts, `11–13` = FEC charts. Slots `08–10` are now in use: `08` = Real Value Index (RVI/inflation cross-check), `09` = Unemployment Context, `10` = Federal Spending Accountability. New political/FEC charts start at `14`.

## Input Data (Committed)

| File | Type | Source | Description |
|------|------|--------|-------------|
| `data/dmv_macro_baselines.csv` | CSV | BLS QCEW, State DOL statutes, Census | Base input: wages, housing costs, benefit caps, taxable wage bases by jurisdiction and year (2010, 2018, 2026) |

## Generated Data (JSON, All Wrapped with `_metadata`)

| File | Generated By | Cycle | Records | Description | Status |
|------|-------------|-------|---------|-------------|--------|
| `data/political/fec_funding_profiles.json` | `fec_integration_v251d.py` | 2024 | ~24 | Cycle-filtered funding profiles with corruption flags (multi_cycle_bleed, committee_transfer, excessive_self_funding, vendor_masquerading) | ✅ Production |
| `data/political/fec_excluded_self_funding.json` | `fec_integration_v251d.py` | 2024 | 219 | Excluded contributions (self-funding, multi-cycle, transfers) that appeared in 2024 Schedule A but were removed from categorization | ✅ Investigative |
| `data/political/fec_quick_results.json` | `fec_quick_test.py` | N/A (multi-cycle) | 3 | Diagnostic connectivity test — NO cycle filter. Returns all-time totals. NOT a data source for analysis. | ⚠️ Diagnostic only |
| `data/political/employer_contribution_gap.json` | `employer_contribution_gap.py` | 2026 | 3 | Per-state gap between frozen statutory SUI wage base and expected wage-indexed base | ✅ Calculated |
| `data/political/political_layer_report.json` | `political_layer_builder.py` | 2024 | 24 | Congress.gov members enriched with Census ACS median income and committee assignments | ✅ Enriched |
| `data/political/fec_audit_log.json` | `api_client.py` | N/A | Variable | All API calls with timestamps, status, URLs, and cache hit/miss | ✅ Audit |

## Missing Data (Not Yet Generated)

| File | Would Be Generated By | Description | Blocker |
|------|----------------------|-------------|---------|
| `data/political/fec_funding_profiles_raw.json` | `fec_integration_raw_investigative.py` | Multi-cycle (all-time) FEC data for delta comparison | Requires API key + runtime |
| `data/political/corruption_delta_analysis.json` | `delta_analyzer.py` | Cross-cycle discrepancy flags and investigative priority ranking | Requires raw profiles above |

## Data Lineage Map

```
dmv_macro_baselines.csv (input)
    ├── ui_index_engine.py → generate_figures.py → figures/01-04.png
    ├── employer_contribution_gap.py → generate_employer_gap_charts.py → figures/05-07.png

FEC API (live, cycle=2024)
    ├── fec_integration_v251d.py → data/political/fec_funding_profiles.json
    │                                      → data/political/fec_excluded_self_funding.json
    │                                      → generate_fec_charts.py → figures/11-13.png
    ├── fec_integration_raw_investigative.py → data/political/fec_funding_profiles_raw.json (NOT YET GENERATED)
    │                                          → delta_analyzer.py → data/political/corruption_delta_analysis.json (NOT YET GENERATED)
    └── fec_quick_test.py → data/political/fec_quick_results.json (diagnostic only)

Congress.gov API + Census ACS
    ├── political_layer_builder.py → data/political/political_layer_report.json
    │                                → political_layer_analyzer.py (analysis)

api_client.py (shared)
    └── data/political/fec_audit_log.json (all API calls)
```

## Metadata Standard

All JSON output files follow this `_metadata` block structure:

```json
{
  "_metadata": {
    "generated_by": "script_name.py",
    "generated_at": "2026-06-11T00:00:00Z",
    "cycle": 2024,
    "api_version": "v1",
    "reconciliation_status": "VALIDATED|INVESTIGATIVE|CALCULATED|ENRICHED|AUDIT",
    "purpose": "What this file is for",
    "methodology": "How the data was produced",
    "data_sources": {"source_name": "description"},
    "caveat": "Known limitations and approximations"
  },
  "data": [...]
}
```

**`reconciliation_status` enum definitions:**
| Value | Meaning |
|---|---|
| `VALIDATED` | Data fetched from live API, cross-checked, all records pass validation |
| `INVESTIGATIVE` | Data excluded from primary analysis but preserved for forensic review |
| `CALCULATED` | Derived from inputs via formula (no live API call) |
| `ENRICHED` | API data augmented with additional sources (e.g., Census join) |
| `AUDIT` | Diagnostic or logging artifact — not a data source for analysis |

**Required fields (all 8 above + these for API-sourced files):**
- `data_sources`: dict — maps each source name to its endpoint/description (enables reproducibility and lineage auditing)
- `fallback_applied`: bool — **required** when hardcoded values replace live API data; omit only for static-calculation scripts

**Optional fields:**
- `query_params`: dict — all parameters passed to the API
- `record_count`: int — number of records in `data` array

## Data Quality Checks

- JSON schema validation: `validate.yml` CI step on every push
- Cycle assertion: `generate_fec_charts.py` asserts `cycle == 2024` on all profiles
- Member count validation: `political_layer_builder.py` validates MD=10, VA=13, DC=1

## Freshness

| Data | Last Updated | Refresh Method |
|------|-------------|----------------|
| dmv_macro_baselines.csv | Manual | Manual (BLS annual) |
| fec_funding_profiles.json | 2026-06-11 | Manual script (API key required) |
| political_layer_report.json | 2026-06-11 | Manual script (API key required) |
| employer_contribution_gap.json | 2026-06-11 | Manual script (static calculation) |

## API Keys Required for Refresh

| Data Source | Key Variable | Get Key At | Stored In |
|-------------|-------------|------------|-----------|
| FEC API | `FEC_API_KEY` | https://api.open.fec.gov/developers/ | `.env` (local only) |
| Census API | `CENSUS_API_KEY` | https://api.census.gov/data/key_signup.html | `.env` (local only) |
| Congress.gov | `DEMO_KEY` (public) | N/A | Hardcoded (public access) |