Getting started

This project uses uv for dependency and environment management.

Install and generate a dataset

uv sync
uv run datagen generate --seed 42 --customers 200 --with-ground-truth --output-dir data

This writes:

data/
  erp/customers.csv
  erp/invoices.csv
  erp/payments.csv
  crm/customers.json
  ground_truth.json        # only with --with-ground-truth
  generation_config.yaml   # the exact config used, for reproducibility/audit

Re-running with the same --seed and other options always reproduces byte-identical output -- every source of randomness, including internal correlation UUIDs, derives from a single master seed.

CLI reference

uv run datagen --help
uv run datagen generate --help
uv run datagen show-config          # print the fully-resolved config as YAML

See the CLI reference for the full flag list, and the data model reference for how ERP and CRM field names/formats diverge.

Using a config file

uv run datagen show-config > my-config.yaml   # start from the resolved defaults
uv run datagen generate --config my-config.yaml --output-dir data

Any generate flag also passed on the command line overrides the corresponding value from --config.

A worked example

See examples/sample_dataset in the repository for a small, committed dataset (8 customers, --seed 1) you can inspect without running anything.

Reconcile the data

Once you have a generated dataset, run the reconciliation pipeline over it:

uv run reconcile data

This ingests data/erp/*.csv and data/crm/customers.json, cleans and validates every row, resolves ERP records against CRM records, and writes:

data/processed/crosswalk.json        # upserted on every run, never overwritten
data/processed/quarantine_log.json   # every row that failed cleaning/validation, or tied on resolution
RECONCILIATION_REPORT.md             # match rate, quarantine breakdown, orphans, payment drift

Re-running reconcile on the same input is idempotent: existing crosswalk entries are kept and only genuinely new ones are appended.

If you generated the dataset --with-ground-truth, score the pipeline's matcher against it (precision/recall/F1) with:

uv run python -m eval.score data --threshold 0.75

eval/score.py is the only place ground_truth.json is ever read -- reconcile itself never consults it. See the CLI reference for the full flag list, and the data model reference for the pipeline's clean-record schemas, crosswalk, and quarantine format.

Load into Postgres (optional)

The JSON crosswalk/quarantine log remain the source of truth reconcile reads back on every run; --postgres-dsn additionally loads the same run into a Postgres target schema (see pipeline.postgres):

docker compose up -d postgres
uv run reconcile data --postgres-dsn postgresql://reconcile:reconcile@localhost:5433/reconcile

This creates the customers / invoices / payments / crosswalk / quarantine_log tables if they don't already exist and loads the run with the same idempotency semantics as the JSON files: crosswalk upserts without reassigning an existing canonical_id, and quarantine_log is replaced in full each run.

Development

uv sync --all-groups
uv run pytest              # 113 tests, ~95% coverage (datagen + pipeline);
                            # Postgres tests skip unless TEST_DATABASE_URL is set
uv run ruff check .        # lint
uv run ruff format .       # format
uv run mypy                # strict type checking
uv run pre-commit install  # run the above automatically on every commit

Local development network note

If your network blocks direct access to files.pythonhosted.org and requires routing package downloads through an internal PyPI-compatible mirror, copy uv.toml.example (in the repository root) to uv.toml (gitignored) and point it at your mirror. uv.lock is likewise gitignored so a mirror-specific lock never leaks into the shared repo -- each environment resolves its own lock from pyproject.toml.

Because uv.toml is gitignored, git worktree doesn't share it: only Git-tracked files are common across worktrees, so copy it into each worktree of this repo that needs it.