dernbu/lawnet-scraper
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

[1] Local runnable product shell #36

New issue
Closed
opened 2026-06-14 05:23:26 +02:00 by dernbu · 2 comments
dernbu commented 2026-06-14 05:23:26 +02:00
Owner
Copy link

Objective

Make the complete application stack boot locally, even before ingestion behavior is complete.

This is a vertical product foundation issue: after it is done, contributors should be able to start the system, reach the web app, confirm database readiness, and run later ingestion work inside the same shape of deployment that production will use.

Scope

  • Add the Docker Compose stack for PostgreSQL, Flyway, data-ingest scheduler, data-ingest worker, and web.
  • Add service directories, package roots, Dockerfiles, startup wiring, .env.example, ignore rules, and Justfile commands.
  • Add PostgreSQL and Flyway migration plumbing.
  • Add minimal data-ingest service entrypoints that connect to the database but do not yet scrape live sources.
  • Add minimal web service with a health endpoint and placeholder UI.
  • Configure internal networking so only web publishes a host port.
  • Document local startup and teardown commands.

Out of scope

  • LawNet or eLitigation scraping.
  • Production deployment workflow.
  • Full schema for source records, artifacts, search, or operations.

Done when

  • just up or equivalent starts the full local Compose stack.
  • PostgreSQL becomes healthy and Flyway runs successfully.
  • The web health endpoint responds from the host.
  • Scheduler and worker containers can connect to PostgreSQL.
  • PostgreSQL, Flyway, worker, and scheduler publish no host ports.
  • A clean checkout has enough documented setup to reproduce the stack.
## Objective Make the complete application stack boot locally, even before ingestion behavior is complete. This is a vertical product foundation issue: after it is done, contributors should be able to start the system, reach the web app, confirm database readiness, and run later ingestion work inside the same shape of deployment that production will use. ## Scope - Add the Docker Compose stack for PostgreSQL, Flyway, data-ingest scheduler, data-ingest worker, and web. - Add service directories, package roots, Dockerfiles, startup wiring, `.env.example`, ignore rules, and `Justfile` commands. - Add PostgreSQL and Flyway migration plumbing. - Add minimal data-ingest service entrypoints that connect to the database but do not yet scrape live sources. - Add minimal web service with a health endpoint and placeholder UI. - Configure internal networking so only web publishes a host port. - Document local startup and teardown commands. ## Out of scope - LawNet or eLitigation scraping. - Production deployment workflow. - Full schema for source records, artifacts, search, or operations. ## Done when - `just up` or equivalent starts the full local Compose stack. - PostgreSQL becomes healthy and Flyway runs successfully. - The web health endpoint responds from the host. - Scheduler and worker containers can connect to PostgreSQL. - PostgreSQL, Flyway, worker, and scheduler publish no host ports. - A clean checkout has enough documented setup to reproduce the stack.
dernbu commented 2026-06-14 05:52:06 +02:00
Author
Owner
Copy link

Implementation plan from scaffold audit

This plan covers the remaining gaps for completing #36: a local runnable product shell where the full Compose stack starts, Flyway runs, the web service is reachable and healthy, and scheduler/worker placeholders prove database connectivity.

Current scaffold status

  • Docker Compose already defines postgres, flyway, data-ingest-scheduler, data-ingest-worker, and web.
  • PostgreSQL uses a named Docker volume and healthcheck.
  • Flyway runs from services/flyway/sql after PostgreSQL is healthy.
  • Only web publishes a host port in the rendered Compose config.
  • Service package roots, Dockerfiles, .env.example, .gitignore, and Justfile exist.
  • Web has a placeholder UI and a health response implementation.
  • Existing lightweight checks pass:
    • docker compose --env-file .env.example config
    • uv run pytest in services/data-ingest
    • pnpm test in services/web

Functional gaps to close

  • Compose healthcheck probes http://127.0.0.1:3000/health, but the implemented route is /api/health. This likely leaves the web container unhealthy even when the app is serving correctly.
  • The issue requires scheduler and worker containers to connect to PostgreSQL, but lawnet-ingest currently only prints Hello world and has no database dependency or connection check.
  • data-ingest-scheduler and data-ingest-worker commands pass scheduler/worker arguments, but the CLI ignores arguments, so there is no role-specific startup behavior or validation.
  • The README says the web service is available at 127.0.0.1:3406, while .env.example sets WEB_HOST_PORT=12345. Local startup docs are therefore not reproducible as written.
  • The web health endpoint only returns a static JSON response. For #36, this may be enough for web-process health, but it does not confirm database readiness unless explicitly added.
  • The Compose stack has not yet been proven end-to-end with just up in this audit. Static config and unit tests pass, but the done criteria require runtime verification.
  • Generated/build artifacts and caches are present in the working tree view (.output, .venv, __pycache__, .pytest_cache, node_modules). They are ignored, but the repo should be checked before publishing to ensure none are tracked.

Implementation checklist

  1. Align the web health route and Compose healthcheck.

    • Choose one canonical host-visible health URL for the local stack.
    • Minimal option: change Compose healthcheck to probe /api/health.
    • Alternative option: add a /health route that delegates to the same health handler and keep Compose unchanged.
    • Update README verification instructions to use the same endpoint.
    • Add or adjust tests so the route contract is explicit.

  2. Add data-ingest runtime configuration.

    • Add a small config module that reads DATABASE_URL from the environment.
    • Fail fast with a clear error if DATABASE_URL is missing or malformed enough to prevent connection.
    • Keep the configuration minimal for #36; source-specific settings belong to later ingestion issues.

  3. Add PostgreSQL connectivity to data-ingest placeholders.

    • Add a PostgreSQL client dependency such as psycopg[binary] or psycopg plus the required runtime package.
    • Implement a reusable database probe that opens a connection to DATABASE_URL and executes a trivial query such as SELECT 1.
    • Close connections cleanly and surface connection errors with useful logs.
    • Keep this as a readiness/startup probe, not a queue or ingestion implementation.

  4. Implement role-aware data-ingest CLI commands.

    • Support lawnet-ingest scheduler and lawnet-ingest worker explicitly.
    • For each role, load config, connect to PostgreSQL, run SELECT 1, log success, and exit successfully.
    • Return non-zero for unknown commands.
    • Preserve a simple --help path.
    • Avoid long-running scheduler/worker logic in Python for this issue; Compose can continue using && exec sleep infinity to keep placeholders alive after the successful probe.

  5. Add data-ingest tests.

    • Unit-test missing DATABASE_URL handling.
    • Unit-test argument parsing for scheduler, worker, and invalid commands.
    • Unit-test that each role invokes the database probe.
    • If feasible without Docker, mock the database connection at the module boundary.
    • Reserve real database integration tests for a Compose smoke path or later database issues unless cheap to add now.

  6. Fix local documentation drift.

    • Decide the local default host port, then make .env.example, README, and any smoke commands agree.
    • Prefer a stable documented local port distinct from staging/prod ports.
    • Document the exact health URL, for example http://127.0.0.1:<WEB_HOST_PORT>/api/health.
    • Document that just down preserves the database volume and docker compose down -v removes it.

  7. Strengthen Justfile local commands if needed.

    • Keep just up as the primary local startup command.
    • Add a just health or just smoke command if useful, using the configured host port from .env.
    • Ensure just migrate still works after any Flyway command changes.
    • Avoid requiring live source access.

  8. Verify Compose networking and host exposure.

    • Re-run docker compose --env-file .env.example config and confirm only web has ports:.
    • Confirm postgres, flyway, data-ingest-scheduler, and data-ingest-worker remain on internal/backend networks without host publication.
    • Confirm the worker still has outbound access through the non-internal egress network.
    • Confirm production-deployment-specific settings do not leak into local defaults.

  9. Run the full local stack smoke test.

    • From a clean enough local checkout, run cp .env.example .env if needed.
    • Run just up.
    • Confirm PostgreSQL becomes healthy.
    • Confirm Flyway completes successfully and records the bootstrap migration.
    • Confirm data-ingest-scheduler and data-ingest-worker logs show successful PostgreSQL probes.
    • Confirm the web container is healthy.
    • Confirm the host health endpoint returns HTTP 200 and {"status":"ok"}.
    • Confirm the placeholder UI loads from the host.
    • Run just down after verification.

  10. Update CI coverage for the scaffold.

  • Keep current lint/typecheck/test jobs.
  • Keep Compose config validation with required env vars.
  • If runtime time is acceptable, add a Compose smoke job that builds the stack, starts it, waits for health, checks the host health endpoint, prints logs on failure, and tears down volumes.
  • If full smoke CI is deferred to #45, document that #36 was manually smoke-tested and keep CI to static/unit checks.
  1. Confirm repository hygiene before PR/update.
  • Check JJ status before and after edits.
  • Ensure ignored artifacts remain untracked.
  • Do not publish local-only l bookmark changes.
  • Keep implementation changes in the feature commit beneath l per repository workflow.

Acceptance verification commands

Recommended local commands after implementation:

docker compose --env-file .env.example config
just lint
just typecheck
just test
just up
curl -fsS http://127.0.0.1:${WEB_HOST_PORT}/api/health
just logs data-ingest-scheduler
just logs data-ingest-worker
just down

If WEB_HOST_PORT is not exported in the shell, use the concrete value from .env.example.

Done criteria mapping

  • just up starts the full local stack: verify with docker compose ps and host web request.
  • PostgreSQL healthy and Flyway successful: verify Compose health/status and Flyway logs/table.
  • Web health endpoint responds from host: verify with curl against the documented health URL.
  • Scheduler and worker can connect to PostgreSQL: verify role-specific startup logs and non-zero failure behavior when DB is unavailable.
  • Non-web services publish no host ports: verify rendered Compose config.
  • Clean checkout is reproducible: verify README, .env.example, and Justfile agree and include teardown/log commands.

Non-goals for this issue

  • No LawNet or eLitigation scraping.
  • No durable queue/scheduler implementation.
  • No full source record/artifact/search schema.
  • No production deployment workflow work unless it directly blocks the local shell.
## Implementation plan from scaffold audit This plan covers the remaining gaps for completing #36: a local runnable product shell where the full Compose stack starts, Flyway runs, the web service is reachable and healthy, and scheduler/worker placeholders prove database connectivity. ### Current scaffold status - Docker Compose already defines `postgres`, `flyway`, `data-ingest-scheduler`, `data-ingest-worker`, and `web`. - PostgreSQL uses a named Docker volume and healthcheck. - Flyway runs from `services/flyway/sql` after PostgreSQL is healthy. - Only `web` publishes a host port in the rendered Compose config. - Service package roots, Dockerfiles, `.env.example`, `.gitignore`, and `Justfile` exist. - Web has a placeholder UI and a health response implementation. - Existing lightweight checks pass: - `docker compose --env-file .env.example config` - `uv run pytest` in `services/data-ingest` - `pnpm test` in `services/web` ### Functional gaps to close - Compose healthcheck probes `http://127.0.0.1:3000/health`, but the implemented route is `/api/health`. This likely leaves the web container unhealthy even when the app is serving correctly. - The issue requires scheduler and worker containers to connect to PostgreSQL, but `lawnet-ingest` currently only prints `Hello world` and has no database dependency or connection check. - `data-ingest-scheduler` and `data-ingest-worker` commands pass `scheduler`/`worker` arguments, but the CLI ignores arguments, so there is no role-specific startup behavior or validation. - The README says the web service is available at `127.0.0.1:3406`, while `.env.example` sets `WEB_HOST_PORT=12345`. Local startup docs are therefore not reproducible as written. - The web health endpoint only returns a static JSON response. For #36, this may be enough for web-process health, but it does not confirm database readiness unless explicitly added. - The Compose stack has not yet been proven end-to-end with `just up` in this audit. Static config and unit tests pass, but the done criteria require runtime verification. - Generated/build artifacts and caches are present in the working tree view (`.output`, `.venv`, `__pycache__`, `.pytest_cache`, `node_modules`). They are ignored, but the repo should be checked before publishing to ensure none are tracked. ### Implementation checklist 1. Align the web health route and Compose healthcheck. - Choose one canonical host-visible health URL for the local stack. - Minimal option: change Compose healthcheck to probe `/api/health`. - Alternative option: add a `/health` route that delegates to the same health handler and keep Compose unchanged. - Update README verification instructions to use the same endpoint. - Add or adjust tests so the route contract is explicit. 2. Add data-ingest runtime configuration. - Add a small config module that reads `DATABASE_URL` from the environment. - Fail fast with a clear error if `DATABASE_URL` is missing or malformed enough to prevent connection. - Keep the configuration minimal for #36; source-specific settings belong to later ingestion issues. 3. Add PostgreSQL connectivity to data-ingest placeholders. - Add a PostgreSQL client dependency such as `psycopg[binary]` or `psycopg` plus the required runtime package. - Implement a reusable database probe that opens a connection to `DATABASE_URL` and executes a trivial query such as `SELECT 1`. - Close connections cleanly and surface connection errors with useful logs. - Keep this as a readiness/startup probe, not a queue or ingestion implementation. 4. Implement role-aware data-ingest CLI commands. - Support `lawnet-ingest scheduler` and `lawnet-ingest worker` explicitly. - For each role, load config, connect to PostgreSQL, run `SELECT 1`, log success, and exit successfully. - Return non-zero for unknown commands. - Preserve a simple `--help` path. - Avoid long-running scheduler/worker logic in Python for this issue; Compose can continue using `&& exec sleep infinity` to keep placeholders alive after the successful probe. 5. Add data-ingest tests. - Unit-test missing `DATABASE_URL` handling. - Unit-test argument parsing for `scheduler`, `worker`, and invalid commands. - Unit-test that each role invokes the database probe. - If feasible without Docker, mock the database connection at the module boundary. - Reserve real database integration tests for a Compose smoke path or later database issues unless cheap to add now. 6. Fix local documentation drift. - Decide the local default host port, then make `.env.example`, README, and any smoke commands agree. - Prefer a stable documented local port distinct from staging/prod ports. - Document the exact health URL, for example `http://127.0.0.1:<WEB_HOST_PORT>/api/health`. - Document that `just down` preserves the database volume and `docker compose down -v` removes it. 7. Strengthen `Justfile` local commands if needed. - Keep `just up` as the primary local startup command. - Add a `just health` or `just smoke` command if useful, using the configured host port from `.env`. - Ensure `just migrate` still works after any Flyway command changes. - Avoid requiring live source access. 8. Verify Compose networking and host exposure. - Re-run `docker compose --env-file .env.example config` and confirm only `web` has `ports:`. - Confirm `postgres`, `flyway`, `data-ingest-scheduler`, and `data-ingest-worker` remain on internal/backend networks without host publication. - Confirm the worker still has outbound access through the non-internal `egress` network. - Confirm production-deployment-specific settings do not leak into local defaults. 9. Run the full local stack smoke test. - From a clean enough local checkout, run `cp .env.example .env` if needed. - Run `just up`. - Confirm PostgreSQL becomes healthy. - Confirm Flyway completes successfully and records the bootstrap migration. - Confirm `data-ingest-scheduler` and `data-ingest-worker` logs show successful PostgreSQL probes. - Confirm the web container is healthy. - Confirm the host health endpoint returns HTTP 200 and `{"status":"ok"}`. - Confirm the placeholder UI loads from the host. - Run `just down` after verification. 10. Update CI coverage for the scaffold. - Keep current lint/typecheck/test jobs. - Keep Compose config validation with required env vars. - If runtime time is acceptable, add a Compose smoke job that builds the stack, starts it, waits for health, checks the host health endpoint, prints logs on failure, and tears down volumes. - If full smoke CI is deferred to #45, document that #36 was manually smoke-tested and keep CI to static/unit checks. 11. Confirm repository hygiene before PR/update. - Check JJ status before and after edits. - Ensure ignored artifacts remain untracked. - Do not publish local-only `l` bookmark changes. - Keep implementation changes in the feature commit beneath `l` per repository workflow. ### Acceptance verification commands Recommended local commands after implementation: ```sh docker compose --env-file .env.example config just lint just typecheck just test just up curl -fsS http://127.0.0.1:${WEB_HOST_PORT}/api/health just logs data-ingest-scheduler just logs data-ingest-worker just down ``` If `WEB_HOST_PORT` is not exported in the shell, use the concrete value from `.env.example`. ### Done criteria mapping - `just up` starts the full local stack: verify with `docker compose ps` and host web request. - PostgreSQL healthy and Flyway successful: verify Compose health/status and Flyway logs/table. - Web health endpoint responds from host: verify with `curl` against the documented health URL. - Scheduler and worker can connect to PostgreSQL: verify role-specific startup logs and non-zero failure behavior when DB is unavailable. - Non-web services publish no host ports: verify rendered Compose config. - Clean checkout is reproducible: verify README, `.env.example`, and `Justfile` agree and include teardown/log commands. ### Non-goals for this issue - No LawNet or eLitigation scraping. - No durable queue/scheduler implementation. - No full source record/artifact/search schema. - No production deployment workflow work unless it directly blocks the local shell.
dernbu commented 2026-06-14 06:04:24 +02:00
Author
Owner
Copy link

Review Findings

Thorough review of the Phase 1 implementation uncovered 3 critical bugs and several improvements. PR: #46

Critical Bugs Found

  1. PostgreSQL volume path (compose.yaml:22): Mounted at /var/lib/postgresql instead of /var/lib/postgresql/data. The PostgreSQL 18 image stores data in the data subdirectory — without this fix, data would not persist across container restarts.

  2. Web health check URL mismatch (compose.yaml:90): Health check fetches /health but the TanStack Start route is defined at /api/health (routes/api/health.ts). This would cause the container to fail health checks and trigger continuous restarts.

  3. COMPOSE_PROJECT_NAME from env file doesn't take effect (Justfile:58): Docker Compose's --env-file only performs variable substitution — it does not set COMPOSE_PROJECT_NAME as an environment variable for compose's own config. Without this fix, production and staging deployments would use the same project name from the compose file's name: field, sharing containers, networks, and volumes.

Improvements

  • Data-ingest CLI now has proper scheduler/worker subcommands with PostgreSQL connectivity check (psycopg)
  • Removed unnecessary web-edge network (port binds to loopback)
  • .env.example port aligned with README (3406)

Verified

  • just test — all 6 tests pass
  • just lint — ruff + eslint pass
  • just typecheck — ty + tsc pass
  • just check-compose — valid compose config
## Review Findings Thorough review of the Phase 1 implementation uncovered **3 critical bugs** and several improvements. PR: https://git.yongbeom.com/dernbu/lawnet-scraper/pulls/46 ### Critical Bugs Found 1. **PostgreSQL volume path** (`compose.yaml:22`): Mounted at `/var/lib/postgresql` instead of `/var/lib/postgresql/data`. The PostgreSQL 18 image stores data in the `data` subdirectory — without this fix, data would not persist across container restarts. 2. **Web health check URL mismatch** (`compose.yaml:90`): Health check fetches `/health` but the TanStack Start route is defined at `/api/health` (`routes/api/health.ts`). This would cause the container to fail health checks and trigger continuous restarts. 3. **`COMPOSE_PROJECT_NAME` from env file doesn't take effect** (`Justfile:58`): Docker Compose's `--env-file` only performs variable substitution — it does not set `COMPOSE_PROJECT_NAME` as an environment variable for compose's own config. Without this fix, production and staging deployments would use the same project name from the compose file's `name:` field, sharing containers, networks, and volumes. ### Improvements - Data-ingest CLI now has proper `scheduler`/`worker` subcommands with PostgreSQL connectivity check (psycopg) - Removed unnecessary `web-edge` network (port binds to loopback) - `.env.example` port aligned with README (3406) ### Verified - `just test` — all 6 tests pass - `just lint` — ruff + eslint pass - `just typecheck` — ty + tsc pass - `just check-compose` — valid compose config
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
temp
feat/53-rename-lawnet-routes
push-xrsnvuvuvsul
feat/37-bff-storage-frontend
l
feat/49-pg-bff-storage
push-zzvxlnkylqsp
feat/quick-and-dirty
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dernbu/lawnet-scraper#36
No description provided.