dernbu/lawnet-scraper
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

[4/4] Wire repository services #15

Merged
dernbu merged 3 commits from feat/1-compose-wiring into main 2026-06-13 21:23:01 +02:00
Conversation 5 Commits 3 Files changed 6 +249 -29
dernbu commented 2026-06-13 19:44:30 +02:00
Owner
Copy link

Summary

  • add root Compose configuration for PostgreSQL, Flyway, scheduler, worker, and web
  • keep the Hello World scheduler/worker placeholders alive through Compose
  • probe PostgreSQL connectivity from both placeholders and TLS egress from the worker
  • enforce backend, egress, and web-edge network membership
  • publish only 127.0.0.1:3406:3000
  • add .env.example, Justfile workflows, README setup guidance, and Compose smoke assertions

Size

370 changed lines.

Verification

  • just config
  • just lint
  • just typecheck
  • just test

The smoke suite passed PostgreSQL/Flyway readiness, application health, exact network membership, port isolation, placeholder PostgreSQL access, and worker TLS egress to both source hosts.

Stack 4 of 4. Depends on and targets feat/1-web-scaffold; merge the preceding PRs first.

Resolves #1

## Summary - add root Compose configuration for PostgreSQL, Flyway, scheduler, worker, and web - keep the Hello World scheduler/worker placeholders alive through Compose - probe PostgreSQL connectivity from both placeholders and TLS egress from the worker - enforce `backend`, `egress`, and `web-edge` network membership - publish only `127.0.0.1:3406:3000` - add `.env.example`, Justfile workflows, README setup guidance, and Compose smoke assertions ## Size 370 changed lines. ## Verification - `just config` - `just lint` - `just typecheck` - `just test` The smoke suite passed PostgreSQL/Flyway readiness, application health, exact network membership, port isolation, placeholder PostgreSQL access, and worker TLS egress to both source hosts. Stack 4 of 4. Depends on and targets `feat/1-web-scaffold`; merge the preceding PRs first. Resolves #1
dernbu force-pushed feat/1-compose-wiring from 5a21455712 to e89ebadec5 2026-06-13 20:05:47 +02:00 Compare
dernbu changed target branch from feat/1-web-scaffold to main 2026-06-13 20:57:05 +02:00
dernbu reviewed 2026-06-13 21:02:24 +02:00
Justfile Outdated
@ -0,0 +1,68 @@
set dotenv-load := true
default:
dernbu commented 2026-06-13 20:58:44 +02:00
Author
Owner
Copy link

Rename recipd to help

Rename recipd to help
Justfile Outdated
@ -0,0 +51,4 @@
smoke:
./scripts/compose-smoke.sh
generate-db-types:
dernbu commented 2026-06-13 21:02:17 +02:00
Author
Owner
Copy link

If its not needed yet, dont implement it

If its not needed yet, dont implement it
compose.yaml Outdated
@ -0,0 +27,4 @@
image: flyway/flyway:12-alpine
command: migrate
environment:
FLYWAY_URL: jdbc:postgresql://postgres:5432/${POSTGRES_DB:-lawnet}
dernbu commented 2026-06-13 20:58:20 +02:00
Author
Owner
Copy link

No fsllback, fast fai, with :?

No fsllback, fast fai, with :?
dernbu force-pushed feat/1-compose-wiring from e89ebadec5 to 3e14875004 2026-06-13 21:07:44 +02:00 Compare
dernbu reviewed 2026-06-13 21:12:47 +02:00
Justfile Outdated
@ -0,0 +46,4 @@
config:
docker compose config
dernbu commented 2026-06-13 21:09:21 +02:00
Author
Owner
Copy link

Again, can you remove all file recipes that aren't needed yet? We'll add them when we need them. Don't add them first and then exit.

Again, can you remove all file recipes that aren't needed yet? We'll add them when we need them. Don't add them first and then exit.
compose.yaml Outdated
@ -0,0 +7,4 @@
POSTGRES_DB: ${POSTGRES_DB:-lawnet}
POSTGRES_USER: ${POSTGRES_USER:-lawnet}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-lawnet_dev}
healthcheck:
dernbu commented 2026-06-13 21:09:50 +02:00
Author
Owner
Copy link

We need a separate health check for the Postgres container because it doesn't have one built in. I thought it would include its own health check.

We need a separate health check for the Postgres container because it doesn't have one built in. I thought it would include its own health check.
dernbu marked this conversation as resolved
compose.yaml Outdated
@ -0,0 +27,4 @@
image: flyway/flyway:12-alpine
command: migrate
environment:
FLYWAY_URL: jdbc:postgresql://postgres:5432/${POSTGRES_DB:-lawnet}
dernbu commented 2026-06-13 21:10:19 +02:00
Author
Owner
Copy link

For this one, fail fast. If the environment variable is not available, use the column‑mark question‑mark syntax.

For this one, fail fast. If the environment variable is not available, use the column‑mark question‑mark syntax.
compose.yaml Outdated
@ -0,0 +40,4 @@
networks:
- backend
data-ingest-scheduler:
dernbu commented 2026-06-13 21:12:20 +02:00
Author
Owner
Copy link

We don't have the implementation for this scheduler. For any service without an implementation, we shouldn't guess the environment variables that follow. Let's trim the compose file as much as possible and remove any unused environment variables.

We don't have the implementation for this scheduler. For any service without an implementation, we shouldn't guess the environment variables that follow. Let's trim the compose file as much as possible and remove any unused environment variables.
dernbu marked this conversation as resolved
compose.yaml Outdated
@ -0,0 +58,4 @@
depends_on:
flyway:
condition: service_completed_successfully
healthcheck:
dernbu commented 2026-06-13 21:11:37 +02:00
Author
Owner
Copy link

We don't need a house check for the walkers.

We don't need a house check for the walkers.
dernbu marked this conversation as resolved
compose.yaml Outdated
@ -0,0 +85,4 @@
depends_on:
flyway:
condition: service_completed_successfully
healthcheck:
dernbu commented 2026-06-13 21:11:27 +02:00
Author
Owner
Copy link

We don't need a health check for the workers here.

We don't need a health check for the workers here.
dernbu marked this conversation as resolved
dernbu commented 2026-06-13 21:16:26 +02:00
Author
Owner
Copy link

Addressed all review comments in commit 8dcc3afb:

  • Justfile default → help — Renamed to help per request.
  • Justfile placeholder recipes — Removed generate-db-types, check-db-types, enqueue-discovery, requeue-dead. Will add when implementations exist.
  • compose.yaml Flyway FLYWAY_URL — Changed :-lawnet to :?POSTGRES_DB is required for fail-fast.
  • compose.yaml scheduler healthcheck — Removed (no implementation yet).
  • compose.yaml worker healthcheck — Removed (no implementation yet).
  • compose.yaml scheduler/worker environment — Replaced the shared env anchor with only DATABASE_URL:?. Removed the implementation-guessing vars (DISCOVERY_INTERVAL_HOURS, ARTIFACT_REFRESH_DAYS, WORKER_ID, JOB_LEASE_SECONDS, JOB_MAX_ATTEMPTS, LAWNET_REQUEST_INTERVAL_SECONDS, ELITIGATION_REQUEST_INTERVAL_SECONDS, HTTP_TIMEOUT_SECONDS). Will add back when the scheduler/worker are implemented.
  • .env.example — Trimmed to only vars currently consumed by compose.
Addressed all review comments in commit `8dcc3afb`: - **Justfile default → help** — Renamed to `help` per request. - **Justfile placeholder recipes** — Removed `generate-db-types`, `check-db-types`, `enqueue-discovery`, `requeue-dead`. Will add when implementations exist. - **compose.yaml Flyway `FLYWAY_URL`** — Changed `:-lawnet` to `:?POSTGRES_DB is required` for fail-fast. - **compose.yaml scheduler healthcheck** — Removed (no implementation yet). - **compose.yaml worker healthcheck** — Removed (no implementation yet). - **compose.yaml scheduler/worker environment** — Replaced the shared env anchor with only `DATABASE_URL:?`. Removed the implementation-guessing vars (`DISCOVERY_INTERVAL_HOURS`, `ARTIFACT_REFRESH_DAYS`, `WORKER_ID`, `JOB_LEASE_SECONDS`, `JOB_MAX_ATTEMPTS`, `LAWNET_REQUEST_INTERVAL_SECONDS`, `ELITIGATION_REQUEST_INTERVAL_SECONDS`, `HTTP_TIMEOUT_SECONDS`). Will add back when the scheduler/worker are implemented. - **.env.example** — Trimmed to only vars currently consumed by compose.
dernbu reviewed 2026-06-13 21:19:06 +02:00
Justfile Outdated
@ -0,0 +41,4 @@
cd services/web && fnm exec --using=22.22.3 pnpm format:check
typecheck:
cd services/data-ingest && uv run pyright
dernbu commented 2026-06-13 21:17:08 +02:00
Author
Owner
Copy link

Use ty for checks

Use ty for checks
compose.yaml Outdated
@ -0,0 +8,4 @@
POSTGRES_USER: ${POSTGRES_USER:-lawnet}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-lawnet_dev}
healthcheck:
test:
dernbu commented 2026-06-13 21:18:01 +02:00
Author
Owner
Copy link

Do we really need a separate health check script for the Postgres container? I expected Postgres would have its own health check.

Do we really need a separate health check script for the Postgres container? I expected Postgres would have its own health check.
dernbu marked this conversation as resolved
dernbu referenced this pull request from a commit 2026-06-13 21:21:27 +02:00
second round of PR #15 review fixes
dernbu commented 2026-06-13 21:21:39 +02:00
Author
Owner
Copy link

Second round addressed in commit 6775bcbd:

  • compose.yaml — Removed Postgres healthcheck (Flyway handles retries via FLYWAY_CONNECT_RETRIES=60). Also removed Flyway's depends_on postgres since Flyway retries on its own.
  • Justfile — Changed Python typecheck from uv run pyright to uv run ty. Recipe name stays typecheck.
  • pyproject.toml — Replaced pyright with ty in dev dependencies.
  • uv.lock — Regenerated with uv lock (removed pyright/nodeenv, added ty).
Second round addressed in commit `6775bcbd`: - **compose.yaml** — Removed Postgres healthcheck (Flyway handles retries via `FLYWAY_CONNECT_RETRIES=60`). Also removed Flyway's `depends_on` postgres since Flyway retries on its own. - **Justfile** — Changed Python typecheck from `uv run pyright` to `uv run ty`. Recipe name stays `typecheck`. - **pyproject.toml** — Replaced `pyright` with `ty` in dev dependencies. - **uv.lock** — Regenerated with `uv lock` (removed `pyright`/`nodeenv`, added `ty`).
dernbu referenced this pull request from a commit 2026-06-13 21:22:54 +02:00
second round of PR #15 review fixes
dernbu force-pushed feat/1-compose-wiring from 6775bcbdb6 to e7080eb24c 2026-06-13 21:22:54 +02:00 Compare
dernbu merged commit e7080eb24c into main 2026-06-13 21:23:01 +02:00
dernbu deleted branch feat/1-compose-wiring 2026-06-13 21:23:01 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dernbu/lawnet-scraper!15
No description provided.