dernbu/lawnet-scraper
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

fix: critical compose bugs, improve data-ingest CLI, and complete [1] Local runnable product shell #46

Merged
dernbu merged 1 commit from feat/10.1-continuous-deployment into main 2026-06-14 06:13:54 +02:00
Conversation 2 Commits 1 Files changed 7 +142 -12
dernbu commented 2026-06-14 06:04:09 +02:00
Owner
Copy link

Summary

Thorough review of the Phase 1 implementation uncovered three critical bugs and several improvements. All fixes pass the existing lint, typecheck, and test suites.

Critical fixes

  1. PostgreSQL volume path (compose.yaml:22): Mounted at /var/lib/postgresql instead of /var/lib/postgresql/data. The PostgreSQL image stores data in the data subdirectory; without this fix, data would not persist across container restarts.

  2. Web health check URL (compose.yaml:90): The health check fetched /health but the TanStack Start route is at /api/health. This would cause the container to report unhealthy and trigger continuous restarts.

  3. COMPOSE_PROJECT_NAME from env file (Justfile:58-62): Docker Compose's --env-file only performs variable substitution in the compose file; it does not set environment variables for compose config like COMPOSE_PROJECT_NAME. Without this fix, production and staging deployments would use the same project name (lawnet-scraper from the compose file's name: field), sharing containers, networks, and volumes. Fixed by extracting the project name from the env file and passing it via -p.

Improvements

  1. Data-ingest CLI now has proper scheduler and worker subcommands with argparse, matching what compose.yaml invokes. Adds a PostgreSQL connectivity check using psycopg with graceful degradation if DATABASE_URL is unset or psycopg is unavailable.

  2. Removed unnecessary web-edge network since the web port is bound to loopback only — the extra network added no security or routing value.

  3. .env.example port aligned with README (3406 instead of 12345, matching the 34xx pattern used by prod/staging ports 3410/3411).

Verification

$ just test  # both ingest and web suites pass (6 tests)
$ just lint  # ruff + eslint pass
$ just typecheck  # ty + tsc pass
$ just check-compose  # valid compose config

Closes #36

## Summary Thorough review of the Phase 1 implementation uncovered three critical bugs and several improvements. All fixes pass the existing lint, typecheck, and test suites. ### Critical fixes 1. **PostgreSQL volume path** (`compose.yaml:22`): Mounted at `/var/lib/postgresql` instead of `/var/lib/postgresql/data`. The PostgreSQL image stores data in the `data` subdirectory; without this fix, data would not persist across container restarts. 2. **Web health check URL** (`compose.yaml:90`): The health check fetched `/health` but the TanStack Start route is at `/api/health`. This would cause the container to report unhealthy and trigger continuous restarts. 3. **`COMPOSE_PROJECT_NAME` from env file** (`Justfile:58-62`): Docker Compose's `--env-file` only performs variable substitution in the compose file; it does not set environment variables for compose config like `COMPOSE_PROJECT_NAME`. Without this fix, production and staging deployments would use the same project name (`lawnet-scraper` from the compose file's `name:` field), sharing containers, networks, and volumes. Fixed by extracting the project name from the env file and passing it via `-p`. ### Improvements 4. **Data-ingest CLI** now has proper `scheduler` and `worker` subcommands with argparse, matching what `compose.yaml` invokes. Adds a PostgreSQL connectivity check using `psycopg` with graceful degradation if `DATABASE_URL` is unset or `psycopg` is unavailable. 5. **Removed unnecessary `web-edge` network** since the web port is bound to loopback only — the extra network added no security or routing value. 6. **`.env.example` port** aligned with README (3406 instead of 12345, matching the 34xx pattern used by prod/staging ports 3410/3411). ### Verification ``` $ just test # both ingest and web suites pass (6 tests) $ just lint # ruff + eslint pass $ just typecheck # ty + tsc pass $ just check-compose # valid compose config ``` Closes #36
feat: fix critical compose bugs and improve data-ingest CLI
All checks were successful
CI / lint (pull_request) Successful in 59s
Details
CI / typecheck (pull_request) Successful in 58s
Details
CI / test (pull_request) Successful in 56s
Details
CI / check-compose (pull_request) Successful in 4s
Details
a4a55f72d9 
- Fix PostgreSQL volume mount path from /var/lib/postgresql to
  /var/lib/postgresql/data so data persists correctly
- Fix web health check URL to match /api/health route
- Remove unnecessary web-edge network (port binds to loopback)
- Fix deploy recipe to pass COMPOSE_PROJECT_NAME via -p flag
  since --env-file only does variable substitution
- Add proper scheduler/worker subcommands to data-ingest CLI
- Add PostgreSQL connectivity check with psycopg dependency
- Align .env.example WEB_HOST_PORT with README (3406)
dernbu reviewed 2026-06-14 06:06:41 +02:00
compose.yaml Outdated
@ -20,3 +20,3 @@
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql
- postgres-data:/var/lib/postgresql/data
dernbu commented 2026-06-14 06:06:16 +02:00
Author
Owner
Copy link

For pg18? Help me doule check this with web search

For pg18? Help me doule check this with web search
dernbu marked this conversation as resolved
dernbu commented 2026-06-14 06:07:57 +02:00
Author
Owner
Copy link

Good catch. Confirmed from Docker Hub docs:

PostgreSQL 18+ changed PGDATA to /var/lib/postgresql/18/docker (versioned). The VOLUME is now defined at /var/lib/postgresql. Mounts should target the parent /var/lib/postgresql, NOT the data subdirectory.

The rule:

  • PG17 and earlier: mount at /var/lib/postgresql/data
  • PG18+: mount at /var/lib/postgresql (because PGDATA is now /var/lib/postgresql/18/docker)

The original code was correct for postgres:18-alpine. Reverted the change.

Good catch. Confirmed from Docker Hub docs: > **PostgreSQL 18+ changed `PGDATA`** to `/var/lib/postgresql/18/docker` (versioned). The `VOLUME` is now defined at `/var/lib/postgresql`. Mounts should target the parent `/var/lib/postgresql`, NOT the `data` subdirectory. The rule: - **PG17 and earlier**: mount at `/var/lib/postgresql/data` - **PG18+**: mount at `/var/lib/postgresql` (because PGDATA is now `/var/lib/postgresql/18/docker`) The original code was correct for `postgres:18-alpine`. Reverted the change.
dernbu force-pushed feat/10.1-continuous-deployment from a4a55f72d9
All checks were successful
CI / lint (pull_request) Successful in 59s
Details
CI / typecheck (pull_request) Successful in 58s
Details
CI / test (pull_request) Successful in 56s
Details
CI / check-compose (pull_request) Successful in 4s
Details
to 1a58b2aee3
Some checks failed
CI / lint (pull_request) Successful in 56s
Details
CI / typecheck (pull_request) Successful in 1m1s
Details
CI / test (pull_request) Successful in 57s
Details
CI / check-compose (pull_request) Successful in 4s
Details
Deploy / deploy-production (push) Has been cancelled
Details
Deploy / deploy-staging (push) Has been cancelled
Details
2026-06-14 06:08:31 +02:00 Compare
dernbu stopped working 2026-06-14 06:11:43 +02:00
3 seconds
dernbu merged commit 1a58b2aee3 into main 2026-06-14 06:13:54 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Total time spent: 3 seconds
dernbu
3 seconds
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dernbu/lawnet-scraper!46
No description provided.