dernbu/lawnet-scraper
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

[1] Repository scaffold #1

New issue
Closed
opened 2026-06-13 18:59:07 +02:00 by dernbu · 3 comments
dernbu commented 2026-06-13 18:59:07 +02:00
Owner
Copy link

Objective

Create the repository and runtime foundation for the LawNet/eLitigation catalog.

Scope

  • Create the root Docker Compose file, .env.example, .gitignore, and Justfile.
  • Scaffold services/data-ingest, services/flyway, and services/web.
  • Add independent Python and web package roots.
  • Add Dockerfiles and health/startup wiring.
  • Configure the backend, egress, and web-edge networks.
  • Publish only the web service at 127.0.0.1:3406:3000.

Completion criteria

  • PostgreSQL starts and passes its health check.
  • Flyway can connect.
  • Web binds only to 127.0.0.1:3406.
  • No other service publishes a host port.
  • The worker has outbound access and PostgreSQL connectivity.

Closure

Complete all child issues, or merge a feature-branch pull request that resolves this issue.

Detailed requirements: docs/plans/high-level-plan.md, Phase 1.

## Objective Create the repository and runtime foundation for the LawNet/eLitigation catalog. ## Scope - Create the root Docker Compose file, `.env.example`, `.gitignore`, and `Justfile`. - Scaffold `services/data-ingest`, `services/flyway`, and `services/web`. - Add independent Python and web package roots. - Add Dockerfiles and health/startup wiring. - Configure the `backend`, `egress`, and `web-edge` networks. - Publish only the web service at `127.0.0.1:3406:3000`. ## Completion criteria - PostgreSQL starts and passes its health check. - Flyway can connect. - Web binds only to `127.0.0.1:3406`. - No other service publishes a host port. - The worker has outbound access and PostgreSQL connectivity. ## Closure Complete all child issues, or merge a feature-branch pull request that resolves this issue. Detailed requirements: `docs/plans/high-level-plan.md`, Phase 1.
dernbu commented 2026-06-13 19:20:23 +02:00
Author
Owner
Copy link

Implementation record

This issue will be implemented as one feature without child issues. The publishable work is on feat/1-repository-scaffold; local-only planning/reference files remain in the Jujutsu l commit.

Confirmed decisions

  • Complete all Phase 1 infrastructure in this issue and open a resolving feature-branch PR after verification.
  • Keep Phase 1 infrastructural; queue processing, catalog schema, ingestion, and the full BFF remain in later phases.
  • Runtime/tooling: PostgreSQL 18, Python 3.13 with uv, Node.js 22.22.3 selected through fnm/.node-version, and pnpm 10.
  • Use Flyway Community and add a minimal V1 bootstrap migration. Application-owned catalog and ingest schemas remain Phase 2 work.
  • Scaffold a real minimal TanStack Start application rather than a temporary HTTP server.
  • Root Compose filename is compose.yaml.
  • Web is the only published service, fixed at 127.0.0.1:3406:3000. PostgreSQL, Flyway, scheduler, and worker publish no host ports.
  • PostgreSQL data uses a named Docker volume. just down preserves it.
  • Networks follow the plan exactly: internal backend; worker also joins egress; web also joins web-edge.
  • Containers run as non-root where application images are controlled, use multi-stage builds, production dependencies, and init handling.
  • .env.example contains usable development defaults and all planned configuration variables. INTERNAL_API_TOKEN may remain blank/disabled during this phase.
  • Implement the planned Justfile surface. Commands owned by later phases fail with explicit phase-specific messages.
  • Add baseline formatting, linting, type checking, unit tests, and Compose smoke tests.
  • Update README setup and verification documentation.
  • CI and dependency automation remain Phase 10 work.
  • The accidental local file named 1, which contained an already-rotated token, is removed and will not enter history.
  • Preserve AGENT.md and other unpublished planning/reference material in the local-only l commit.

Flyway and health design

Readiness is checked at three levels:

  1. PostgreSQL must pass pg_isready.
  2. Flyway runs as a one-shot service after PostgreSQL is healthy; web, scheduler, and worker use service_completed_successfully and cannot start until migration succeeds.
  3. GET /health checks PostgreSQL connectivity and queries public.flyway_schema_history, requiring the successful V1 bootstrap migration and no failed migration entries. It returns HTTP 503 without exposing SQL errors or credentials when dependencies are unavailable.

Scheduler and worker placeholder behavior

Both processes:

  • Connect to PostgreSQL on startup.
  • Emit structured JSON readiness/status logs.
  • Remain alive and periodically verify database connectivity.
  • Perform no scheduling, job claiming, or source requests in normal operation.
  • Expose no ports.

Worker outbound access is verified only by an explicit Compose smoke check using HTTPS connections to the documented LawNet and eLitigation hosts; startup does not contact either source.

Verification target

The automated checks will cover package tests/lint/type checking plus:

  • PostgreSQL health.
  • Flyway successful completion and V1 history entry.
  • Web /health database/Flyway readiness.
  • Scheduler and worker database health.
  • Exact network membership.
  • Exact web loopback port binding and absence of all other published ports.
  • Worker outbound HTTPS connectivity.

Current progress

Repository topology and feature bookmark are established. Root Compose/configuration, the Flyway bootstrap, Python service skeleton, TanStack Start web skeleton, Dockerfiles, Justfile, README, and smoke-test harness have been drafted. Lockfile generation, formatting, builds, runtime smoke testing, and final fixes are still in progress.

## Implementation record This issue will be implemented as one feature without child issues. The publishable work is on `feat/1-repository-scaffold`; local-only planning/reference files remain in the Jujutsu `l` commit. ### Confirmed decisions - Complete all Phase 1 infrastructure in this issue and open a resolving feature-branch PR after verification. - Keep Phase 1 infrastructural; queue processing, catalog schema, ingestion, and the full BFF remain in later phases. - Runtime/tooling: PostgreSQL 18, Python 3.13 with `uv`, Node.js `22.22.3` selected through `fnm`/`.node-version`, and pnpm 10. - Use Flyway Community and add a minimal V1 bootstrap migration. Application-owned `catalog` and `ingest` schemas remain Phase 2 work. - Scaffold a real minimal TanStack Start application rather than a temporary HTTP server. - Root Compose filename is `compose.yaml`. - Web is the only published service, fixed at `127.0.0.1:3406:3000`. PostgreSQL, Flyway, scheduler, and worker publish no host ports. - PostgreSQL data uses a named Docker volume. `just down` preserves it. - Networks follow the plan exactly: internal `backend`; worker also joins `egress`; web also joins `web-edge`. - Containers run as non-root where application images are controlled, use multi-stage builds, production dependencies, and init handling. - `.env.example` contains usable development defaults and all planned configuration variables. `INTERNAL_API_TOKEN` may remain blank/disabled during this phase. - Implement the planned Justfile surface. Commands owned by later phases fail with explicit phase-specific messages. - Add baseline formatting, linting, type checking, unit tests, and Compose smoke tests. - Update README setup and verification documentation. - CI and dependency automation remain Phase 10 work. - The accidental local file named `1`, which contained an already-rotated token, is removed and will not enter history. - Preserve `AGENT.md` and other unpublished planning/reference material in the local-only `l` commit. ### Flyway and health design Readiness is checked at three levels: 1. PostgreSQL must pass `pg_isready`. 2. Flyway runs as a one-shot service after PostgreSQL is healthy; web, scheduler, and worker use `service_completed_successfully` and cannot start until migration succeeds. 3. `GET /health` checks PostgreSQL connectivity and queries `public.flyway_schema_history`, requiring the successful V1 bootstrap migration and no failed migration entries. It returns HTTP 503 without exposing SQL errors or credentials when dependencies are unavailable. ### Scheduler and worker placeholder behavior Both processes: - Connect to PostgreSQL on startup. - Emit structured JSON readiness/status logs. - Remain alive and periodically verify database connectivity. - Perform no scheduling, job claiming, or source requests in normal operation. - Expose no ports. Worker outbound access is verified only by an explicit Compose smoke check using HTTPS connections to the documented LawNet and eLitigation hosts; startup does not contact either source. ### Verification target The automated checks will cover package tests/lint/type checking plus: - PostgreSQL health. - Flyway successful completion and V1 history entry. - Web `/health` database/Flyway readiness. - Scheduler and worker database health. - Exact network membership. - Exact web loopback port binding and absence of all other published ports. - Worker outbound HTTPS connectivity. ### Current progress Repository topology and feature bookmark are established. Root Compose/configuration, the Flyway bootstrap, Python service skeleton, TanStack Start web skeleton, Dockerfiles, Justfile, README, and smoke-test harness have been drafted. Lockfile generation, formatting, builds, runtime smoke testing, and final fixes are still in progress.
dernbu commented 2026-06-13 19:30:04 +02:00
Author
Owner
Copy link

Implementation complete

The scaffold has been implemented and published in PR #11.

Delivered

  • PostgreSQL 18 with a persistent named volume and health check.
  • Flyway 12 Community with a V1 bootstrap migration and successful-completion startup gate.
  • Database-aware idle scheduler and worker processes with structured JSON logs and Docker health checks.
  • Worker-only outbound network access and explicit HTTPS egress verification.
  • Minimal production-built TanStack Start application on Node.js 22.22.3, managed locally through fnm.
  • /health reporting process, PostgreSQL, and Flyway schema-history readiness without leaking dependency errors.
  • Exact backend, egress, and web-edge topology.
  • Only 127.0.0.1:3406:3000 is published; every other service has no host port.
  • Independent Python/uv and web/pnpm package roots with lockfiles, linting, formatting, type checking, and unit tests.
  • Root .env.example, .gitignore, compose.yaml, Justfile, README documentation, Dockerfiles, and Compose smoke harness.

Verification completed

  • just config
  • just lint
  • just typecheck
  • just test

Results:

  • Python: 3 tests passed; Ruff and Pyright clean.
  • Web: 1 test passed; ESLint, Prettier, and strict TypeScript clean.
  • Both application images build and run as non-root users.
  • PostgreSQL healthy; Flyway V1 applied and repeated migration runs are successful.
  • Web, scheduler, and worker healthy.
  • /health returned 200 with PostgreSQL connected and Flyway ready.
  • Network membership and port-isolation assertions passed.
  • Worker HTTPS egress reached both documented source hosts.
  • just down stopped the stack while preserving lawnet-scraper_postgres-data.

The publishable feature is commit 5cdd1a9d7a7a on feat/1-repository-scaffold. Local-only files remain in the unpushed l descendant and are not part of the PR.

## Implementation complete The scaffold has been implemented and published in PR [#11](https://git.yongbeom.com/dernbu/lawnet-scraper/pulls/11). ### Delivered - PostgreSQL 18 with a persistent named volume and health check. - Flyway 12 Community with a V1 bootstrap migration and successful-completion startup gate. - Database-aware idle scheduler and worker processes with structured JSON logs and Docker health checks. - Worker-only outbound network access and explicit HTTPS egress verification. - Minimal production-built TanStack Start application on Node.js 22.22.3, managed locally through `fnm`. - `/health` reporting process, PostgreSQL, and Flyway schema-history readiness without leaking dependency errors. - Exact `backend`, `egress`, and `web-edge` topology. - Only `127.0.0.1:3406:3000` is published; every other service has no host port. - Independent Python/`uv` and web/pnpm package roots with lockfiles, linting, formatting, type checking, and unit tests. - Root `.env.example`, `.gitignore`, `compose.yaml`, `Justfile`, README documentation, Dockerfiles, and Compose smoke harness. ### Verification completed - `just config` - `just lint` - `just typecheck` - `just test` Results: - Python: 3 tests passed; Ruff and Pyright clean. - Web: 1 test passed; ESLint, Prettier, and strict TypeScript clean. - Both application images build and run as non-root users. - PostgreSQL healthy; Flyway V1 applied and repeated migration runs are successful. - Web, scheduler, and worker healthy. - `/health` returned `200` with PostgreSQL connected and Flyway ready. - Network membership and port-isolation assertions passed. - Worker HTTPS egress reached both documented source hosts. - `just down` stopped the stack while preserving `lawnet-scraper_postgres-data`. The publishable feature is commit `5cdd1a9d7a7a` on `feat/1-repository-scaffold`. Local-only files remain in the unpushed `l` descendant and are not part of the PR.
dernbu commented 2026-06-13 19:45:09 +02:00
Author
Owner
Copy link

PR split

The original combined PR #11 has been closed without merge and replaced by four service-scoped stacked PRs:

  1. #12 Scaffold data-ingest service — base main, 484 lines including uv.lock.
  2. #13 Scaffold Flyway service — base feat/1-data-ingest-scaffold, 3 lines.
  3. #14 Scaffold web service — base feat/1-flyway-scaffold, 445 non-lockfile lines plus the required generated 4,544-line pnpm-lock.yaml.
  4. #15 Wire repository services — base feat/1-web-scaffold, 352 lines, and retains Resolves #1.

Merge in numerical order. Each PR contains only its stated service/wiring scope. Forgejo's changed-file lists were checked and contain no node_modules, .output, virtual environments, or cache directories.

Verification was rerun after splitting:

  • data-ingest: Ruff, Pyright, 3 pytest tests, Docker build.
  • web: frozen pnpm install, ESLint, TypeScript, 1 Vitest test, production build, Docker build.
  • final stack: just config, just lint, just typecheck, and just test, including full Compose health/network/port/egress assertions.
## PR split The original combined PR #11 has been closed without merge and replaced by four service-scoped stacked PRs: 1. [#12 Scaffold data-ingest service](https://git.yongbeom.com/dernbu/lawnet-scraper/pulls/12) — base `main`, 484 lines including `uv.lock`. 2. [#13 Scaffold Flyway service](https://git.yongbeom.com/dernbu/lawnet-scraper/pulls/13) — base `feat/1-data-ingest-scaffold`, 3 lines. 3. [#14 Scaffold web service](https://git.yongbeom.com/dernbu/lawnet-scraper/pulls/14) — base `feat/1-flyway-scaffold`, 445 non-lockfile lines plus the required generated 4,544-line `pnpm-lock.yaml`. 4. [#15 Wire repository services](https://git.yongbeom.com/dernbu/lawnet-scraper/pulls/15) — base `feat/1-web-scaffold`, 352 lines, and retains `Resolves #1`. Merge in numerical order. Each PR contains only its stated service/wiring scope. Forgejo's changed-file lists were checked and contain no `node_modules`, `.output`, virtual environments, or cache directories. Verification was rerun after splitting: - data-ingest: Ruff, Pyright, 3 pytest tests, Docker build. - web: frozen pnpm install, ESLint, TypeScript, 1 Vitest test, production build, Docker build. - final stack: `just config`, `just lint`, `just typecheck`, and `just test`, including full Compose health/network/port/egress assertions.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
temp
feat/53-rename-lawnet-routes
push-xrsnvuvuvsul
feat/37-bff-storage-frontend
l
feat/49-pg-bff-storage
push-zzvxlnkylqsp
feat/quick-and-dirty
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dernbu/lawnet-scraper#1
No description provided.