dernbu/lawnet-scraper
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

[37.2] PostgreSQL dependency injection for LawNet BFF storage #49

New issue
Closed
opened 2026-06-14 08:47:17 +02:00 by dernbu · 1 comment
dernbu commented 2026-06-14 08:47:17 +02:00
Owner
Copy link

Parent

#37

Objective

Add an environment-variable‑driven storage provider switch so the same BFF routes can run against PostgreSQL or file‑system storage, implement the PostgreSQL storage provider, add the required Flyway schema, and test the Postgres path.

Dependencies

Depends on #37 and #48 (37.1).

Scope

  • Add an environment variable (e.g. BFF_STORAGE) that selects filesystem or postgres at startup.
  • Wire dependency injection so BFF routes receive the resolved storage provider.
  • Implement the PostgreSQL storage provider for the LawNet catalog and case‑detail contract.
  • Define the LawNet‑related PostgreSQL schema (catalog records, document renditions, artifact metadata) in a new Flyway migration.
  • Add PostgreSQL‑backed integration tests for the storage contract.
  • Ensure the file‑system provider from #37.1 still works when BFF_STORAGE=filesystem.

Out of scope

  • UI/frontend dashboard or detail pages.
  • Live LawNet ingestion or worker processes.
  • Full‑text judgment search.

Done when

  • Setting BFF_STORAGE=filesystem or BFF_STORAGE=postgres makes the same BFF routes use the correct backend.
  • Flyway migration applies cleanly and is a no‑op on reapply.
  • PostgreSQL‑backed tests pass against the new schema.
  • The file‑system path from #37.1 continues to pass unchanged.
## Parent #37 ## Objective Add an environment-variable‑driven storage provider switch so the same BFF routes can run against PostgreSQL or file‑system storage, implement the PostgreSQL storage provider, add the required Flyway schema, and test the Postgres path. ## Dependencies Depends on #37 and #48 (37.1). ## Scope - Add an environment variable (e.g. `BFF_STORAGE`) that selects `filesystem` or `postgres` at startup. - Wire dependency injection so BFF routes receive the resolved storage provider. - Implement the PostgreSQL storage provider for the LawNet catalog and case‑detail contract. - Define the LawNet‑related PostgreSQL schema (catalog records, document renditions, artifact metadata) in a new Flyway migration. - Add PostgreSQL‑backed integration tests for the storage contract. - Ensure the file‑system provider from #37.1 still works when `BFF_STORAGE=filesystem`. ## Out of scope - UI/frontend dashboard or detail pages. - Live LawNet ingestion or worker processes. - Full‑text judgment search. ## Done when - Setting `BFF_STORAGE=filesystem` or `BFF_STORAGE=postgres` makes the same BFF routes use the correct backend. - Flyway migration applies cleanly and is a no‑op on reapply. - PostgreSQL‑backed tests pass against the new schema. - The file‑system path from #37.1 continues to pass unchanged.
dernbu commented 2026-06-14 10:52:51 +02:00
Author
Owner
Copy link

Planning update

Agreed sequencing for #49:

  1. First implement and merge #53 as a separate PR: rename the existing LawNet BFF API surface under /api/lawnet/* while preserving current filesystem-backed behavior.
  2. After #53 is merged, implement the PostgreSQL storage-provider work for this issue.

Agreed decisions for the #49 PostgreSQL implementation

  • Use BFF_STORAGE as the backend selector for now.
  • Keep the current LawnetStore behavior/contract for this slice, including the citation lookup and detail shape, while moving the route namespace via #53.
  • Use raw pg; defer Kysely and generated DB types.
  • Use the high-level schema direction, subject to review before implementation:
    • catalog.judgment
    • catalog.lawnet_judgment
    • catalog.lawnet_coram
    • catalog.lawnet_catchword
    • catalog.lawnet_artifact
  • Include catalog.judgment and nullable judgment_id now, even though live citation parsing/correlation is not part of this issue.
  • Use public opaque result IDs in the high-level format, including lawnet:<source-row-id> now and judgment:<canonical-id> later.
  • Keep LawNet docid as the unique source key while using an internal identity id for BFF result IDs.
  • Add coram and catchword child tables because the current detail response requires them.
  • Store artifact bytes in PostgreSQL bytea.
  • Treat artifact kind and LawNet rendition_type as separate concepts.
  • Fix original HTML artifact responses to download as attachments rather than render inline.
  • Keep limit/offset pagination for this slice.
  • Keep list behavior to the current behavior; do not add search/filtering in #49.
  • Compose may default BFF_STORAGE=postgres; standalone development should continue to default to filesystem.
  • PostgreSQL integration tests should be guarded by WEB_TEST_DATABASE_URL and skipped when absent.
  • Seed PostgreSQL integration tests directly with small SQL fixtures.
  • Use a single first-cut Flyway migration for the #49 LawNet BFF storage schema, with the exact SQL to be reviewed before implementation.

Explicitly deferred

  • Malformed/incomplete source-record handling is deferred to the reconciliation-table task. Long-term TODO doc added locally: docs/todos/long-term/malformed-source-record-reconciliation.md.
  • Kysely and generated DB types are deferred. Long-term TODO doc added locally: docs/todos/long-term/kysely-database-types.md.

Schema review still needed before coding

The next planning step is to review the high-level schema columns and constraints for the #49 migration, especially:

  • Whether catalog.lawnet_judgment.citation remains not null for #49 because malformed records are deferred.
  • Exact artifact kind allowed values for the BFF slice.
  • Exact LawNet rendition_type allowed values.
  • Whether sha256 should be bytea as in the high-level plan or text for easier API/test assertions.
  • Whether artifact filenames should be stored explicitly or derived safely from artifact metadata.
  • Which indexes are required for current listJudgments, getJudgment, citation lookup, and artifact lookup behavior.
## Planning update Agreed sequencing for #49: 1. First implement and merge #53 as a separate PR: rename the existing LawNet BFF API surface under `/api/lawnet/*` while preserving current filesystem-backed behavior. 2. After #53 is merged, implement the PostgreSQL storage-provider work for this issue. ## Agreed decisions for the #49 PostgreSQL implementation - Use `BFF_STORAGE` as the backend selector for now. - Keep the current `LawnetStore` behavior/contract for this slice, including the citation lookup and detail shape, while moving the route namespace via #53. - Use raw `pg`; defer Kysely and generated DB types. - Use the high-level schema direction, subject to review before implementation: - `catalog.judgment` - `catalog.lawnet_judgment` - `catalog.lawnet_coram` - `catalog.lawnet_catchword` - `catalog.lawnet_artifact` - Include `catalog.judgment` and nullable `judgment_id` now, even though live citation parsing/correlation is not part of this issue. - Use public opaque result IDs in the high-level format, including `lawnet:<source-row-id>` now and `judgment:<canonical-id>` later. - Keep LawNet `docid` as the unique source key while using an internal identity `id` for BFF result IDs. - Add coram and catchword child tables because the current detail response requires them. - Store artifact bytes in PostgreSQL `bytea`. - Treat artifact `kind` and LawNet `rendition_type` as separate concepts. - Fix original HTML artifact responses to download as attachments rather than render inline. - Keep `limit`/`offset` pagination for this slice. - Keep list behavior to the current behavior; do not add search/filtering in #49. - Compose may default `BFF_STORAGE=postgres`; standalone development should continue to default to filesystem. - PostgreSQL integration tests should be guarded by `WEB_TEST_DATABASE_URL` and skipped when absent. - Seed PostgreSQL integration tests directly with small SQL fixtures. - Use a single first-cut Flyway migration for the #49 LawNet BFF storage schema, with the exact SQL to be reviewed before implementation. ## Explicitly deferred - Malformed/incomplete source-record handling is deferred to the reconciliation-table task. Long-term TODO doc added locally: `docs/todos/long-term/malformed-source-record-reconciliation.md`. - Kysely and generated DB types are deferred. Long-term TODO doc added locally: `docs/todos/long-term/kysely-database-types.md`. ## Schema review still needed before coding The next planning step is to review the high-level schema columns and constraints for the #49 migration, especially: - Whether `catalog.lawnet_judgment.citation` remains `not null` for #49 because malformed records are deferred. - Exact artifact `kind` allowed values for the BFF slice. - Exact LawNet `rendition_type` allowed values. - Whether `sha256` should be `bytea` as in the high-level plan or text for easier API/test assertions. - Whether artifact filenames should be stored explicitly or derived safely from artifact metadata. - Which indexes are required for current `listJudgments`, `getJudgment`, citation lookup, and artifact lookup behavior.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
temp
feat/53-rename-lawnet-routes
push-xrsnvuvuvsul
feat/37-bff-storage-frontend
l
feat/49-pg-bff-storage
push-zzvxlnkylqsp
feat/quick-and-dirty
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dernbu/lawnet-scraper#49
No description provided.