dernbu/lawnet-scraper
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

[3] LawNet PostgreSQL ingestion and artifacts #38

New issue
Open
opened 2026-06-14 05:23:47 +02:00 by dernbu · 4 comments
dernbu commented 2026-06-14 05:23:47 +02:00
Owner
Copy link

Objective

Implement LawNet ingestion against PostgreSQL using the storage/schema shape established by #37.

After this slice, real or fixture-backed LawNet catalog and document ingestion should populate the Postgres-backed store that the BFF/frontend already knows how to read.

Dependencies

Depends on #37.

Scope

  • Implement LawNet discovery/listing client behavior against fixtures first.
  • Persist LawNet source keys, citation, title, decision date, court, case number, availability, and raw source payload into PostgreSQL.
  • Preserve malformed or incomplete records rather than dropping them.
  • Handle duplicate LawNet source records idempotently.
  • Add an operator path to run a mocked or targeted LawNet discovery.
  • Implement LawNet document fetch jobs for HTML renditions and PDFs.
  • Support documented LawNet response variants, including singular and plural document shapes.
  • Preserve judgment, SLR, and unknown HTML renditions independently.
  • Select a primary judgment rendition using documented rules without deleting alternatives.
  • Store PDF artifacts independently from document fetch success.
  • Add fixture-backed tests for LawNet catalog ingest, document ingest, and artifact edge cases.
  • Verify the Postgres-backed BFF/frontend from #37 can read records produced by ingestion.

Out of scope

  • New frontend or BFF storage-contract behavior except fixes needed to consume ingested data correctly.
  • eLitigation records or comparison.
  • Sanitized public HTML rendering as the final reader experience.
  • Full durable queue hardening beyond the minimum needed to retry this flow.
  • Full-text judgment search.

Done when

  • A mocked LawNet discovery run stores records in PostgreSQL.
  • Records survive container restart.
  • Duplicate LawNet source records are handled idempotently.
  • Malformed source records are visible enough for debugging instead of being silently discarded.
  • A LawNet record can move from catalog discovery to document/artifact ingestion.
  • Judgment and SLR renditions are stored separately.
  • PDF-only failures do not discard document metadata or HTML.
  • Interrupted or failed fetch work can be retried manually.
  • The BFF/frontend from #37 can list and open ingested LawNet records from PostgreSQL.
## Objective Implement LawNet ingestion against PostgreSQL using the storage/schema shape established by #37. After this slice, real or fixture-backed LawNet catalog and document ingestion should populate the Postgres-backed store that the BFF/frontend already knows how to read. ## Dependencies Depends on #37. ## Scope - Implement LawNet discovery/listing client behavior against fixtures first. - Persist LawNet source keys, citation, title, decision date, court, case number, availability, and raw source payload into PostgreSQL. - Preserve malformed or incomplete records rather than dropping them. - Handle duplicate LawNet source records idempotently. - Add an operator path to run a mocked or targeted LawNet discovery. - Implement LawNet document fetch jobs for HTML renditions and PDFs. - Support documented LawNet response variants, including singular and plural document shapes. - Preserve judgment, SLR, and unknown HTML renditions independently. - Select a primary judgment rendition using documented rules without deleting alternatives. - Store PDF artifacts independently from document fetch success. - Add fixture-backed tests for LawNet catalog ingest, document ingest, and artifact edge cases. - Verify the Postgres-backed BFF/frontend from #37 can read records produced by ingestion. ## Out of scope - New frontend or BFF storage-contract behavior except fixes needed to consume ingested data correctly. - eLitigation records or comparison. - Sanitized public HTML rendering as the final reader experience. - Full durable queue hardening beyond the minimum needed to retry this flow. - Full-text judgment search. ## Done when - A mocked LawNet discovery run stores records in PostgreSQL. - Records survive container restart. - Duplicate LawNet source records are handled idempotently. - Malformed source records are visible enough for debugging instead of being silently discarded. - A LawNet record can move from catalog discovery to document/artifact ingestion. - Judgment and SLR renditions are stored separately. - PDF-only failures do not discard document metadata or HTML. - Interrupted or failed fetch work can be retried manually. - The BFF/frontend from #37 can list and open ingested LawNet records from PostgreSQL.
dernbu changed title from [3] LawNet document ingestion and case detail to [3] LawNet PostgreSQL ingestion and artifacts 2026-06-14 07:13:28 +02:00
dernbu commented 2026-06-16 17:22:57 +02:00
Author
Owner
Copy link

Implementation plan update from discussion:

  • Use two LawNet discovery modes:
    • Hourly incremental head scan for new cases.
    • Periodic full scan, weekly/biweekly/monthly, for stragglers, older backfills, availability reconciliation, and lower-priority metadata updates.
  • Incremental discovery always starts at page 1 using LawNet date-desc listing order.
  • Default page_size = 20 and known_overlap_required = 20.
  • During an incremental run:
    • Fetch pages from the head.
    • Insert/enqueue every unknown docid.
    • Update last_seen_at for known records encountered.
    • Count consecutive known docids.
    • Stop once consecutive_known >= known_overlap_required, normally one full known page.
    • Continue to page 2+ automatically if more than 20 new cases arrived since the last run.
  • Incremental discovery must not mark records unavailable. Only a successful periodic full scan may do that.
  • Keep persistence simple for this slice:
    • Add first_seen_at and last_seen_at to catalog.lawnet_judgment.
    • Add one ingest.lawnet_discovery_state row for LawNet with last incremental/full timestamps, last total/error, and configurable known_overlap_required defaulting to 20.
  • Defer heavier discovery_run / discovery_page / discovery_record audit checkpoint tables until we actually need resumable full scans or detailed drift diagnostics.
Implementation plan update from discussion: - Use two LawNet discovery modes: - Hourly incremental head scan for new cases. - Periodic full scan, weekly/biweekly/monthly, for stragglers, older backfills, availability reconciliation, and lower-priority metadata updates. - Incremental discovery always starts at page 1 using LawNet `date-desc` listing order. - Default `page_size = 20` and `known_overlap_required = 20`. - During an incremental run: - Fetch pages from the head. - Insert/enqueue every unknown `docid`. - Update `last_seen_at` for known records encountered. - Count consecutive known `docid`s. - Stop once `consecutive_known >= known_overlap_required`, normally one full known page. - Continue to page 2+ automatically if more than 20 new cases arrived since the last run. - Incremental discovery must not mark records unavailable. Only a successful periodic full scan may do that. - Keep persistence simple for this slice: - Add `first_seen_at` and `last_seen_at` to `catalog.lawnet_judgment`. - Add one `ingest.lawnet_discovery_state` row for LawNet with last incremental/full timestamps, last total/error, and configurable `known_overlap_required` defaulting to 20. - Defer heavier `discovery_run` / `discovery_page` / `discovery_record` audit checkpoint tables until we actually need resumable full scans or detailed drift diagnostics.
dernbu commented 2026-06-16 17:45:10 +02:00
Author
Owner
Copy link

Additional scheduler persistence update:

  • ingest.lawnet_discovery_state now also owns the full LawNet listing scan schedule.
  • The state row tracks:
    • full_scan_interval, default 14 days.
    • full_scan_retry_delay, default 1 hour.
    • next_full_scan_at, default now() for the initial scheduled full scan.
    • last_full_at, last_reported_total, and last_error.
  • The data-ingest scheduler reads next_full_scan_at from Postgres on each poll and only runs a full listing scan when the DB says it is due.
  • On successful full scan, the scheduler advances next_full_scan_at by the DB-configured interval.
  • On failed full scan, the scheduler advances next_full_scan_at by the DB-configured retry delay, avoiding immediate repeated attempts after restart.
  • This keeps the schedule durable across service restarts and avoids triggering full scans simply because the scheduler container restarted.

Current scope note: the implemented full scan upserts LawNet listing records and updates scan state. Availability/disappearance reconciliation remains deferred to the fuller reconciliation work.

Additional scheduler persistence update: - `ingest.lawnet_discovery_state` now also owns the full LawNet listing scan schedule. - The state row tracks: - `full_scan_interval`, default 14 days. - `full_scan_retry_delay`, default 1 hour. - `next_full_scan_at`, default `now()` for the initial scheduled full scan. - `last_full_at`, `last_reported_total`, and `last_error`. - The data-ingest scheduler reads `next_full_scan_at` from Postgres on each poll and only runs a full listing scan when the DB says it is due. - On successful full scan, the scheduler advances `next_full_scan_at` by the DB-configured interval. - On failed full scan, the scheduler advances `next_full_scan_at` by the DB-configured retry delay, avoiding immediate repeated attempts after restart. - This keeps the schedule durable across service restarts and avoids triggering full scans simply because the scheduler container restarted. Current scope note: the implemented full scan upserts LawNet listing records and updates scan state. Availability/disappearance reconciliation remains deferred to the fuller reconciliation work.
dernbu commented 2026-06-16 17:50:28 +02:00
Author
Owner
Copy link

Additional incremental schedule update:

  • ingest.lawnet_discovery_state now also tracks the next incremental head scan.
  • Added DB-owned fields:
    • incremental_scan_interval, default 1 hour.
    • incremental_scan_retry_delay, default 15 minutes.
    • next_incremental_scan_at, default now().
  • The scheduler now reads both next_full_scan_at and next_incremental_scan_at from Postgres.
  • Full scans have priority when both are due.
  • A successful full scan also advances next_incremental_scan_at, so the scheduler does not immediately run a duplicate head scan after refreshing the full listing.
  • A successful incremental scan advances next_incremental_scan_at by the DB-configured incremental interval.
  • A failed incremental scan advances next_incremental_scan_at by the DB-configured incremental retry delay.

This keeps both periodic scan types restart-safe and DB-driven.

Additional incremental schedule update: - `ingest.lawnet_discovery_state` now also tracks the next incremental head scan. - Added DB-owned fields: - `incremental_scan_interval`, default 1 hour. - `incremental_scan_retry_delay`, default 15 minutes. - `next_incremental_scan_at`, default `now()`. - The scheduler now reads both `next_full_scan_at` and `next_incremental_scan_at` from Postgres. - Full scans have priority when both are due. - A successful full scan also advances `next_incremental_scan_at`, so the scheduler does not immediately run a duplicate head scan after refreshing the full listing. - A successful incremental scan advances `next_incremental_scan_at` by the DB-configured incremental interval. - A failed incremental scan advances `next_incremental_scan_at` by the DB-configured incremental retry delay. This keeps both periodic scan types restart-safe and DB-driven.
dernbu commented 2026-06-16 18:19:24 +02:00
Author
Owner
Copy link

Schema follow-up:

  • Trimmed ingest.lawnet_discovery_state so it stores durable state only:
    • next_incremental_scan_at
    • next_full_scan_at
    • last-run timestamps, last reported total/error, and updated_at
  • Removed discovery configuration from the state table:
    • known_overlap_required
    • incremental_scan_interval
    • incremental_scan_retry_delay
    • full_scan_interval
    • full_scan_retry_delay
  • The scan cadence and retry delays now remain runtime defaults in the ingest code, while the next due timestamps remain persisted across restarts.
  • Scheduler overlap threshold is now runtime configuration (--known-overlap, default 20) instead of DB state.
Schema follow-up: - Trimmed `ingest.lawnet_discovery_state` so it stores durable state only: - `next_incremental_scan_at` - `next_full_scan_at` - last-run timestamps, last reported total/error, and `updated_at` - Removed discovery configuration from the state table: - `known_overlap_required` - `incremental_scan_interval` - `incremental_scan_retry_delay` - `full_scan_interval` - `full_scan_retry_delay` - The scan cadence and retry delays now remain runtime defaults in the ingest code, while the next due timestamps remain persisted across restarts. - Scheduler overlap threshold is now runtime configuration (`--known-overlap`, default 20) instead of DB state.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
temp
feat/53-rename-lawnet-routes
push-xrsnvuvuvsul
feat/37-bff-storage-frontend
l
feat/49-pg-bff-storage
push-zzvxlnkylqsp
feat/quick-and-dirty
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dernbu/lawnet-scraper#38
No description provided.