dernbu/lawnet-scraper
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

[8] Reliable ingestion operations #43

New issue
Open
opened 2026-06-14 05:24:40 +02:00 by dernbu · 0 comments
dernbu commented 2026-06-14 05:24:40 +02:00
Owner
Copy link

Objective

Make ingestion reliable enough to run continuously without manual babysitting.

This issue hardens the operational substrate after the main product ingestion paths exist, rather than making queue infrastructure the first product milestone.

Dependencies

Depends on #42.

Scope

  • Implement durable job insertion, atomic claiming, completion, retry, and dead-job state.
  • Use leases with expiry recovery so interrupted workers do not strand work forever.
  • Ensure one worker processes one job at a time unless future scope explicitly changes that.
  • Add scheduler behavior for daily discovery and four-week artifact refresh TTLs.
  • Make discovery continuation bounded and restart-safe.
  • Add operator CLI commands for enqueueing discovery, forcing refreshes, inspecting queue state, and requeueing dead jobs.
  • Add token-protected internal endpoints for reconciliation issues and dead jobs if useful for operations.
  • Add structured logging and enough metadata to diagnose failed source jobs.

Out of scope

  • Multi-worker concurrency.
  • Administrator UI.
  • Backup/restore automation.
  • Live source access in CI.

Done when

  • Jobs survive service and database restarts.
  • Expired leases are reclaimed.
  • Repeated failures produce dead jobs with inspectable error context.
  • Restarts do not create duplicate recurring jobs or force discovery immediately.
  • Operators can inspect and requeue failed work through documented commands.
  • The existing LawNet, eLitigation, artifact, and search flows run through the hardened job path.
## Objective Make ingestion reliable enough to run continuously without manual babysitting. This issue hardens the operational substrate after the main product ingestion paths exist, rather than making queue infrastructure the first product milestone. ## Dependencies Depends on #42. ## Scope - Implement durable job insertion, atomic claiming, completion, retry, and dead-job state. - Use leases with expiry recovery so interrupted workers do not strand work forever. - Ensure one worker processes one job at a time unless future scope explicitly changes that. - Add scheduler behavior for daily discovery and four-week artifact refresh TTLs. - Make discovery continuation bounded and restart-safe. - Add operator CLI commands for enqueueing discovery, forcing refreshes, inspecting queue state, and requeueing dead jobs. - Add token-protected internal endpoints for reconciliation issues and dead jobs if useful for operations. - Add structured logging and enough metadata to diagnose failed source jobs. ## Out of scope - Multi-worker concurrency. - Administrator UI. - Backup/restore automation. - Live source access in CI. ## Done when - Jobs survive service and database restarts. - Expired leases are reclaimed. - Repeated failures produce dead jobs with inspectable error context. - Restarts do not create duplicate recurring jobs or force discovery immediately. - Operators can inspect and requeue failed work through documented commands. - The existing LawNet, eLitigation, artifact, and search flows run through the hardened job path.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
temp
feat/53-rename-lawnet-routes
push-xrsnvuvuvsul
feat/37-bff-storage-frontend
l
feat/49-pg-bff-storage
push-zzvxlnkylqsp
feat/quick-and-dirty
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dernbu/lawnet-scraper#43
No description provided.