finishing QDRO section

2025-08-15 17:19:51 -05:00
parent 006ef3d7b1
commit abc7f289d1
22 changed files with 2753 additions and 46 deletions
--- a/docs/MISSING_FEATURES_TODO.md
+++ b/docs/MISSING_FEATURES_TODO.md
@@ -12,37 +12,37 @@ Based on the comprehensive analysis of the legacy Paradox system, this document

 **Legacy Feature**: Sophisticated legal document generation with WordPerfect integration

-**Current Status**: ❌ Not implemented
+**Current Status**: ✅ **COMPLETED**

 **Required Components**:

 #### 1.1 Document Template Management
- [ ] Create `DocumentTemplate` model
+- [x] Create `DocumentTemplate` model
  - Template name, description, file path
  - Category/area of law classification
  - Status (active/inactive)
  - Version tracking
- [ ] Template upload and storage system
- [ ] Template preview capabilities
- [ ] Template search and filtering
+- [x] Template upload and storage system
+- [x] Template preview capabilities
+- [x] Template search and filtering

 #### 1.2 Template Keyword/Index System
- [ ] Create `TemplateKeyword` model for searchable tags
- [ ] Keyword management interface
- [ ] Advanced template search by keywords
- [ ] Template categorization system
+- [x] Create `TemplateKeyword` model for searchable tags
+- [x] Keyword management interface
+- [x] Advanced template search by keywords
+- [x] Template categorization system

 #### 1.3 Data Merge Engine
- [ ] Variable extraction from current data context
- [ ] Merge field mapping system
- [ ] Template variable substitution engine
- [ ] Support for multiple output formats (PDF, DOCX)
+- [x] Variable extraction from current data context
+- [x] Merge field mapping system
+- [x] Template variable substitution engine
+- [x] Support for multiple output formats (PDF, DOCX)

 #### 1.4 Form Selection Interface
- [ ] Multi-template selection UI
- [ ] Template preview and description display
- [ ] Batch document generation
- [ ] Generated document management
+- [x] Multi-template selection UI
+- [x] Template preview and description display
+- [ ] Batch document generation (planned for future iteration)
+- [ ] Generated document management (planned for future iteration)

 **API Endpoints Needed**:
 ```
@@ -53,6 +53,73 @@ GET    /api/templates/{id}/preview
 POST   /api/documents/generate-batch
 ```

+#### 1.5 Storage and Upload (DOCX/PDF)
+- [x] Accept `.docx` (mergeable) and `.pdf` (static/stampable) uploads
+- [x] Store originals in object storage (S3/MinIO) with server-side encryption
+- [x] Persist metadata: filename, MIME type, byte size, checksum (SHA-256), uploader, created_at
+- [ ] Virus scan and file-type sniffing (MIME > extension) (security enhancement for future)
+- [x] Max size and extension allowlist validation
+- [x] Optional tenant-aware storage prefixing (by environment/firm)
+
+#### 1.6 Template Versioning
+- [x] Create `DocumentTemplate` and `DocumentTemplateVersion` models
+  - `DocumentTemplate`: name, description, category, active, current_version_id
+  - `DocumentTemplateVersion`: template_id, semantic version, storage_path, mime_type, checksum, changelog, created_by, created_at, is_approved
+- [x] Upload new versions without breaking existing merges
+- [x] Ability to pin a version for a batch/job
+- [x] Approval workflow for promoting a draft version to current
+
+#### 1.7 Merge Engine (uses `FormVariable`/`ReportVariable`)
+- [x] Variable syntax: `{{ variable_identifier }}` inside `.docx` templates
+- [x] Resolution order (short-circuit on first hit):
+  1) Explicit context passed by API (e.g., file_no, customer, logged_in_user)
+  2) `FormVariable` by identifier → resolve via a safe, predefined repository function mapped from `query`
+  3) `ReportVariable` by identifier → same safe resolution
+  4) Built-ins/utilities (dates, formatting helpers)
+- [x] Missing variable strategies: leave token, empty string, or raise warning (configurable)
+- [x] Rendering engine: restricted Jinja2-like context with `docxtpl` for `.docx`
+- [ ] Output formats: `.docx` (native) and `.pdf` (convert via headless LibreOffice) (PDF conversion for future iteration)
+- [x] Robust error reporting: unresolved variables list, stack traces (server logs), user-friendly message
+- [x] Auditing: record variables resolved and their sources (context vs `FormVariable`/`ReportVariable`)
+
+#### 1.8 Batch Generation
+- [ ] Async queue jobs for batch merges (Celery/RQ) with progress tracking (future iteration)
+- [ ] Idempotency keys to avoid duplicate batches (future iteration)
+- [ ] Per-item success/failure reporting; partial retry support (future iteration)
+- [ ] Output bundling: store each generated document; optional ZIP download of the set (future iteration)
+- [ ] Throttling and concurrency limits (future iteration)
+- [ ] Audit trail: who initiated, when, template/version used, filters applied (future iteration)
+
+#### 1.9 UI for Templates, Keywords, Previews
+- [x] Templates list/detail with version history and diff of changelog
+- [x] Upload new version UI with changelog and approval checkbox
+- [x] Keyword/tag management (create/read/update/delete) and filter by tags/category
+- [x] Variable inspector: show `{{ }}` tokens discovered in the template and their resolution source
+- [x] Preview with sample data (single-record merge); show unresolved variables warning
+- [ ] Batch generation wizard: choose template/version, select scope (files/customers), review count and estimated time, run asynchronously (future iteration)
+
+#### 1.10 Additional API Endpoints
+```
+✅ GET    /api/templates/{id}
+✅ PUT    /api/templates/{id}
+✅ POST   /api/templates/{id}/versions          # upload new version
+✅ GET    /api/templates/{id}/versions          # list versions
+✅ PUT    /api/templates/{id}/versions/{vid}    # approve/pin version
+✅ GET    /api/templates/{id}/variables         # introspect template tokens
+✅ POST   /api/templates/{id}/preview           # preview merge with sample/context
+⏳ POST   /api/documents/jobs                   # start batch job (future iteration)
+⏳ GET    /api/documents/jobs/{job_id}          # job status/progress (future iteration)
+⏳ GET    /api/documents/jobs/{job_id}/result   # download bundle (ZIP) (future iteration)
+✅ POST   /api/templates/{id}/keywords          # add/remove keywords
+```
+
+#### 1.11 Acceptance Criteria
+- [x] Upload `.docx` template, add keywords, create v1.0.0, preview succeeds with sample data
+- [x] Merge uses `FormVariable`/`ReportVariable` when identifiers match tokens
+- [ ] Generate a batch of N files into PDFs; job completes with progress and downloadable ZIP (future iteration)
+- [x] Version pinning respected; publishing new version does not alter prior batch outcomes
+- [x] All actions logged with user and timestamp; unresolved variables surfaced in UI
+
 ### 🔴 2. QDRO (Pension Division) Module

 **Legacy Feature**: Specialized module for Qualified Domestic Relations Orders
--- a/docs/PENSIONS.md
+++ b/docs/PENSIONS.md
@@ -0,0 +1,175 @@
+# Pensions API Reference (Fields and Sorting)
+
+This document summarizes key fields for pensions-related endpoints and the allowed sort fields for each list.
+
+## Models (key fields)
+
+- Pension
+  - file_no (string), version (string), plan_id (string), plan_name (string)
+  - title, first, last, birth, race, sex
+  - info, valu, accrued, vested_per, start_age
+  - cola, max_cola, withdrawal, pre_dr, post_dr, tax_rate
+
+- PensionSchedule
+  - id, file_no, version, vests_on (date), vests_at (number), frequency (string)
+
+- MarriageHistory
+  - id, file_no, version, spouse_name (string), notes (text)
+  - married_from (date), married_to (date), married_years (number)
+  - service_from (date), service_to (date), service_years (number)
+  - marital_percent (number)
+
+- DeathBenefit
+  - id, file_no, version
+  - beneficiary_name (string), benefit_type (string), notes (text)
+  - lump1, lump2, growth1, growth2, disc1, disc2 (numbers)
+  - created_at, updated_at (timestamps)
+
+- SeparationAgreement
+  - id, file_no, version, agreement_date (date), terms (text), notes (text)
+
+## Allowed sort fields
+
+- Schedules (`GET /api/pensions/schedules`)
+  - id, file_no, version, vests_on, vests_at
+
+- Marriages (`GET /api/pensions/marriages`)
+  - id, file_no, version, married_from, married_to, marital_percent, service_from, service_to
+
+- Death benefits (`GET /api/pensions/death-benefits`)
+  - id, file_no, version, lump1, lump2, growth1, growth2, disc1, disc2, created
+
+- Separations (`GET /api/pensions/separations`)
+  - id, file_no, version, agreement_date
+
+## Notes
+
+- All list endpoints support `skip`, `limit`, `sort_by`, `sort_dir`, and `include_total`.
+- Whitelisted fields only are accepted for sorting; unknown fields fall back to defaults.
+- Tokenized search semantics: AND across tokens, OR across listed fields. Matching is case-insensitive and trims simple punctuation.
+- Sorting behavior: for string columns, sorting is case-insensitive (performed on lowercased values) for stable ordering.
+- Basic tokenized search is available per endpoint:
+  - Schedules: version, frequency
+  - Marriages: version, spouse_name, notes
+  - Death benefits: version, beneficiary_name, benefit_type, notes
+  - Separations: version, terms, notes
+
+## Examples (curl)
+
+Schedules with filters, sorting, include_total:
+```bash
+curl \
+  "http://localhost:6920/api/pensions/schedules?file_no=F-1&version=02&vests_at_min=10&vests_at_max=50&sort_by=vests_on&sort_dir=asc&include_total=true"
+```
+
+Marriages with tokenized search and sorting:
+```bash
+curl \
+  "http://localhost:6920/api/pensions/marriages?file_no=F-1&search=Gamma&sort_by=married_from&sort_dir=desc"
+```
+
+Death benefits with numeric ranges and sorting:
+```bash
+curl \
+  "http://localhost:6920/api/pensions/death-benefits?file_no=F-1&lump1_min=150&lump1_max=250&growth1_min=1.5&growth1_max=2.5&disc1_min=0.55&disc1_max=0.65&sort_by=lump1&sort_dir=desc"
+```
+
+Separations with date range filter and sorting:
+```bash
+curl \
+  "http://localhost:6920/api/pensions/separations?file_no=F-1&start=2024-01-01&end=2024-03-31&sort_by=agreement_date&sort_dir=asc"
+```
+
+Nested detail with independent paging/sorting per section:
+```bash
+curl \
+  "http://localhost:6920/api/pensions/F-1/detail?s_sort_by=vests_on&s_limit=5&m_sort_by=married_from&d_sort_by=lump1&sep_sort_by=agreement_date"
+```
+
+## Example responses (shapes)
+
+Schedules as a plain list (default):
+```json
+[
+  {
+    "id": 1,
+    "file_no": "F-1",
+    "version": "01",
+    "vests_on": "2024-01-01",
+    "vests_at": 10.0,
+    "frequency": "Monthly"
+  },
+  {
+    "id": 2,
+    "file_no": "F-1",
+    "version": "01",
+    "vests_on": "2024-02-01",
+    "vests_at": 20.0,
+    "frequency": "Monthly"
+  }
+]
+```
+
+Schedules with include_total=true:
+```json
+{
+  "items": [
+    {
+      "id": 1,
+      "file_no": "F-1",
+      "version": "01",
+      "vests_on": "2024-01-01",
+      "vests_at": 10.0,
+      "frequency": "Monthly"
+    }
+  ],
+  "total": 12
+}
+```
+
+Pension detail with nested lists (trimmed):
+```json
+{
+  "pension": {
+    "id": 123,
+    "file_no": "F-1",
+    "version": "01",
+    "plan_id": "PID1",
+    "plan_name": "Plan A",
+    "vested_per": 50.0,
+    "tax_rate": 25.0
+  },
+  "schedules": { "items": [/* ... */], "total": 3 },
+  "marriages": { "items": [/* ... */], "total": 1 },
+  "death_benefits": { "items": [/* ... */], "total": 0 },
+  "separations": { "items": [/* ... */], "total": 2 }
+}
+```
+
+Sample Pension record (GET /api/pensions/{pension_id}):
+```json
+{
+  "id": 123,
+  "file_no": "F-1",
+  "version": "01",
+  "plan_id": "PID1",
+  "plan_name": "Plan A",
+  "title": "Mr.",
+  "first": "John",
+  "last": "Doe",
+  "birth": "1970-05-01",
+  "race": "W",
+  "sex": "M",
+  "info": "Notes...",
+  "valu": 100000.0,
+  "accrued": 50000.0,
+  "vested_per": 50.0,
+  "start_age": 65,
+  "cola": 2.0,
+  "max_cola": 3.0,
+  "withdrawal": "ANNUITY",
+  "pre_dr": 4.0,
+  "post_dr": 3.0,
+  "tax_rate": 25.0
+}
+```