delphi-database-v2/docs/IMPORT_GUIDE.md

# CSV Import System - User Guide

## Overview

The CSV import system allows you to migrate legacy Paradox database data into the Delphi Database application. The system works in two stages:

1. **Import Stage**: Load CSV files into legacy database models (preserving original schema)
2. **Sync Stage**: Synchronize legacy data to modern simplified models

## Prerequisites

- Admin access to the application
- CSV files from the legacy Paradox database
- Docker container running

## Import Order

**IMPORTANT**: Import tables in this exact order to avoid foreign key errors:

### Stage 1: Reference Tables (Import First)
These tables contain lookup data used by other tables.

1. TRNSTYPE.csv - Transaction types
2. TRNSLKUP.csv - Transaction lookup
3. FOOTERS.csv - Footer templates
4. FILESTAT.csv - File status codes
5. EMPLOYEE.csv - Employee records
6. GRUPLKUP.csv - Group lookup
7. FILETYPE.csv - File type codes
8. FVARLKUP.csv - File variable lookup
9. RVARLKUP.csv - Rolodex variable lookup

### Stage 2: Core Data Tables

10. ROLODEX.csv - Client/contact information
11. PHONE.csv - Phone numbers (references ROLODEX)
12. ROLEX_V.csv - Rolodex variables (references ROLODEX)
13. FILES.csv - Case/file records (references ROLODEX)
14. FILES_R.csv - File relationships (references FILES)
15. FILES_V.csv - File variables (references FILES)
16. FILENOTS.csv - File notes/memos (references FILES)
17. LEDGER.csv - Transaction ledger (references FILES)
18. DEPOSITS.csv - Deposit records
19. PAYMENTS.csv - Payment records (references FILES)

### Stage 3: Specialized Tables

20. PLANINFO.csv - Pension plan information
21. QDROS.csv - QDRO documents (references FILES)
22. PENSIONS.csv - Pension records (references FILES)
23. Pensions/MARRIAGE.csv - Marriage calculations (references PENSIONS)
24. Pensions/DEATH.csv - Death benefit calculations (references PENSIONS)
25. Pensions/SCHEDULE.csv - Vesting schedules (references PENSIONS)
26. Pensions/SEPARATE.csv - Separation calculations (references PENSIONS)
27. Pensions/RESULTS.csv - Pension calculation results (references PENSIONS)

## Step-by-Step Import Process

### 1. Access Admin Panel

1. Navigate to `http://localhost:8000/admin`
2. Login with admin credentials (default: admin/admin)

### 2. Upload CSV Files

1. Scroll to the **File Upload** section
2. Click "Select CSV Files" and choose your CSV files
3. You can upload multiple files at once
4. Choose whether to enable "Auto-import after upload" (default ON). When enabled, the system will import the uploaded files immediately following the Import Order Guide and will stop after the first file that reports any row errors.
5. Click "Upload Files"
6. Review the upload and auto-import results to ensure files were recognized and processed correctly

### 3. Import Reference Tables First

1. Scroll to the **Data Import** section
2. You'll see files grouped by type (trnstype, trnslkup, footers, etc.)
3. For each reference table group:
   - Select the checkbox next to the file(s) you want to import
   - Click the "Import" button for that group
   - Wait for the import to complete
   - Review the results

**Tip**: Use the "Select All" button to quickly select all files in a group.

### 4. Import Core Data Tables

Following the same process as step 3, import the core tables in order:

1. Import **rolodex** files
2. Import **phone** files
3. Import **rolex_v** files
4. Import **files** files
5. Import **files_r** files
6. Import **files_v** files
7. Import **filenots** files
8. Import **ledger** files
9. Import **deposits** files
10. Import **payments** files

### 5. Import Specialized Tables

Finally, import the specialized tables:

1. Import **planinfo** files
2. Import **qdros** files
3. Import **pensions** files
4. Import **pension_marriage** files
5. Import **pension_death** files
6. Import **pension_schedule** files
7. Import **pension_separate** files
8. Import **pension_results** files

### 6. Sync to Modern Models

After all legacy data is imported:

1. Scroll to the **Sync to Modern Models** section
2. **OPTIONAL**: Check "Clear existing modern data before sync" if you want to replace all current data
   - ⚠️ **WARNING**: This will delete all existing Client, Phone, Case, Transaction, Payment, and Document records!
3. Click "Start Sync Process"
4. Confirm the action in the dialog
5. Review the sync results

The sync process will:
- Convert Rolodex → Client
- Convert LegacyPhone → Phone
- Convert LegacyFile → Case
- Convert Ledger → Transaction
- Convert LegacyPayment → Payment
- Convert Qdros → Document

## Monitoring Import Progress

### Import Results

After each import operation, you'll see:

- **Total Rows**: Number of rows in the CSV file
- **Success Count**: Records successfully imported
- **Error Count**: Records that failed to import
- **Detailed Errors**: Specific error messages for failed records (first 10 shown)

### Import History

The **Recent Import History** section shows the last 10 import operations with:

- Import type
- Filename
- Status (Completed/Failed/Running)
- Record counts
- Timestamp

### Sync Results

After syncing, you'll see:

- **Records Synced**: Total records successfully synced to modern models
- **Records Skipped**: Records that couldn't be synced (e.g., missing foreign keys)
- **Errors**: Count of errors encountered
- **Per-Table Details**: Breakdown showing results for each modern table (Client, Phone, Case, etc.)

## Troubleshooting

### Import Errors

**"Foreign key constraint failed"**
- You imported tables out of order
- Solution: Import reference tables first, then core tables

**"No client found for rolodex ID"**
- The ROLODEX file wasn't imported before dependent files
- Solution: Import ROLODEX.csv first

**"No case found for file"**
- The FILES file wasn't imported before LEDGER/PAYMENTS
- Solution: Import FILES.csv before LEDGER.csv and PAYMENTS.csv

**"Encoding error" or "Unable to open file"**
- CSV file has unusual encoding
- The system tries multiple encodings automatically
- Check the error message for details

### Sync Errors

**"Records Skipped"**
- Legacy records reference non-existent parent records
- This is normal for incomplete datasets
- Review skipped record details in the sync results

**"Duplicate key error"**
- Running sync multiple times without clearing existing data
- Solution: Check "Clear existing modern data before sync" option

## Data Validation

After importing and syncing, verify your data:

### Check Record Counts

1. Navigate to the Dashboard (`/`)
2. Review the statistics cards showing counts for:
   - Total Clients
   - Active Cases
   - Total Transactions
   - Recent Payments

### Verify Relationships

1. Go to the Rolodex page (`/rolodex`)
2. Click on a client
3. Verify that:
   - Phone numbers are displayed correctly
   - Associated cases are shown
   - Case details link to transactions and payments

### Run Reports

Test the reporting functionality:

1. Generate a Phone Book report
2. Generate a Payments report
3. Verify data appears correctly in PDFs

## Best Practices

1. **Backup First**: Always backup your database before importing
2. **Test with Sample Data**: Test the import process with a small subset of data first
3. **Import Order Matters**: Always follow the recommended import order
4. **Review Errors**: Check import results carefully and address errors before proceeding
5. **Sync Last**: Only run the sync process after all legacy data is successfully imported
6. **Monitor Progress**: For large imports, monitor the Import History section
7. **Document Issues**: Keep notes of any import errors for troubleshooting

## File Naming Conventions

The system recognizes files by their names. Supported patterns:

- `ROLODEX*.csv` → rolodex import type
- `PHONE*.csv` → phone import type
- `FILES*.csv` → files import type
- `LEDGER*.csv` → ledger import type
- `PAYMENTS*.csv` → payments import type
- `TRNSTYPE*.csv` → trnstype import type
- etc.

## Performance Notes

- **Batch Processing**: Imports process 500 records per batch for optimal performance
- **Large Files**: Files with 10,000+ records may take several minutes
- **Database Locks**: Only one import operation should run at a time
- **Memory Usage**: Very large files (100,000+ records) may require increased Docker memory allocation

## Getting Help

If you encounter issues:

1. Check the application logs: `docker-compose logs delphi-db`
2. Review the Import History for error details
3. Verify your CSV file format matches the expected schema
4. Consult the legacy schema documentation in `docs/legacy-schema.md`

## Example: Complete Import Sequence

```bash
# 1. Start the application
docker-compose up -d

# 2. Access admin panel
# Navigate to http://localhost:8000/admin

# 3. Upload all CSV files at once
# Use the file upload form to select all CSV files

# 4. Import in order:
# - Reference tables (TRNSTYPE, TRNSLKUP, FOOTERS, etc.)
# - Core data (ROLODEX, PHONE, FILES, LEDGER, etc.)
# - Specialized (PLANINFO, QDROS, PENSIONS, etc.)

# 5. Sync to modern models
# Click "Start Sync Process" with or without clearing existing data

# 6. Verify
# Navigate to dashboard and verify record counts
```

## Technical Details

### Import Module: `app/import_legacy.py`

Contains import functions for all legacy tables with:
- Encoding detection (UTF-8, CP1252, Latin-1, etc.)
- Date parsing (MM/DD/YYYY, MM/DD/YY, YYYY-MM-DD)
- Decimal conversion with proper precision
- Batch insert optimization
- Structured logging

### Sync Module: `app/sync_legacy_to_modern.py`

Contains sync functions that:
- Map legacy IDs to modern table IDs
- Handle missing foreign key references gracefully
- Skip orphaned records with warnings
- Maintain referential integrity
- Support incremental or full replacement sync

### Database Models

- **Legacy Models**: Preserve original Paradox schema (Rolodex, LegacyPhone, LegacyFile, Ledger, etc.)
- **Modern Models**: Simplified application schema (Client, Phone, Case, Transaction, Payment, Document)