Add comprehensive CSV import system documentation
- Created IMPORT_GUIDE.md: Complete user guide with step-by-step instructions * Detailed import order for all 27+ tables * Troubleshooting guide * Data validation procedures * Best practices and performance notes - Created IMPORT_SYSTEM_SUMMARY.md: Technical implementation summary * Complete list of all implemented functions (28 import + 7 sync) * Architecture and data flow diagrams * Module organization * Testing status and next steps * ~3,000 lines of code documented
This commit is contained in:
HotSwapp
309
docs/IMPORT_GUIDE.md
Normal file
309
docs/IMPORT_GUIDE.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# CSV Import System - User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The CSV import system allows you to migrate legacy Paradox database data into the Delphi Database application. The system works in two stages:
|
||||
|
||||
1. **Import Stage**: Load CSV files into legacy database models (preserving original schema)
|
||||
2. **Sync Stage**: Synchronize legacy data to modern simplified models
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Admin access to the application
|
||||
- CSV files from the legacy Paradox database
|
||||
- Docker container running
|
||||
|
||||
## Import Order
|
||||
|
||||
**IMPORTANT**: Import tables in this exact order to avoid foreign key errors:
|
||||
|
||||
### Stage 1: Reference Tables (Import First)
|
||||
These tables contain lookup data used by other tables.
|
||||
|
||||
1. TRNSTYPE.csv - Transaction types
|
||||
2. TRNSLKUP.csv - Transaction lookup
|
||||
3. FOOTERS.csv - Footer templates
|
||||
4. FILESTAT.csv - File status codes
|
||||
5. EMPLOYEE.csv - Employee records
|
||||
6. GRUPLKUP.csv - Group lookup
|
||||
7. FILETYPE.csv - File type codes
|
||||
8. FVARLKUP.csv - File variable lookup
|
||||
9. RVARLKUP.csv - Rolodex variable lookup
|
||||
|
||||
### Stage 2: Core Data Tables
|
||||
|
||||
10. ROLODEX.csv - Client/contact information
|
||||
11. PHONE.csv - Phone numbers (references ROLODEX)
|
||||
12. ROLEX_V.csv - Rolodex variables (references ROLODEX)
|
||||
13. FILES.csv - Case/file records (references ROLODEX)
|
||||
14. FILES_R.csv - File relationships (references FILES)
|
||||
15. FILES_V.csv - File variables (references FILES)
|
||||
16. FILENOTS.csv - File notes/memos (references FILES)
|
||||
17. LEDGER.csv - Transaction ledger (references FILES)
|
||||
18. DEPOSITS.csv - Deposit records
|
||||
19. PAYMENTS.csv - Payment records (references FILES)
|
||||
|
||||
### Stage 3: Specialized Tables
|
||||
|
||||
20. PLANINFO.csv - Pension plan information
|
||||
21. QDROS.csv - QDRO documents (references FILES)
|
||||
22. PENSIONS.csv - Pension records (references FILES)
|
||||
23. Pensions/MARRIAGE.csv - Marriage calculations (references PENSIONS)
|
||||
24. Pensions/DEATH.csv - Death benefit calculations (references PENSIONS)
|
||||
25. Pensions/SCHEDULE.csv - Vesting schedules (references PENSIONS)
|
||||
26. Pensions/SEPARATE.csv - Separation calculations (references PENSIONS)
|
||||
27. Pensions/RESULTS.csv - Pension calculation results (references PENSIONS)
|
||||
|
||||
## Step-by-Step Import Process
|
||||
|
||||
### 1. Access Admin Panel
|
||||
|
||||
1. Navigate to `http://localhost:8000/admin`
|
||||
2. Login with admin credentials (default: admin/admin)
|
||||
|
||||
### 2. Upload CSV Files
|
||||
|
||||
1. Scroll to the **File Upload** section
|
||||
2. Click "Select CSV Files" and choose your CSV files
|
||||
3. You can upload multiple files at once
|
||||
4. Click "Upload Files"
|
||||
5. Review the upload results to ensure files were recognized correctly
|
||||
|
||||
### 3. Import Reference Tables First
|
||||
|
||||
1. Scroll to the **Data Import** section
|
||||
2. You'll see files grouped by type (trnstype, trnslkup, footers, etc.)
|
||||
3. For each reference table group:
|
||||
- Select the checkbox next to the file(s) you want to import
|
||||
- Click the "Import" button for that group
|
||||
- Wait for the import to complete
|
||||
- Review the results
|
||||
|
||||
**Tip**: Use the "Select All" button to quickly select all files in a group.
|
||||
|
||||
### 4. Import Core Data Tables
|
||||
|
||||
Following the same process as step 3, import the core tables in order:
|
||||
|
||||
1. Import **rolodex** files
|
||||
2. Import **phone** files
|
||||
3. Import **rolex_v** files
|
||||
4. Import **files** files
|
||||
5. Import **files_r** files
|
||||
6. Import **files_v** files
|
||||
7. Import **filenots** files
|
||||
8. Import **ledger** files
|
||||
9. Import **deposits** files
|
||||
10. Import **payments** files
|
||||
|
||||
### 5. Import Specialized Tables
|
||||
|
||||
Finally, import the specialized tables:
|
||||
|
||||
1. Import **planinfo** files
|
||||
2. Import **qdros** files
|
||||
3. Import **pensions** files
|
||||
4. Import **pension_marriage** files
|
||||
5. Import **pension_death** files
|
||||
6. Import **pension_schedule** files
|
||||
7. Import **pension_separate** files
|
||||
8. Import **pension_results** files
|
||||
|
||||
### 6. Sync to Modern Models
|
||||
|
||||
After all legacy data is imported:
|
||||
|
||||
1. Scroll to the **Sync to Modern Models** section
|
||||
2. **OPTIONAL**: Check "Clear existing modern data before sync" if you want to replace all current data
|
||||
- ⚠️ **WARNING**: This will delete all existing Client, Phone, Case, Transaction, Payment, and Document records!
|
||||
3. Click "Start Sync Process"
|
||||
4. Confirm the action in the dialog
|
||||
5. Review the sync results
|
||||
|
||||
The sync process will:
|
||||
- Convert Rolodex → Client
|
||||
- Convert LegacyPhone → Phone
|
||||
- Convert LegacyFile → Case
|
||||
- Convert Ledger → Transaction
|
||||
- Convert LegacyPayment → Payment
|
||||
- Convert Qdros → Document
|
||||
|
||||
## Monitoring Import Progress
|
||||
|
||||
### Import Results
|
||||
|
||||
After each import operation, you'll see:
|
||||
|
||||
- **Total Rows**: Number of rows in the CSV file
|
||||
- **Success Count**: Records successfully imported
|
||||
- **Error Count**: Records that failed to import
|
||||
- **Detailed Errors**: Specific error messages for failed records (first 10 shown)
|
||||
|
||||
### Import History
|
||||
|
||||
The **Recent Import History** section shows the last 10 import operations with:
|
||||
|
||||
- Import type
|
||||
- Filename
|
||||
- Status (Completed/Failed/Running)
|
||||
- Record counts
|
||||
- Timestamp
|
||||
|
||||
### Sync Results
|
||||
|
||||
After syncing, you'll see:
|
||||
|
||||
- **Records Synced**: Total records successfully synced to modern models
|
||||
- **Records Skipped**: Records that couldn't be synced (e.g., missing foreign keys)
|
||||
- **Errors**: Count of errors encountered
|
||||
- **Per-Table Details**: Breakdown showing results for each modern table (Client, Phone, Case, etc.)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Import Errors
|
||||
|
||||
**"Foreign key constraint failed"**
|
||||
- You imported tables out of order
|
||||
- Solution: Import reference tables first, then core tables
|
||||
|
||||
**"No client found for rolodex ID"**
|
||||
- The ROLODEX file wasn't imported before dependent files
|
||||
- Solution: Import ROLODEX.csv first
|
||||
|
||||
**"No case found for file"**
|
||||
- The FILES file wasn't imported before LEDGER/PAYMENTS
|
||||
- Solution: Import FILES.csv before LEDGER.csv and PAYMENTS.csv
|
||||
|
||||
**"Encoding error" or "Unable to open file"**
|
||||
- CSV file has unusual encoding
|
||||
- The system tries multiple encodings automatically
|
||||
- Check the error message for details
|
||||
|
||||
### Sync Errors
|
||||
|
||||
**"Records Skipped"**
|
||||
- Legacy records reference non-existent parent records
|
||||
- This is normal for incomplete datasets
|
||||
- Review skipped record details in the sync results
|
||||
|
||||
**"Duplicate key error"**
|
||||
- Running sync multiple times without clearing existing data
|
||||
- Solution: Check "Clear existing modern data before sync" option
|
||||
|
||||
## Data Validation
|
||||
|
||||
After importing and syncing, verify your data:
|
||||
|
||||
### Check Record Counts
|
||||
|
||||
1. Navigate to the Dashboard (`/`)
|
||||
2. Review the statistics cards showing counts for:
|
||||
- Total Clients
|
||||
- Active Cases
|
||||
- Total Transactions
|
||||
- Recent Payments
|
||||
|
||||
### Verify Relationships
|
||||
|
||||
1. Go to the Rolodex page (`/rolodex`)
|
||||
2. Click on a client
|
||||
3. Verify that:
|
||||
- Phone numbers are displayed correctly
|
||||
- Associated cases are shown
|
||||
- Case details link to transactions and payments
|
||||
|
||||
### Run Reports
|
||||
|
||||
Test the reporting functionality:
|
||||
|
||||
1. Generate a Phone Book report
|
||||
2. Generate a Payments report
|
||||
3. Verify data appears correctly in PDFs
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Backup First**: Always backup your database before importing
|
||||
2. **Test with Sample Data**: Test the import process with a small subset of data first
|
||||
3. **Import Order Matters**: Always follow the recommended import order
|
||||
4. **Review Errors**: Check import results carefully and address errors before proceeding
|
||||
5. **Sync Last**: Only run the sync process after all legacy data is successfully imported
|
||||
6. **Monitor Progress**: For large imports, monitor the Import History section
|
||||
7. **Document Issues**: Keep notes of any import errors for troubleshooting
|
||||
|
||||
## File Naming Conventions
|
||||
|
||||
The system recognizes files by their names. Supported patterns:
|
||||
|
||||
- `ROLODEX*.csv` → rolodex import type
|
||||
- `PHONE*.csv` → phone import type
|
||||
- `FILES*.csv` → files import type
|
||||
- `LEDGER*.csv` → ledger import type
|
||||
- `PAYMENTS*.csv` → payments import type
|
||||
- `TRNSTYPE*.csv` → trnstype import type
|
||||
- etc.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- **Batch Processing**: Imports process 500 records per batch for optimal performance
|
||||
- **Large Files**: Files with 10,000+ records may take several minutes
|
||||
- **Database Locks**: Only one import operation should run at a time
|
||||
- **Memory Usage**: Very large files (100,000+ records) may require increased Docker memory allocation
|
||||
|
||||
## Getting Help
|
||||
|
||||
If you encounter issues:
|
||||
|
||||
1. Check the application logs: `docker-compose logs delphi-db`
|
||||
2. Review the Import History for error details
|
||||
3. Verify your CSV file format matches the expected schema
|
||||
4. Consult the legacy schema documentation in `docs/legacy-schema.md`
|
||||
|
||||
## Example: Complete Import Sequence
|
||||
|
||||
```bash
|
||||
# 1. Start the application
|
||||
docker-compose up -d
|
||||
|
||||
# 2. Access admin panel
|
||||
# Navigate to http://localhost:8000/admin
|
||||
|
||||
# 3. Upload all CSV files at once
|
||||
# Use the file upload form to select all CSV files
|
||||
|
||||
# 4. Import in order:
|
||||
# - Reference tables (TRNSTYPE, TRNSLKUP, FOOTERS, etc.)
|
||||
# - Core data (ROLODEX, PHONE, FILES, LEDGER, etc.)
|
||||
# - Specialized (PLANINFO, QDROS, PENSIONS, etc.)
|
||||
|
||||
# 5. Sync to modern models
|
||||
# Click "Start Sync Process" with or without clearing existing data
|
||||
|
||||
# 6. Verify
|
||||
# Navigate to dashboard and verify record counts
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Import Module: `app/import_legacy.py`
|
||||
|
||||
Contains import functions for all legacy tables with:
|
||||
- Encoding detection (UTF-8, CP1252, Latin-1, etc.)
|
||||
- Date parsing (MM/DD/YYYY, MM/DD/YY, YYYY-MM-DD)
|
||||
- Decimal conversion with proper precision
|
||||
- Batch insert optimization
|
||||
- Structured logging
|
||||
|
||||
### Sync Module: `app/sync_legacy_to_modern.py`
|
||||
|
||||
Contains sync functions that:
|
||||
- Map legacy IDs to modern table IDs
|
||||
- Handle missing foreign key references gracefully
|
||||
- Skip orphaned records with warnings
|
||||
- Maintain referential integrity
|
||||
- Support incremental or full replacement sync
|
||||
|
||||
### Database Models
|
||||
|
||||
- **Legacy Models**: Preserve original Paradox schema (Rolodex, LegacyPhone, LegacyFile, Ledger, etc.)
|
||||
- **Modern Models**: Simplified application schema (Client, Phone, Case, Transaction, Payment, Document)
|
||||
|
||||
Reference in New Issue
Block a user