- Increased read size from 20KB to 50KB in both main.py and import_legacy.py
- This ensures problematic bytes at position 3738+ are caught during encoding detection
- Provides maximum robustness for legacy CSV files with deeply embedded encoding issues
- Maintains all previous improvements including fallback mechanisms
- Increased encoding detection read size from 10KB to 20KB in both main.py and import_legacy.py
- This ensures problematic bytes deeper in files (like position 3738) are caught during encoding detection
- Maintains backwards compatibility with properly encoded UTF-8 files
- Provides more robust handling of legacy CSV files with mixed encodings
- Updated open_text_with_fallbacks() in both main.py and import_legacy.py
- Increased fallback read size from 1KB to 10KB to catch encoding issues deeper in files
- Added proper fallback mechanism to main.py version of the function
- This fixes the 'utf-8' codec can't decode byte 0xa9 error when uploading planinfo.csv files
The import_log.status was incorrectly set to 'failed' when there were NO errors.
The condition 'if result["errors"]' evaluates to False when errors list is empty,
causing the logic to be inverted.
Fixed: 'completed' if not result['errors'] else 'failed'
This caused successful imports with 0 errors to show as 'Failed' in the UI.
- Document composite primary key handling for pension tables
- Add code examples for both single and composite key duplicate detection
- List all pension-related tables with duplicate protection
- Added duplicate detection and handling for pensions, pension_death, pension_separate, and pension_results imports
- Tracks (file_no, version) composite keys in-memory during import
- Checks database for existing records before insert
- Handles IntegrityError gracefully with fallback to row-by-row insertion
- Returns 'skipped' count in import results
- Prevents transaction rollback cascades that previously caused all subsequent rows to fail
- Consistent with existing rolodex duplicate handling pattern
- Enhanced get_import_type_from_filename() to recognize model class names (LegacyFile, FilesR, etc.) in addition to legacy CSV names
- Added import functions for States, Printers, and Setup reference tables
- Updated VALID_IMPORT_TYPES and IMPORT_ORDER to include new tables
- Updated admin panel table counts to display new reference tables
- Created UPLOAD_FIX.md documentation explaining the changes and how to handle existing unknown files
This fixes the issue where files uploaded with model class names (e.g., LegacyFile.csv) were being categorized as 'unknown' instead of being properly detected.
- Changed primary key from composite (file_no, version) to auto-increment id
- A pension can have multiple vesting schedule entries (e.g., vests 20% at year 1, 100% at year 5)
- Added index on (file_no, version) for efficient lookups
- Successfully imported 502 vesting schedule entries for 416 unique pensions
- Some pensions have up to 6 vesting milestones
- Track skipped_no_phone and skipped_no_id separately
- Display skip information in admin UI with warning icon
- Clarify that empty phone numbers cannot be imported (PK constraint)
- Update documentation to explain expected skip behavior
- Example: 143 rows without phone numbers is correct, not an error
When importing PHONE.csv with empty phone numbers:
- Rows are properly skipped (cannot have NULL in primary key)
- User sees: '⚠️ Skipped: 143 rows without phone number'
- This is expected behavior, not a bug
- Implement upsert logic in import_phone() function
- Check for existing (id, phone) combinations before insert
- Track duplicates within CSV to skip gracefully
- Update existing records instead of failing on duplicates
- Add detailed statistics: inserted, updated, skipped counts
- Align with upsert pattern used in other import functions
- Add documentation in docs/PHONE_IMPORT_FIX.md
Fixes: UNIQUE constraint failed: phone.id, phone.phone error
when re-importing or uploading CSV with duplicate entries
- Implement upsert (INSERT or UPDATE) logic for all reference table imports
- Fixed functions: import_trnstype, import_trnslkup, import_footers,
import_filestat, import_employee, import_gruplkup, import_filetype,
import_fvarlkup, import_rvarlkup
- Now checks if record exists before inserting; updates if exists
- Makes imports idempotent - can safely re-run without errors
- Added tracking of inserted vs updated counts in result dict
- Maintains batch commit performance for large imports
- Fixes sqlite3.IntegrityError when re-importing CSV files
- Added duplicate tracking within import session (seen_in_import set)
- Skip records that already exist in database
- Added fallback to row-by-row insert when bulk insert fails
- Track skipped records in result
- Prevents cascade errors after UNIQUE constraint violation
- Gracefully handles legacy data with duplicate IDs
- Changed encoding fallback order to prioritize iso-8859-1/latin-1 over cp1252
- Increased encoding test from 1KB to 10KB to catch issues deeper in files
- Added proper file handle cleanup on encoding failures
- Resolves 'charmap codec can't decode byte 0x9d' error in rolodex import
- Tested with rolodex file containing 52,100 rows successfully
- Enhanced open_text_with_fallbacks() function to handle problematic bytes
- Added CP1250 encoding to fallback list for better character set support
- Added graceful error handling with replacement characters for edge cases
- Ensures rolodex CSV import works with legacy encoding issues
Fixes: 'charmap' codec can't decode byte 0x9d error during rolodex import
- New POST /admin/map-files to reclassify unknown files to a chosen import type
- Centralize VALID_IMPORT_TYPES and pass to admin template
- UI: dropdown + 'Map Selected' button in Unknown card
- JS: mapSelectedFiles() posts selection and reloads on success
- Keeps UUID suffix, prevents traversal, logs actions
- Added delete button (trash icon) next to each uploaded file in the import section
- Implemented DELETE endpoint at /admin/delete-file/{filename} with authentication and validation
- Added JavaScript function to handle file deletion with confirmation dialog
- Includes security checks for directory traversal and file existence
- Logs file deletion actions with username for audit trail
- UI automatically refreshes after successful deletion
- Added table_counts query in /admin route to get record counts for all tables
* Reference tables (TrnsType, TrnsLkup, Footers, FileStat, Employee, etc.)
* Core data tables (Rolodex, LegacyPhone, LegacyFile, Ledger, etc.)
* Specialized tables (PlanInfo, Qdros, Pensions, etc.)
* Modern models (Client, Phone, Case, Transaction, Payment, Document)
- Created Database Status UI section in admin.html
* Four-column layout showing all table categories
* Color-coded badges (green=has data, gray=empty)
* Check mark icons for populated tables
* Table row highlighting based on data presence
* Legend explaining the visual indicators
- Helps users track import progress at a glance
- Shows which tables have been successfully imported
- Distinguishes between legacy and modern model data
- Created IMPORT_GUIDE.md: Complete user guide with step-by-step instructions
* Detailed import order for all 27+ tables
* Troubleshooting guide
* Data validation procedures
* Best practices and performance notes
- Created IMPORT_SYSTEM_SUMMARY.md: Technical implementation summary
* Complete list of all implemented functions (28 import + 7 sync)
* Architecture and data flow diagrams
* Module organization
* Testing status and next steps
* ~3,000 lines of code documented
- Added 5 new legacy models to app/models.py (FileType, FileNots, RolexV, FVarLkup, RVarLkup)
- Created app/import_legacy.py with import functions for all legacy tables:
* Reference tables: TRNSTYPE, TRNSLKUP, FOOTERS, FILESTAT, EMPLOYEE, GRUPLKUP, FILETYPE, FVARLKUP, RVARLKUP
* Core tables: ROLODEX, PHONE, ROLEX_V, FILES, FILES_R, FILES_V, FILENOTS, LEDGER, DEPOSITS, PAYMENTS
* Specialized: PLANINFO, QDROS, PENSIONS and all pension-related tables
- Created app/sync_legacy_to_modern.py with sync functions to populate modern models from legacy data
- Updated admin routes in app/main.py:
* Extended process_csv_import to support all new import types
* Added /admin/sync endpoint for syncing legacy to modern models
* Updated get_import_type_from_filename to recognize all CSV file patterns
- Enhanced app/templates/admin.html with:
* Import Order Guide showing recommended import sequence
* Sync to Modern Models section with confirmation dialog
* Sync results display with detailed per-table statistics
* Updated supported file formats list
- All import functions use batch processing (500 rows), proper error handling, and structured logging
- Sync functions maintain foreign key integrity and skip orphaned records with warnings
- Added windows-1252, cp1250, iso-8859-1 to encoding fallback list
- Enhanced error logging in open_text_with_fallbacks function
- Improved error messages to show all attempted encodings
- Added warning logs for each encoding attempt that fails
This should resolve 'charmap' codec errors and other encoding issues with legacy CSV files that use different Windows codepages or ISO encodings.
- Updated import_phone_data to use open_text_with_fallbacks for encoding support
- Updated import_files_data to use open_text_with_fallbacks for encoding support
- Updated import_ledger_data to use open_text_with_fallbacks for encoding support
- Updated import_qdros_data to use open_text_with_fallbacks for encoding support
- Updated import_payments_data to use open_text_with_fallbacks for encoding support
All CSV import functions now use the same encoding fallback pattern that tries utf-8, utf-8-sig, cp1252, and latin-1 encodings to handle legacy CSV files with different encodings.
- Added 'Stored Filename' column to Upload Results table showing the actual filename used for storage
- Added 'Select All' button for each import type section to quickly select/deselect all files
- Improved JavaScript to handle select all/deselect all functionality with proper button state management
- Enhanced UI to clearly distinguish between original and stored filenames
- Increased logo size from 60x60 to 120x120px with proper styling
- Enhanced card layout with better padding and rounded corners
- Added modern gradient background and improved visual hierarchy
- Improved form styling with larger inputs and better spacing
- Enhanced password visibility toggle with better UX
- Improved error message styling with icons and rounded corners
- Added responsive design improvements for better mobile experience
- Updated color scheme with modern gradients and improved contrast
- PDF builders in app/reporting.py (envelope, phone+address, rolodex info)
- Endpoints in app/main.py with auth, filtering, logging, Content-Disposition
- New HTML template report_phone_book_address.html
- Rolodex bulk actions updated with buttons/links
- JS helper to submit selections to alternate endpoints
Tested via docker compose build/up and health check.
