- Implement upsert (INSERT or UPDATE) logic for all reference table imports
- Fixed functions: import_trnstype, import_trnslkup, import_footers,
import_filestat, import_employee, import_gruplkup, import_filetype,
import_fvarlkup, import_rvarlkup
- Now checks if record exists before inserting; updates if exists
- Makes imports idempotent - can safely re-run without errors
- Added tracking of inserted vs updated counts in result dict
- Maintains batch commit performance for large imports
- Fixes sqlite3.IntegrityError when re-importing CSV files
- Added duplicate tracking within import session (seen_in_import set)
- Skip records that already exist in database
- Added fallback to row-by-row insert when bulk insert fails
- Track skipped records in result
- Prevents cascade errors after UNIQUE constraint violation
- Gracefully handles legacy data with duplicate IDs
- Changed encoding fallback order to prioritize iso-8859-1/latin-1 over cp1252
- Increased encoding test from 1KB to 10KB to catch issues deeper in files
- Added proper file handle cleanup on encoding failures
- Resolves 'charmap codec can't decode byte 0x9d' error in rolodex import
- Tested with rolodex file containing 52,100 rows successfully
- Enhanced open_text_with_fallbacks() function to handle problematic bytes
- Added CP1250 encoding to fallback list for better character set support
- Added graceful error handling with replacement characters for edge cases
- Ensures rolodex CSV import works with legacy encoding issues
Fixes: 'charmap' codec can't decode byte 0x9d error during rolodex import
- Added 5 new legacy models to app/models.py (FileType, FileNots, RolexV, FVarLkup, RVarLkup)
- Created app/import_legacy.py with import functions for all legacy tables:
* Reference tables: TRNSTYPE, TRNSLKUP, FOOTERS, FILESTAT, EMPLOYEE, GRUPLKUP, FILETYPE, FVARLKUP, RVARLKUP
* Core tables: ROLODEX, PHONE, ROLEX_V, FILES, FILES_R, FILES_V, FILENOTS, LEDGER, DEPOSITS, PAYMENTS
* Specialized: PLANINFO, QDROS, PENSIONS and all pension-related tables
- Created app/sync_legacy_to_modern.py with sync functions to populate modern models from legacy data
- Updated admin routes in app/main.py:
* Extended process_csv_import to support all new import types
* Added /admin/sync endpoint for syncing legacy to modern models
* Updated get_import_type_from_filename to recognize all CSV file patterns
- Enhanced app/templates/admin.html with:
* Import Order Guide showing recommended import sequence
* Sync to Modern Models section with confirmation dialog
* Sync results display with detailed per-table statistics
* Updated supported file formats list
- All import functions use batch processing (500 rows), proper error handling, and structured logging
- Sync functions maintain foreign key integrity and skip orphaned records with warnings
