- Increased read size from 20KB to 50KB in both main.py and import_legacy.py
- This ensures problematic bytes at position 3738+ are caught during encoding detection
- Provides maximum robustness for legacy CSV files with deeply embedded encoding issues
- Maintains all previous improvements including fallback mechanisms
- Increased encoding detection read size from 10KB to 20KB in both main.py and import_legacy.py
- This ensures problematic bytes deeper in files (like position 3738) are caught during encoding detection
- Maintains backwards compatibility with properly encoded UTF-8 files
- Provides more robust handling of legacy CSV files with mixed encodings
- Updated open_text_with_fallbacks() in both main.py and import_legacy.py
- Increased fallback read size from 1KB to 10KB to catch encoding issues deeper in files
- Added proper fallback mechanism to main.py version of the function
- This fixes the 'utf-8' codec can't decode byte 0xa9 error when uploading planinfo.csv files
- Added duplicate detection and handling for pensions, pension_death, pension_separate, and pension_results imports
- Tracks (file_no, version) composite keys in-memory during import
- Checks database for existing records before insert
- Handles IntegrityError gracefully with fallback to row-by-row insertion
- Returns 'skipped' count in import results
- Prevents transaction rollback cascades that previously caused all subsequent rows to fail
- Consistent with existing rolodex duplicate handling pattern
- Enhanced get_import_type_from_filename() to recognize model class names (LegacyFile, FilesR, etc.) in addition to legacy CSV names
- Added import functions for States, Printers, and Setup reference tables
- Updated VALID_IMPORT_TYPES and IMPORT_ORDER to include new tables
- Updated admin panel table counts to display new reference tables
- Created UPLOAD_FIX.md documentation explaining the changes and how to handle existing unknown files
This fixes the issue where files uploaded with model class names (e.g., LegacyFile.csv) were being categorized as 'unknown' instead of being properly detected.
- Track skipped_no_phone and skipped_no_id separately
- Display skip information in admin UI with warning icon
- Clarify that empty phone numbers cannot be imported (PK constraint)
- Update documentation to explain expected skip behavior
- Example: 143 rows without phone numbers is correct, not an error
When importing PHONE.csv with empty phone numbers:
- Rows are properly skipped (cannot have NULL in primary key)
- User sees: '⚠️ Skipped: 143 rows without phone number'
- This is expected behavior, not a bug
- Implement upsert logic in import_phone() function
- Check for existing (id, phone) combinations before insert
- Track duplicates within CSV to skip gracefully
- Update existing records instead of failing on duplicates
- Add detailed statistics: inserted, updated, skipped counts
- Align with upsert pattern used in other import functions
- Add documentation in docs/PHONE_IMPORT_FIX.md
Fixes: UNIQUE constraint failed: phone.id, phone.phone error
when re-importing or uploading CSV with duplicate entries
- Implement upsert (INSERT or UPDATE) logic for all reference table imports
- Fixed functions: import_trnstype, import_trnslkup, import_footers,
import_filestat, import_employee, import_gruplkup, import_filetype,
import_fvarlkup, import_rvarlkup
- Now checks if record exists before inserting; updates if exists
- Makes imports idempotent - can safely re-run without errors
- Added tracking of inserted vs updated counts in result dict
- Maintains batch commit performance for large imports
- Fixes sqlite3.IntegrityError when re-importing CSV files
- Added duplicate tracking within import session (seen_in_import set)
- Skip records that already exist in database
- Added fallback to row-by-row insert when bulk insert fails
- Track skipped records in result
- Prevents cascade errors after UNIQUE constraint violation
- Gracefully handles legacy data with duplicate IDs
- Changed encoding fallback order to prioritize iso-8859-1/latin-1 over cp1252
- Increased encoding test from 1KB to 10KB to catch issues deeper in files
- Added proper file handle cleanup on encoding failures
- Resolves 'charmap codec can't decode byte 0x9d' error in rolodex import
- Tested with rolodex file containing 52,100 rows successfully
- Enhanced open_text_with_fallbacks() function to handle problematic bytes
- Added CP1250 encoding to fallback list for better character set support
- Added graceful error handling with replacement characters for edge cases
- Ensures rolodex CSV import works with legacy encoding issues
Fixes: 'charmap' codec can't decode byte 0x9d error during rolodex import
- Added 5 new legacy models to app/models.py (FileType, FileNots, RolexV, FVarLkup, RVarLkup)
- Created app/import_legacy.py with import functions for all legacy tables:
* Reference tables: TRNSTYPE, TRNSLKUP, FOOTERS, FILESTAT, EMPLOYEE, GRUPLKUP, FILETYPE, FVARLKUP, RVARLKUP
* Core tables: ROLODEX, PHONE, ROLEX_V, FILES, FILES_R, FILES_V, FILENOTS, LEDGER, DEPOSITS, PAYMENTS
* Specialized: PLANINFO, QDROS, PENSIONS and all pension-related tables
- Created app/sync_legacy_to_modern.py with sync functions to populate modern models from legacy data
- Updated admin routes in app/main.py:
* Extended process_csv_import to support all new import types
* Added /admin/sync endpoint for syncing legacy to modern models
* Updated get_import_type_from_filename to recognize all CSV file patterns
- Enhanced app/templates/admin.html with:
* Import Order Guide showing recommended import sequence
* Sync to Modern Models section with confirmation dialog
* Sync results display with detailed per-table statistics
* Updated supported file formats list
- All import functions use batch processing (500 rows), proper error handling, and structured logging
- Sync functions maintain foreign key integrity and skip orphaned records with warnings
