- Enhanced get_import_type_from_filename() to recognize model class names (LegacyFile, FilesR, etc.) in addition to legacy CSV names
- Added import functions for States, Printers, and Setup reference tables
- Updated VALID_IMPORT_TYPES and IMPORT_ORDER to include new tables
- Updated admin panel table counts to display new reference tables
- Created UPLOAD_FIX.md documentation explaining the changes and how to handle existing unknown files
This fixes the issue where files uploaded with model class names (e.g., LegacyFile.csv) were being categorized as 'unknown' instead of being properly detected.
- Track skipped_no_phone and skipped_no_id separately
- Display skip information in admin UI with warning icon
- Clarify that empty phone numbers cannot be imported (PK constraint)
- Update documentation to explain expected skip behavior
- Example: 143 rows without phone numbers is correct, not an error
When importing PHONE.csv with empty phone numbers:
- Rows are properly skipped (cannot have NULL in primary key)
- User sees: '⚠️ Skipped: 143 rows without phone number'
- This is expected behavior, not a bug
- Changed encoding fallback order to prioritize iso-8859-1/latin-1 over cp1252
- Increased encoding test from 1KB to 10KB to catch issues deeper in files
- Added proper file handle cleanup on encoding failures
- Resolves 'charmap codec can't decode byte 0x9d' error in rolodex import
- Tested with rolodex file containing 52,100 rows successfully
- New POST /admin/map-files to reclassify unknown files to a chosen import type
- Centralize VALID_IMPORT_TYPES and pass to admin template
- UI: dropdown + 'Map Selected' button in Unknown card
- JS: mapSelectedFiles() posts selection and reloads on success
- Keeps UUID suffix, prevents traversal, logs actions
- Added delete button (trash icon) next to each uploaded file in the import section
- Implemented DELETE endpoint at /admin/delete-file/{filename} with authentication and validation
- Added JavaScript function to handle file deletion with confirmation dialog
- Includes security checks for directory traversal and file existence
- Logs file deletion actions with username for audit trail
- UI automatically refreshes after successful deletion
- Added table_counts query in /admin route to get record counts for all tables
* Reference tables (TrnsType, TrnsLkup, Footers, FileStat, Employee, etc.)
* Core data tables (Rolodex, LegacyPhone, LegacyFile, Ledger, etc.)
* Specialized tables (PlanInfo, Qdros, Pensions, etc.)
* Modern models (Client, Phone, Case, Transaction, Payment, Document)
- Created Database Status UI section in admin.html
* Four-column layout showing all table categories
* Color-coded badges (green=has data, gray=empty)
* Check mark icons for populated tables
* Table row highlighting based on data presence
* Legend explaining the visual indicators
- Helps users track import progress at a glance
- Shows which tables have been successfully imported
- Distinguishes between legacy and modern model data
- Added 5 new legacy models to app/models.py (FileType, FileNots, RolexV, FVarLkup, RVarLkup)
- Created app/import_legacy.py with import functions for all legacy tables:
* Reference tables: TRNSTYPE, TRNSLKUP, FOOTERS, FILESTAT, EMPLOYEE, GRUPLKUP, FILETYPE, FVARLKUP, RVARLKUP
* Core tables: ROLODEX, PHONE, ROLEX_V, FILES, FILES_R, FILES_V, FILENOTS, LEDGER, DEPOSITS, PAYMENTS
* Specialized: PLANINFO, QDROS, PENSIONS and all pension-related tables
- Created app/sync_legacy_to_modern.py with sync functions to populate modern models from legacy data
- Updated admin routes in app/main.py:
* Extended process_csv_import to support all new import types
* Added /admin/sync endpoint for syncing legacy to modern models
* Updated get_import_type_from_filename to recognize all CSV file patterns
- Enhanced app/templates/admin.html with:
* Import Order Guide showing recommended import sequence
* Sync to Modern Models section with confirmation dialog
* Sync results display with detailed per-table statistics
* Updated supported file formats list
- All import functions use batch processing (500 rows), proper error handling, and structured logging
- Sync functions maintain foreign key integrity and skip orphaned records with warnings
- Added windows-1252, cp1250, iso-8859-1 to encoding fallback list
- Enhanced error logging in open_text_with_fallbacks function
- Improved error messages to show all attempted encodings
- Added warning logs for each encoding attempt that fails
This should resolve 'charmap' codec errors and other encoding issues with legacy CSV files that use different Windows codepages or ISO encodings.
- Updated import_phone_data to use open_text_with_fallbacks for encoding support
- Updated import_files_data to use open_text_with_fallbacks for encoding support
- Updated import_ledger_data to use open_text_with_fallbacks for encoding support
- Updated import_qdros_data to use open_text_with_fallbacks for encoding support
- Updated import_payments_data to use open_text_with_fallbacks for encoding support
All CSV import functions now use the same encoding fallback pattern that tries utf-8, utf-8-sig, cp1252, and latin-1 encodings to handle legacy CSV files with different encodings.
- PDF builders in app/reporting.py (envelope, phone+address, rolodex info)
- Endpoints in app/main.py with auth, filtering, logging, Content-Disposition
- New HTML template report_phone_book_address.html
- Rolodex bulk actions updated with buttons/links
- JS helper to submit selections to alternate endpoints
Tested via docker compose build/up and health check.
- Add ImportLog model for tracking import history and results
- Create admin.html template with file upload form and progress display
- Implement POST /admin/upload route for CSV file handling with validation
- Build CSV import engine with dispatcher routing by filename patterns:
* ROLODEX*.csv → Client model import
* PHONE*.csv → Phone model import with client linking
* FILES*.csv → Case model import
* LEDGER*.csv → Transaction model import
* QDROS*.csv → Document model import
* PAYMENTS*.csv → Payment model import
- Add POST /admin/import/{data_type} route for triggering imports
- Implement comprehensive validation, error handling, and progress tracking
- Support for CSV header validation, data type conversions, and duplicate handling
- Real-time progress tracking with ImportLog database model
- Responsive UI with Bootstrap components for upload and results display
- Enhanced navigation with admin panel link already in place
- Tested import functionality with validation and error handling
The admin panel enables bulk importing of legacy CSV data from the old-csv/ directory, making the system fully functional with real data.
- Add POST /case/{id}/update route for editing case fields (status, case_type, description, open_date, close_date)
- Add POST /case/{id}/close route to set status='closed' and close_date=current date
- Add POST /case/{id}/reopen route to set status='active' and clear close_date
- Update case.html template with edit form, success/error messaging, and action buttons
- Include comprehensive validation for dates and status values
- Add proper error handling with session-based error storage
- Preserve existing view content and styling consistency
- Add SECRET_KEY environment variable and .env file for session management
- Configure SessionMiddleware with FastAPI for user session handling
- Set up Jinja2 template engine with template directory configuration
- Mount static files directory for CSS, JS, and image assets
- Create comprehensive base.html template with Bootstrap 5 CDN
- Add Bootstrap Icons and custom styling support
- Include responsive navigation with user authentication state
- Create placeholder CSS and JavaScript files for customization
- Add aiofiles dependency for static file serving
This establishes the web framework foundation with session management
and templating system ready for frontend development.
- Created app/database.py with SQLAlchemy engine, session management, and connection utilities
- Added comprehensive database models in app/models.py for User, Client, Phone, Case, Transaction, Document, and Payment
- Implemented FastAPI application with database lifecycle management in app/main.py
- Added health check endpoint to verify database connectivity
- Created README.md with database configuration documentation
- Verified database connection works correctly with SQLite backend
