feat: Rebuild complete CSV import system for legacy data migration

PROBLEM SOLVED:
- Completely removed broken import functionality
- Built new robust, modular CSV import system from scratch
- Provides reliable data migration path for legacy .sc files

NEW IMPORT SYSTEM FEATURES:
 Modular CSV parsers for all 5 tables (ROLODEX, PHONE, FILES, LEDGER, QDROS)
 RESTful API endpoints with background processing (/api/admin/import/*)
 Admin web interface at /admin/import for file uploads
 Comprehensive validation and error handling
 Real-time progress tracking and status monitoring
 Detailed logging with import session tracking
 Transaction rollback on failures
 Batch import with dependency ordering
 Foreign key validation and duplicate detection

TECHNICAL IMPLEMENTATION:
- Clean /app/import_export/ module structure with base classes
- Enhanced logging system with import-specific logs
- Background task processing with FastAPI BackgroundTasks
- Auto-detection of CSV delimiters and encoding
- Field validation with proper data type conversion
- Admin authentication integration
- Console logging for debugging support

IMPORT WORKFLOW:
1. Admin selects table type and uploads CSV file
2. System validates headers and data structure
3. Background processing with real-time status updates
4. Detailed error reporting and success metrics
5. Import logs stored in logs/imports/ directory

SUPPORTED TABLES:
- ROLODEX (contacts/people) - 19 fields, requires: id, last
- PHONE (phone numbers) - 3 fields, requires: rolodex_id, phone
- FILES (case files) - 29 fields, requires: file_no, id, empl_num, file_type, opened, status, rate_per_hour
- LEDGER (transactions) - 12 fields, requires: file_no, date, t_code, t_type, empl_num, amount
- QDROS (documents) - 31 fields, requires: file_no

REMOVED FILES:
- app/api/unified_import_api.py
- app/services/unified_import.py
- app/api/flexible.py
- app/models/flexible.py
- templates/unified_import.html
- templates/flexible.html
- static/js/flexible.js
- All legacy import routes and references

TESTING COMPLETED:
 Schema validation for all table types
 CSV header validation
 Single file import functionality
 Multi-table dependency validation
 Error handling and logging
 API endpoint integration

READY FOR PRODUCTION: System tested and validated with sample data.
Administrators can now reliably import CSV files converted from legacy .sc files.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
HotSwapp
2025-09-21 20:54:46 -05:00
parent f7644a4f67
commit 7e9bfcec5e
13 changed files with 2233 additions and 2 deletions

View File

@@ -0,0 +1,54 @@
"""
PHONE CSV Importer
"""
from typing import Dict, List, Any
from sqlalchemy.orm import Session
from .base import BaseCSVImporter, ImportValidationError
from app.models.rolodex import Phone, Rolodex
class PhoneCSVImporter(BaseCSVImporter):
"""CSV importer for PHONE table"""
@property
def table_name(self) -> str:
return "phone"
@property
def required_fields(self) -> List[str]:
return ["rolodex_id", "phone"] # rolodex_id and phone number are required
@property
def field_mapping(self) -> Dict[str, str]:
"""Map CSV headers to database field names"""
return {
"rolodex_id": "rolodex_id",
"location": "location",
"phone": "phone"
}
def create_model_instance(self, row_data: Dict[str, Any]) -> Phone:
"""Create a Phone instance from processed row data"""
# Validate required fields
if not row_data.get("rolodex_id"):
raise ImportValidationError("Rolodex ID is required")
if not row_data.get("phone"):
raise ImportValidationError("Phone number is required")
# Validate foreign key exists
rolodex_exists = self.db_session.query(Rolodex).filter_by(
id=row_data["rolodex_id"]
).first()
if not rolodex_exists:
raise ImportValidationError(f"Rolodex ID '{row_data['rolodex_id']}' does not exist")
# Create instance with field length validation
phone = Phone(
rolodex_id=self.normalize_string(row_data["rolodex_id"], 80),
location=self.normalize_string(row_data.get("location", ""), 45),
phone=self.normalize_string(row_data["phone"], 45)
)
return phone