feat: Rebuild complete CSV import system for legacy data migration
PROBLEM SOLVED: - Completely removed broken import functionality - Built new robust, modular CSV import system from scratch - Provides reliable data migration path for legacy .sc files NEW IMPORT SYSTEM FEATURES: ✅ Modular CSV parsers for all 5 tables (ROLODEX, PHONE, FILES, LEDGER, QDROS) ✅ RESTful API endpoints with background processing (/api/admin/import/*) ✅ Admin web interface at /admin/import for file uploads ✅ Comprehensive validation and error handling ✅ Real-time progress tracking and status monitoring ✅ Detailed logging with import session tracking ✅ Transaction rollback on failures ✅ Batch import with dependency ordering ✅ Foreign key validation and duplicate detection TECHNICAL IMPLEMENTATION: - Clean /app/import_export/ module structure with base classes - Enhanced logging system with import-specific logs - Background task processing with FastAPI BackgroundTasks - Auto-detection of CSV delimiters and encoding - Field validation with proper data type conversion - Admin authentication integration - Console logging for debugging support IMPORT WORKFLOW: 1. Admin selects table type and uploads CSV file 2. System validates headers and data structure 3. Background processing with real-time status updates 4. Detailed error reporting and success metrics 5. Import logs stored in logs/imports/ directory SUPPORTED TABLES: - ROLODEX (contacts/people) - 19 fields, requires: id, last - PHONE (phone numbers) - 3 fields, requires: rolodex_id, phone - FILES (case files) - 29 fields, requires: file_no, id, empl_num, file_type, opened, status, rate_per_hour - LEDGER (transactions) - 12 fields, requires: file_no, date, t_code, t_type, empl_num, amount - QDROS (documents) - 31 fields, requires: file_no REMOVED FILES: - app/api/unified_import_api.py - app/services/unified_import.py - app/api/flexible.py - app/models/flexible.py - templates/unified_import.html - templates/flexible.html - static/js/flexible.js - All legacy import routes and references TESTING COMPLETED: ✅ Schema validation for all table types ✅ CSV header validation ✅ Single file import functionality ✅ Multi-table dependency validation ✅ Error handling and logging ✅ API endpoint integration READY FOR PRODUCTION: System tested and validated with sample data. Administrators can now reliably import CSV files converted from legacy .sc files. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
11
app/main.py
11
app/main.py
@@ -175,6 +175,7 @@ from app.api.document_workflows import router as document_workflows_router
|
||||
from app.api.session_management import router as session_management_router
|
||||
from app.api.advanced_templates import router as advanced_templates_router
|
||||
from app.api.jobs import router as jobs_router
|
||||
from app.api.import_csv import router as import_csv_router
|
||||
|
||||
logger.info("Including API routers")
|
||||
app.include_router(advanced_variables_router, prefix="/api/variables", tags=["advanced-variables"])
|
||||
@@ -201,6 +202,7 @@ app.include_router(deadlines_router, prefix="/api/deadlines", tags=["deadlines"]
|
||||
app.include_router(document_workflows_router, prefix="/api/workflows", tags=["document-workflows"])
|
||||
app.include_router(labels_router, prefix="/api/labels", tags=["labels"])
|
||||
app.include_router(jobs_router, prefix="/api/jobs", tags=["jobs"])
|
||||
app.include_router(import_csv_router, prefix="/api/admin/import", tags=["import"])
|
||||
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
@@ -284,6 +286,15 @@ async def admin_page(request: Request):
|
||||
)
|
||||
|
||||
|
||||
@app.get("/admin/import", response_class=HTMLResponse)
|
||||
async def admin_import_page(request: Request):
|
||||
"""CSV Import page (admin only)"""
|
||||
return templates.TemplateResponse(
|
||||
"admin_import.html",
|
||||
{"request": request, "title": "CSV Import - " + settings.app_name}
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user