feat: Rebuild complete CSV import system for legacy data migration
PROBLEM SOLVED: - Completely removed broken import functionality - Built new robust, modular CSV import system from scratch - Provides reliable data migration path for legacy .sc files NEW IMPORT SYSTEM FEATURES: ✅ Modular CSV parsers for all 5 tables (ROLODEX, PHONE, FILES, LEDGER, QDROS) ✅ RESTful API endpoints with background processing (/api/admin/import/*) ✅ Admin web interface at /admin/import for file uploads ✅ Comprehensive validation and error handling ✅ Real-time progress tracking and status monitoring ✅ Detailed logging with import session tracking ✅ Transaction rollback on failures ✅ Batch import with dependency ordering ✅ Foreign key validation and duplicate detection TECHNICAL IMPLEMENTATION: - Clean /app/import_export/ module structure with base classes - Enhanced logging system with import-specific logs - Background task processing with FastAPI BackgroundTasks - Auto-detection of CSV delimiters and encoding - Field validation with proper data type conversion - Admin authentication integration - Console logging for debugging support IMPORT WORKFLOW: 1. Admin selects table type and uploads CSV file 2. System validates headers and data structure 3. Background processing with real-time status updates 4. Detailed error reporting and success metrics 5. Import logs stored in logs/imports/ directory SUPPORTED TABLES: - ROLODEX (contacts/people) - 19 fields, requires: id, last - PHONE (phone numbers) - 3 fields, requires: rolodex_id, phone - FILES (case files) - 29 fields, requires: file_no, id, empl_num, file_type, opened, status, rate_per_hour - LEDGER (transactions) - 12 fields, requires: file_no, date, t_code, t_type, empl_num, amount - QDROS (documents) - 31 fields, requires: file_no REMOVED FILES: - app/api/unified_import_api.py - app/services/unified_import.py - app/api/flexible.py - app/models/flexible.py - templates/unified_import.html - templates/flexible.html - static/js/flexible.js - All legacy import routes and references TESTING COMPLETED: ✅ Schema validation for all table types ✅ CSV header validation ✅ Single file import functionality ✅ Multi-table dependency validation ✅ Error handling and logging ✅ API endpoint integration READY FOR PRODUCTION: System tested and validated with sample data. Administrators can now reliably import CSV files converted from legacy .sc files. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
HotSwapp
133
templates/admin_import.html
Normal file
133
templates/admin_import.html
Normal file
@@ -0,0 +1,133 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{{ title }}</title>
|
||||
<link href="/static/css/output.css" rel="stylesheet">
|
||||
<script src="/static/js/http.js"></script>
|
||||
<script src="/static/js/alerts.js"></script>
|
||||
<script src="/static/js/main.js"></script>
|
||||
</head>
|
||||
<body class="bg-gray-100">
|
||||
<div id="main-container" class="min-h-screen">
|
||||
<!-- Navigation -->
|
||||
<nav class="bg-blue-600 text-white shadow-lg">
|
||||
<div class="max-w-7xl mx-auto px-4">
|
||||
<div class="flex justify-between h-16">
|
||||
<div class="flex items-center">
|
||||
<h1 class="text-xl font-bold">Data Import</h1>
|
||||
</div>
|
||||
<div class="flex items-center space-x-4">
|
||||
<a href="/admin" class="hover:text-blue-200">← Back to Admin</a>
|
||||
<div id="userDropdown" class="relative">
|
||||
<button class="flex items-center space-x-2 hover:text-blue-200">
|
||||
<span>Loading...</span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</nav>
|
||||
|
||||
<!-- Main Content -->
|
||||
<div class="max-w-4xl mx-auto py-8 px-4">
|
||||
<!-- Page Header -->
|
||||
<div class="mb-8">
|
||||
<h2 class="text-3xl font-bold text-gray-900 mb-4">CSV Data Import</h2>
|
||||
<p class="text-gray-600">
|
||||
Import CSV files converted from legacy .sc files into the database.
|
||||
Please ensure your CSV files follow the correct format for each table.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Import Form -->
|
||||
<div class="bg-white rounded-lg shadow-md p-6 mb-8">
|
||||
<h3 class="text-xl font-semibold mb-4">Single File Import</h3>
|
||||
|
||||
<form id="importForm" class="space-y-4">
|
||||
<div>
|
||||
<label for="tableSelect" class="block text-sm font-medium text-gray-700 mb-2">
|
||||
Select Table Type
|
||||
</label>
|
||||
<select id="tableSelect" name="table_name"
|
||||
class="w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent">
|
||||
<option value="">-- Select Table --</option>
|
||||
<option value="rolodex">ROLODEX (Contacts)</option>
|
||||
<option value="phone">PHONE (Phone Numbers)</option>
|
||||
<option value="files">FILES (Case Files)</option>
|
||||
<option value="ledger">LEDGER (Financial Transactions)</option>
|
||||
<option value="qdros">QDROS (QDRO Documents)</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label for="csvFile" class="block text-sm font-medium text-gray-700 mb-2">
|
||||
Select CSV File
|
||||
</label>
|
||||
<input type="file" id="csvFile" name="file" accept=".csv"
|
||||
class="w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent">
|
||||
<p class="text-sm text-gray-500 mt-1">Only CSV files are supported. Max file size: 100MB</p>
|
||||
</div>
|
||||
|
||||
<div class="flex space-x-4">
|
||||
<button type="button" id="validateBtn"
|
||||
class="px-4 py-2 bg-yellow-600 text-white rounded-md hover:bg-yellow-700 focus:outline-none focus:ring-2 focus:ring-yellow-500 disabled:opacity-50">
|
||||
Validate Headers
|
||||
</button>
|
||||
<button type="submit" id="importBtn"
|
||||
class="px-4 py-2 bg-blue-600 text-white rounded-md hover:bg-blue-700 focus:outline-none focus:ring-2 focus:ring-blue-500 disabled:opacity-50">
|
||||
Import Data
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
<!-- Schema Information -->
|
||||
<div id="schemaInfo" class="bg-blue-50 rounded-lg p-6 mb-8 hidden">
|
||||
<h3 class="text-lg font-semibold text-blue-900 mb-3">Required Columns</h3>
|
||||
<div id="schemaDetails" class="text-sm text-blue-800"></div>
|
||||
</div>
|
||||
|
||||
<!-- Import Progress -->
|
||||
<div id="importProgress" class="bg-white rounded-lg shadow-md p-6 mb-8 hidden">
|
||||
<h3 class="text-xl font-semibold mb-4">Import Progress</h3>
|
||||
<div id="progressDetails"></div>
|
||||
</div>
|
||||
|
||||
<!-- Import Results -->
|
||||
<div id="importResults" class="bg-white rounded-lg shadow-md p-6 hidden">
|
||||
<h3 class="text-xl font-semibold mb-4">Import Results</h3>
|
||||
<div id="resultsContent"></div>
|
||||
</div>
|
||||
|
||||
<!-- Batch Import Section -->
|
||||
<div class="bg-white rounded-lg shadow-md p-6">
|
||||
<h3 class="text-xl font-semibold mb-4">Batch Import (Multiple Files)</h3>
|
||||
<p class="text-gray-600 mb-4">
|
||||
Import multiple CSV files at once. Files will be processed in dependency order
|
||||
(ROLODEX → PHONE/FILES → LEDGER/QDROS).
|
||||
</p>
|
||||
|
||||
<div id="batchFiles" class="space-y-3">
|
||||
<!-- Batch file inputs will be added here -->
|
||||
</div>
|
||||
|
||||
<div class="flex space-x-4 mt-4">
|
||||
<button type="button" id="addBatchFile"
|
||||
class="px-4 py-2 bg-green-600 text-white rounded-md hover:bg-green-700 focus:outline-none focus:ring-2 focus:ring-green-500">
|
||||
Add File
|
||||
</button>
|
||||
<button type="button" id="batchImportBtn"
|
||||
class="px-4 py-2 bg-purple-600 text-white rounded-md hover:bg-purple-700 focus:outline-none focus:ring-2 focus:ring-purple-500 disabled:opacity-50">
|
||||
Start Batch Import
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Include the import JavaScript -->
|
||||
<script src="/static/js/admin_import.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user