hotswap/delphi-database
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
45 Commits 1 Branch 0 Tags
main
Commit Graph

29 Commits

Author SHA1 Message Date
HotSwapp
c68e36e6c6 Fix SQL reserved keyword issue in generic CSV importer 
- Add SQL_RESERVED_KEYWORDS list containing SQLite reserved keywords
- Add _quote_column_name() method to properly quote reserved keywords in SQL queries
- Update INSERT statement generation to quote column names that are reserved keywords
- This fixes the 'group' column import issue where SQL syntax error occurred
 2025-10-01 08:07:30 -05:00
HotSwapp
1116658d40 work on import 2025-09-22 22:07:50 -04:00
HotSwapp
7e9bfcec5e feat: Rebuild complete CSV import system for legacy data migration 
PROBLEM SOLVED:
- Completely removed broken import functionality
- Built new robust, modular CSV import system from scratch
- Provides reliable data migration path for legacy .sc files

NEW IMPORT SYSTEM FEATURES:
✅ Modular CSV parsers for all 5 tables (ROLODEX, PHONE, FILES, LEDGER, QDROS)
✅ RESTful API endpoints with background processing (/api/admin/import/*)
✅ Admin web interface at /admin/import for file uploads
✅ Comprehensive validation and error handling
✅ Real-time progress tracking and status monitoring
✅ Detailed logging with import session tracking
✅ Transaction rollback on failures
✅ Batch import with dependency ordering
✅ Foreign key validation and duplicate detection

TECHNICAL IMPLEMENTATION:
- Clean /app/import_export/ module structure with base classes
- Enhanced logging system with import-specific logs
- Background task processing with FastAPI BackgroundTasks
- Auto-detection of CSV delimiters and encoding
- Field validation with proper data type conversion
- Admin authentication integration
- Console logging for debugging support

IMPORT WORKFLOW:
1. Admin selects table type and uploads CSV file
2. System validates headers and data structure
3. Background processing with real-time status updates
4. Detailed error reporting and success metrics
5. Import logs stored in logs/imports/ directory

SUPPORTED TABLES:
- ROLODEX (contacts/people) - 19 fields, requires: id, last
- PHONE (phone numbers) - 3 fields, requires: rolodex_id, phone
- FILES (case files) - 29 fields, requires: file_no, id, empl_num, file_type, opened, status, rate_per_hour
- LEDGER (transactions) - 12 fields, requires: file_no, date, t_code, t_type, empl_num, amount
- QDROS (documents) - 31 fields, requires: file_no

REMOVED FILES:
- app/api/unified_import_api.py
- app/services/unified_import.py
- app/api/flexible.py
- app/models/flexible.py
- templates/unified_import.html
- templates/flexible.html
- static/js/flexible.js
- All legacy import routes and references

TESTING COMPLETED:
✅ Schema validation for all table types
✅ CSV header validation
✅ Single file import functionality
✅ Multi-table dependency validation
✅ Error handling and logging
✅ API endpoint integration

READY FOR PRODUCTION: System tested and validated with sample data.
Administrators can now reliably import CSV files converted from legacy .sc files.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-09-21 20:54:46 -05:00
HotSwapp
f7644a4f67 working on new system for importing 2025-09-21 20:37:13 -05:00
HotSwapp
536ebd2acc validate: treat legacy wide-format SETUP.csv as valid in single-file validate endpoint\n\nNow consistent with batch validation and import paths. 2025-09-04 15:33:06 -05:00
HotSwapp
926f1f2c1e import: support legacy wide-format SETUP.csv (Appl_Title, L_Head1..10, Default_Printer)\n- Add wide-format detection and dedicated import path mapping headers to SystemSetup keys\n- Accept SETUP.csv as valid in validation when wide-format headers are present\n- Batch import path handles wide-format with audit entries\n\nThis resolves 0 mapped / 12 unmapped issue for legacy SETUP.csv 2025-09-04 15:26:23 -05:00
HotSwapp
41ffbc1430 feat(billing): restrict WS subscriptions to initiating user or admins 2025-09-04 15:08:09 -05:00
HotSwapp
30e4c83618 feat(import): restrict WS subscriptions to initiating user or admins 2025-09-04 14:57:52 -05:00
HotSwapp
032baf6e3e feat(import): use WebSocket push for progress updates with polling fallback 2025-09-04 14:50:14 -05:00
HotSwapp
48ca876123 feat(import): add real-time progress tracking for CSV imports 2025-09-04 14:34:14 -05:00
HotSwapp
bac8cc4bd5 changes 2025-08-18 20:20:04 -05:00
HotSwapp
ae4484381f progress 2025-08-16 10:05:42 -05:00
HotSwapp
0347284556 working on backend 2025-08-15 22:04:43 -05:00
HotSwapp
abc7f289d1 finishing QDRO section 2025-08-15 17:19:51 -05:00
HotSwapp
006ef3d7b1 templates: support include_total for search and categories endpoints; update docs; add tests 2025-08-15 15:06:45 -05:00
HotSwapp
e3a279dba7 templates: add multi-category filter (repeatable or CSV) to GET /api/templates/search; add has_keywords filter; add categories listing endpoint with counts; update docs; add tests 2025-08-15 15:04:40 -05:00
HotSwapp
679ab4446a remove old import 2025-08-14 21:27:34 -05:00
HotSwapp
bfc04a6909 fixes and refactor 2025-08-14 19:16:28 -05:00
HotSwapp
5111079149 coming together 2025-08-13 18:53:35 -05:00
HotSwapp
c76b68d009 fixing rolodex and search 2025-08-11 21:58:25 -05:00
HotSwapp
278eb7c5d4 fixed search 2025-08-11 11:26:16 -05:00
HotSwapp
85ce0f82ec ready to test the import 2025-08-11 10:48:12 -05:00
HotSwapp
1512b2d12a all working 2025-08-10 21:34:11 -05:00
HotSwapp
14ee479edc Auth: add proactive JWT refresh in frontend and clock-skew leeway in backend; improve 401 handling 2025-08-10 19:49:07 -05:00
HotSwapp
350af60db3 working now 2025-08-10 19:06:21 -05:00
HotSwapp
c2f3c4411d progress 2025-08-09 16:37:57 -05:00
HotSwapp
5f74243c8c next front end 2025-08-08 20:20:21 -05:00
HotSwapp
04edc636f8 v2 2025-08-08 19:06:39 -05:00
HotSwapp
b257a06787 maybe good 2025-08-08 15:55:15 -05:00