ac98bded69
- Track skipped_no_phone and skipped_no_id separately
- Display skip information in admin UI with warning icon
- Clarify that empty phone numbers cannot be imported (PK constraint)
- Update documentation to explain expected skip behavior
- Example: 143 rows without phone numbers is correct, not an error
When importing PHONE.csv with empty phone numbers:
- Rows are properly skipped (cannot have NULL in primary key)
- User sees: '⚠️ Skipped: 143 rows without phone number'
- This is expected behavior, not a bug
101 lines
3.8 KiB
Markdown
101 lines
3.8 KiB
Markdown
# Phone Import Unique Constraint Fix
|
|
|
|
## Issue
|
|
When uploading `PHONE.csv`, the import would fail with a SQLite integrity error:
|
|
|
|
```
|
|
UNIQUE constraint failed: phone.id, phone.phone
|
|
```
|
|
|
|
This error occurred at row 507 and cascaded to subsequent rows due to transaction rollback.
|
|
|
|
## Root Cause
|
|
The `LegacyPhone` model has a **composite primary key** on `(id, phone)` to prevent duplicate phone number entries for the same person/entity. The original `import_phone()` function used bulk inserts without checking for existing records, causing the constraint violation when:
|
|
|
|
1. Re-importing the same CSV file
|
|
2. The CSV contains duplicate `(id, phone)` combinations
|
|
3. Partial imports left some data in the database
|
|
|
|
## Solution
|
|
Updated `import_phone()` in `/app/import_legacy.py` to implement an **upsert strategy**:
|
|
|
|
### Changes Made
|
|
1. **Check for duplicates within CSV**: Track seen `(id, phone)` combinations to skip duplicates in the same import
|
|
2. **Check database for existing records**: Query for existing `(id, phone)` before inserting
|
|
3. **Update or Insert**:
|
|
- If record exists → update the `location` field
|
|
- If record doesn't exist → insert new record
|
|
4. **Enhanced error handling**: Rollback only the failed row, not the entire batch
|
|
5. **Better logging**: Track `inserted`, `updated`, and `skipped` counts separately
|
|
|
|
### Code Changes
|
|
```python
|
|
# Before: Bulk insert without checking
|
|
db.bulk_save_objects(batch)
|
|
db.commit()
|
|
|
|
# After: Upsert with duplicate handling
|
|
existing = db.query(LegacyPhone).filter(
|
|
LegacyPhone.id == rolodex_id,
|
|
LegacyPhone.phone == phone
|
|
).first()
|
|
|
|
if existing:
|
|
existing.location = clean_string(row.get('Location'))
|
|
result['updated'] += 1
|
|
else:
|
|
record = LegacyPhone(...)
|
|
db.add(record)
|
|
result['inserted'] += 1
|
|
```
|
|
|
|
## Result Tracking
|
|
The function now returns detailed statistics:
|
|
- `success`: Total successfully processed rows
|
|
- `inserted`: New records added
|
|
- `updated`: Existing records updated
|
|
- `skipped`: Duplicate combinations within the CSV
|
|
- `skipped_no_phone`: Rows without a phone number (cannot import - phone is part of primary key)
|
|
- `skipped_no_id`: Rows without an ID (cannot import - required field)
|
|
- `errors`: List of error messages for failed rows
|
|
- `total_rows`: Total rows in CSV
|
|
|
|
### Understanding Skipped Rows
|
|
**Important**: The `phone` field is part of the composite primary key `(id, phone)`. This means:
|
|
- You **cannot** import a phone record without a phone number
|
|
- Empty phone numbers will be skipped (this is expected and correct behavior)
|
|
- The web UI will display: `⚠️ Skipped: X rows without phone number`
|
|
|
|
**Example**: If your CSV has 26,437 rows and 143 have empty phone numbers:
|
|
- Total rows: 26,437
|
|
- Success: 26,294
|
|
- Skipped (no phone): 143
|
|
- **This is working correctly** - those 143 rows don't have phone numbers to import
|
|
|
|
## Testing
|
|
After deploying this fix:
|
|
1. Uploading `PHONE.csv` for the first time will insert all records
|
|
2. Re-uploading the same file will update existing records (no errors)
|
|
3. Uploading a CSV with internal duplicates will skip duplicates gracefully
|
|
|
|
## Consistency with Other Imports
|
|
This fix aligns `import_phone()` with the upsert pattern already used in:
|
|
- `import_rolodex()` - handles duplicates by ID
|
|
- `import_trnstype()` - upserts by T_Type
|
|
- `import_trnslkup()` - upserts by T_Code
|
|
- `import_footers()` - upserts by F_Code
|
|
- And other reference table imports
|
|
|
|
## Related Files
|
|
- `/app/import_legacy.py` - Contains the fixed `import_phone()` function
|
|
- `/app/models.py` - Defines `LegacyPhone` model with composite PK
|
|
- `/app/main.py` - Routes CSV uploads to import functions
|
|
|
|
## Prevention
|
|
To prevent similar issues in future imports:
|
|
1. Always use upsert logic for tables with unique constraints
|
|
2. Test re-imports of the same CSV file
|
|
3. Handle duplicates within the CSV gracefully
|
|
4. Provide detailed success/error statistics to users
|
|
|