delphi-database-v2/docs/PHONE_IMPORT_FIX.md

Phone Import Unique Constraint Fix

Issue

When uploading PHONE.csv, the import would fail with a SQLite integrity error:

UNIQUE constraint failed: phone.id, phone.phone

This error occurred at row 507 and cascaded to subsequent rows due to transaction rollback.

Root Cause

The LegacyPhone model has a composite primary key on (id, phone) to prevent duplicate phone number entries for the same person/entity. The original import_phone() function used bulk inserts without checking for existing records, causing the constraint violation when:

1. Re-importing the same CSV file
2. The CSV contains duplicate (id, phone) combinations
3. Partial imports left some data in the database

Solution

Updated import_phone() in /app/import_legacy.py to implement an upsert strategy:

Changes Made

1. Check for duplicates within CSV: Track seen (id, phone) combinations to skip duplicates in the same import
2. Check database for existing records: Query for existing (id, phone) before inserting
3. Update or Insert:
   • If record exists → update the location field
   • If record doesn't exist → insert new record
4. Enhanced error handling: Rollback only the failed row, not the entire batch
5. Better logging: Track inserted, updated, and skipped counts separately

Code Changes

# Before: Bulk insert without checking
db.bulk_save_objects(batch)
db.commit()

# After: Upsert with duplicate handling
existing = db.query(LegacyPhone).filter(
    LegacyPhone.id == rolodex_id,
    LegacyPhone.phone == phone
).first()

if existing:
    existing.location = clean_string(row.get('Location'))
    result['updated'] += 1
else:
    record = LegacyPhone(...)
    db.add(record)
    result['inserted'] += 1

Result Tracking

The function now returns detailed statistics:

• success: Total successfully processed rows
• inserted: New records added
• updated: Existing records updated
• skipped: Duplicate combinations within the CSV
• skipped_no_phone: Rows without a phone number (cannot import - phone is part of primary key)
• skipped_no_id: Rows without an ID (cannot import - required field)
• errors: List of error messages for failed rows
• total_rows: Total rows in CSV

Understanding Skipped Rows

Important: The phone field is part of the composite primary key (id, phone). This means:

• You cannot import a phone record without a phone number
• Empty phone numbers will be skipped (this is expected and correct behavior)
• The web UI will display: ⚠️ Skipped: X rows without phone number

Example: If your CSV has 26,437 rows and 143 have empty phone numbers:

• Total rows: 26,437
• Success: 26,294
• Skipped (no phone): 143
• This is working correctly - those 143 rows don't have phone numbers to import

Testing

After deploying this fix:

1. Uploading PHONE.csv for the first time will insert all records
2. Re-uploading the same file will update existing records (no errors)
3. Uploading a CSV with internal duplicates will skip duplicates gracefully

Consistency with Other Imports

This fix aligns import_phone() with the upsert pattern already used in:

• import_rolodex() - handles duplicates by ID
• import_trnstype() - upserts by T_Type
• import_trnslkup() - upserts by T_Code
• import_footers() - upserts by F_Code
• And other reference table imports

Related Files

• /app/import_legacy.py - Contains the fixed import_phone() function
• /app/models.py - Defines LegacyPhone model with composite PK
• /app/main.py - Routes CSV uploads to import functions

Prevention

To prevent similar issues in future imports:

1. Always use upsert logic for tables with unique constraints
2. Test re-imports of the same CSV file
3. Handle duplicates within the CSV gracefully
4. Provide detailed success/error statistics to users