delphi-database-v2/docs/TROUBLESHOOTING_IMPORTS.md

Troubleshooting Import Issues

Encoding Errors

Problem: "charmap codec can't decode byte"

Symptoms:

Fatal error: 'charmap' codec can't decode byte 0x9d in position 7244: character maps to <undefined>

Solution: This has been fixed in the latest version. If you still see this error:

1. Rebuild Docker container (changes don't apply until rebuild):

   docker-compose down
   docker-compose build
   docker-compose up -d
   

2. Verify the fix is active:

   # Check encoding order includes iso-8859-1 early
   docker-compose exec delphi-db grep "encodings = " app/import_legacy.py
   
   # Should show: ["utf-8", "utf-8-sig", "iso-8859-1", "latin-1", ...]
   

3. Check container is using new image:

   docker-compose ps
   # Note the CREATED time - should be recent
   

Problem: File not found during import

Symptoms:

• "File not found" error
• Import appears to succeed but no data imported

Solution:

1. Make sure files are in the data-import/ directory
2. Files must be accessible inside the Docker container
3. Check docker-compose.yml mounts the correct volume:

   volumes:
     - ./data-import:/app/data-import
   

Import Workflow Issues

Problem: Import hangs or takes too long

Symptoms:

• Import never completes
• Browser shows loading indefinitely

Solution:

1. Check Docker logs:

   docker-compose logs -f delphi-db
   

2. Large files (50k+ rows) may take several minutes

3. Check for duplicate key violations in logs

Problem: "Unknown import type"

Symptoms:

• File uploads but shows as "unknown"
• Cannot auto-import

Solution:

1. File must match expected naming pattern (e.g., rolodex_*.csv)
2. Use manual mapping in admin interface:
   • Go to Admin → Import section
   • Find the unknown file
   • Click "Map to Import Type"
   • Select correct import type
   • Save and import

Database Issues

Problem: Duplicate key violations

Symptoms:

UNIQUE constraint failed: rolodex.id
IntegrityError: duplicate key value

Solution:

1. Data already exists - safe to ignore if re-importing
2. To clear and re-import:
   • Delete existing database: rm delphi.db
   • Restart container: docker-compose restart
   • Import files in correct order (reference data first)

Problem: Foreign key violations

Symptoms:

FOREIGN KEY constraint failed

Solution: Import files in dependency order:

1. Reference tables first (employee, filetype, etc.)
2. Core tables next (rolodex, files)
3. Related tables last (phone, ledger, payments)

See IMPORT_ORDER in admin interface for correct sequence.

Debugging Steps

1. Check Application Logs

docker-compose logs -f delphi-db

2. Access Container Shell

docker-compose exec delphi-db bash

3. Test Import Manually

# Inside container
python3 << EOF
from app.import_legacy import open_text_with_fallbacks
import csv

f, encoding = open_text_with_fallbacks('data-import/rolodex_*.csv')
print(f"Encoding: {encoding}")
reader = csv.DictReader(f)
print(f"Columns: {reader.fieldnames}")
EOF

4. Check File Permissions

ls -la data-import/
# Files should be readable

5. Verify Python Environment

docker-compose exec delphi-db python3 --version
docker-compose exec delphi-db pip list | grep -i sql

Common Mistakes

1. Not rebuilding after code changes - Docker containers use cached images
2. Wrong file format - Must be CSV with headers
3. File encoding not supported - Should be handled by fallback, but exotic encodings may fail
4. Importing in wrong order - Dependencies must be imported first
5. Missing required columns - Check CSV has expected columns

Getting Help

If issues persist:

1. Check logs: docker-compose logs delphi-db
2. Note exact error message and stack trace
3. Check which import function is failing
4. Verify file format matches expected schema
5. Try importing a small test file (10 rows) to isolate issue