hotswap/delphi-database-v2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix: Improved CSV encoding detection for legacy data with non-standard characters

Browse Source 
- Changed encoding fallback order to prioritize iso-8859-1/latin-1 over cp1252
- Increased encoding test from 1KB to 10KB to catch issues deeper in files
- Added proper file handle cleanup on encoding failures
- Resolves 'charmap codec can't decode byte 0x9d' error in rolodex import
- Tested with rolodex file containing 52,100 rows successfully
This commit is contained in:
HotSwapp
2025-10-12 19:19:25 -05:00
parent f4c5b9019b
commit 7958556613
16 changed files with 438 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

82
app/templates/admin.html
View File

@@ -53,6 +53,14 @@
TRNSTYPE, TRNSLKUP, FOOTERS, FILESTAT, EMPLOYEE, GRUPLKUP, FILETYPE, and all related tables (*.csv)
</div>
</div>
<div class="form-check mb-3">
<input class="form-check-input" type="checkbox" id="auto_import" name="auto_import" checked>
<label class="form-check-label" for="auto_import">
<strong>Auto-import after upload (follows Import Order Guide)</strong>
<br>
<small class="text-muted">Will stop on the first file that reports any row errors.</small>
</label>
</div>
<button type="submit" class="btn btn-primary">
<i class="bi bi-cloud-upload me-2"></i>Upload Files
</button>
@@ -117,6 +125,80 @@
</div>
{% endif %}
<!-- Auto Import Results -->
{% if auto_import_results %}
<div class="card mb-4">
<div class="card-header bg-info text-white">
<h5 class="mb-0">
<i class="bi bi-lightning-charge me-2"></i>Auto Import Results
</h5>
</div>
<div class="card-body">
{% if auto_import_results.stopped %}
<div class="alert alert-warning">
<i class="bi bi-exclamation-triangle me-2"></i>
Stopped after {{ auto_import_results.files|length }} file(s) due to errors in <code>{{ auto_import_results.stopped_on }}</code>.
</div>
{% endif %}
<div class="table-responsive">
<table class="table table-sm table-bordered">
<thead>
<tr>
<th>Filename</th>
<th>Type</th>
<th>Status</th>
<th>Total</th>
<th>Success</th>
<th>Errors</th>
<th>Error Details</th>
</tr>
</thead>
<tbody>
{% for item in auto_import_results.files %}
<tr>
<td>{{ item.filename }}</td>
<td><span class="badge bg-secondary">{{ item.import_type }}</span></td>
<td>
{% if item.status == 'success' %}
<span class="badge bg-success">Completed</span>
{% else %}
<span class="badge bg-danger">Failed</span>
{% endif %}
</td>
<td>{{ item.total_rows }}</td>
<td class="text-success">{{ item.success_count }}</td>
<td class="text-danger">{{ item.error_count }}</td>
<td>
{% if item.errors %}
<details>
<summary class="text-danger">View Errors ({{ item.errors|length }})</summary>
<ul class="mt-2 mb-0">
{% for err in item.errors %}
<li><small>{{ err }}</small></li>
{% endfor %}
</ul>
</details>
{% else %}
<span class="text-muted">None</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% if auto_import_results.skipped_unknowns and auto_import_results.skipped_unknowns|length > 0 %}
<div class="alert alert-info mt-3">
<i class="bi bi-info-circle me-2"></i>
{{ auto_import_results.skipped_unknowns|length }} unknown file(s) were skipped. Map them in the Data Import section.
</div>
{% endif %}
</div>
</div>
{% endif %}
<!-- Upload Errors -->
{% if upload_errors %}
<div class="card mb-4">