hotswap/delphi-database
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e3a279dba729f5a097247859e994c81123dedc0d
delphi-database/docs/README.md

20 KiB
Raw Blame History

Delphi Consulting Group Database System

A modern Python web application built with FastAPI to replace the legacy Pascal-based database system. This system maintains the familiar keyboard shortcuts and workflows while providing a robust, modular backend with a clean web interface.

🏢 Company Information

Delphi Consulting Group Inc.
Modern database system for legal practice management, financial tracking, and document management.

🎯 Project Goals

  • Replace legacy Pascal system with modern, maintainable technology
  • Preserve keyboard shortcuts for user familiarity
  • Fully modular system - easy to add/remove sections
  • Backend-first approach with solid API endpoints
  • Simple HTML with minimal JavaScript - prioritize reliability
  • Single SQLite file for easy backup/restore

🛠️ Technology Stack

  • Backend: Python 3.12, FastAPI, SQLAlchemy 2.0+
  • Database: SQLite (single file)
  • Frontend: Jinja2 templates, Tailwind CSS 3, vanilla JavaScript
  • Authentication: JWT with bcrypt password hashing
  • Validation: Pydantic v2

⚡ Search Performance (FTS + Cache)

  • Full-text search is enabled via SQLite FTS5 for Customers (rolodex), Files, Ledger, and QDRO.
    • The app creates virtual FTS tables and sync triggers at startup.
    • On engines without FTS5, search falls back to standard ILIKE queries.
  • Common filter columns are indexed for faster filtering: files(status, file_type, empl_num) and ledger(t_type, empl_num).
  • Response caching (optional) uses Redis for global search and suggestions.
    • Cache TTL: ~90s for global search, ~60s for suggestions.
    • Cache is auto-invalidated on create/update/delete affecting customers, files, ledger, or QDROs.

Enable cache:

export CACHE_ENABLED=true
export REDIS_URL=redis://localhost:6379/0

Diagnostics:

  • GET /api/search/_debug reports whether FTS tables exist and Redis is available (requires auth).

📊 Database Structure

Based on analysis of legacy Pascal system:

Core Tables

  1. ROLODEX - Customer/client information and contact details
  2. PHONE - Phone numbers linked to customers
  3. FILES - Legal cases/files with financial tracking
  4. LEDGER - Financial transactions per case
  5. QDROS - Legal documents (Qualified Domestic Relations Orders)
  6. USERS - System authentication and authorization

⌨️ Keyboard Shortcuts

Maintains legacy system shortcuts for user familiarity:

Navigation

  • Alt+C - Customers/Rolodex
  • Alt+F - File Cabinet
  • Alt+L - Ledger/Financial
  • Alt+D - Documents/QDROs
  • Alt+A - Admin Panel
  • Ctrl+F - Global Search

Forms

  • Ctrl+N - New Record
  • Ctrl+S - Save
  • F9 - Edit Mode
  • F2 - Complete/Save
  • F8 - Clear/Cancel
  • Del - Delete Record
  • Esc - Cancel/Close

Legacy Functions

  • F1 - Help/Shortcuts
  • F10 - Menu
  • Alt+M - Memo/Notes
  • Alt+T - Time Tracker
  • Alt+B - Balance Summary
  • +/- - Change dates by day

🚀 Quick Start

Option 1: Docker (Recommended)

# Clone repository
git clone <repository-url>
cd delphi-database

# Set up secure configuration
python scripts/setup-security.py

# Development mode
docker-compose -f docker-compose.dev.yml up

# Production mode
docker-compose up -d

Option 2: Local Installation

# Install dependencies
pip install -r requirements.txt

# Create database and admin user
python create_admin.py

# Run application
python -m uvicorn app.main:app --reload

The application will be available at: http://localhost:6920

📖 For detailed Docker deployment instructions, see DOCKER.md

📁 Project Structure

delphi-database/
├── app/
│   ├── main.py              # FastAPI application entry point
│   ├── config.py            # Configuration settings
│   ├── models/              # SQLAlchemy database models
│   │   ├── user.py          # User authentication
│   │   ├── rolodex.py       # Customer/phone models  
│   │   ├── files.py         # File cabinet model
│   │   ├── ledger.py        # Financial transactions
│   │   └── qdro.py          # Legal documents
│   ├── api/                 # API route handlers
│   │   ├── auth.py          # Authentication endpoints
│   │   ├── customers.py     # Customer management
│   │   ├── files.py         # File management
│   │   ├── financial.py     # Ledger/financial
│   │   ├── documents.py     # Document management
│   │   ├── search.py        # Search functionality
│   │   └── admin.py         # Admin functions
│   ├── auth/                # Authentication system
│   ├── database/            # Database configuration
│   └── import_export/       # Data import/export utilities
├── templates/               # Jinja2 HTML templates
│   ├── base.html           # Base template with nav/shortcuts
│   └── dashboard.html      # Main dashboard
├── static/                 # Static files (CSS, JS, images)
│   ├── css/main.css        # Main stylesheet
│   ├── js/keyboard-shortcuts.js  # Keyboard shortcut system
│   └── js/main.js          # Main JavaScript utilities
├── old database/           # Legacy Pascal files (reference)
├── uploads/                # File uploads
├── backups/                # Database backups
├── requirements.txt        # Python dependencies
├── create_admin.py         # Admin user creation script
└── README.md              # This file

🔧 API Endpoints

Common pagination, sorting, and totals

  • Many list endpoints support the same query parameters:
    • skip (int): offset for pagination. Default varies per endpoint.
    • limit (int): page size. Most endpoints cap at 200–1000.
    • sort_by (str): whitelisted field name per endpoint.
    • sort_dir (str): asc or desc.
    • include_total (bool): when true, the response is an object { items, total }; otherwise a plain list is returned for backwards compatibility.
  • Some endpoints also support search (tokenized across multiple columns with AND semantics) for simple text filtering.

Examples:

# Support tickets (admin)
curl \
  "http://localhost:6920/api/support/tickets?include_total=true&limit=10&sort_by=created&sort_dir=desc"

# My support tickets (current user)
curl \
  "http://localhost:6920/api/support/my-tickets?include_total=true&limit=10&sort_by=updated&sort_dir=desc"

# QDROs for a file
curl \
  "http://localhost:6920/api/documents/qdros/FILE-123?include_total=true&sort_by=updated&sort_dir=desc"

# Ledger entries for a file
curl \
  "http://localhost:6920/api/financial/ledger/FILE-123?include_total=true&sort_by=date&sort_dir=desc"

# Customer phones
curl \
  "http://localhost:6920/api/customers/CUST-1/phones?include_total=true&sort_by=location&sort_dir=asc"

Allowed sort fields (high level):

  • Support tickets: created, updated, resolved, priority, status, subject
  • My tickets: created, updated, resolved, priority, status, subject
  • QDROs (list and by file): file_no, version, status, created, updated
  • Ledger by file: date, item_no, amount, billed
  • Templates: form_id, form_name, category, created, updated
  • Files: file_no, client, opened, closed, status, amount_owing, total_charges
  • Admin users: username, email, first_name, last_name, created, updated
  • Customer phones: location, phone

Authentication

  • POST /api/auth/login - User login
  • POST /api/auth/register - Register user (admin only)
  • GET /api/auth/me - Current user info

Customers (Rolodex)

  • GET /api/customers/ - List customers
  • POST /api/customers/ - Create customer
  • GET /api/customers/{id} - Get customer details
  • PUT /api/customers/{id} - Update customer
  • DELETE /api/customers/{id} - Delete customer
  • GET /api/customers/{id}/phones - Get phone numbers
  • POST /api/customers/{id}/phones - Add phone number

Files

  • GET /api/files/ - List files
  • POST /api/files/ - Create file
  • GET /api/files/{file_no} - Get file details
  • PUT /api/files/{file_no} - Update file
  • DELETE /api/files/{file_no} - Delete file

Financial (Ledger)

  • GET /api/financial/ledger/{file_no} - Get ledger entries (supports pagination, sorting, include_total)
  • POST /api/financial/ledger/ - Create transaction
  • PUT /api/financial/ledger/{id} - Update transaction
  • DELETE /api/financial/ledger/{id} - Delete transaction
  • GET /api/financial/reports/{file_no} - Financial reports

Documents (QDROs)

  • GET /api/documents/qdros/{file_no} - Get QDROs for file (supports pagination, sorting, include_total)
  • POST /api/documents/qdros/ - Create QDRO
  • GET /api/documents/qdros/{file_no}/{id} - Get specific QDRO
  • PUT /api/documents/qdros/{file_no}/{id} - Update QDRO
  • DELETE /api/documents/qdros/{file_no}/{id} - Delete QDRO

📚 See also: PENSIONS.md for detailed pensions API fields, sorting, and examples.

Pensions

  • GET /api/pensions/schedules - List pension schedules for a file
    • Query params: file_no (required), skip, limit, sort_by (id,file_no,version,vests_on,vests_at), sort_dir, include_total, filters: start, end, version, numeric ranges: vests_at_min, vests_at_max, search: search (tokenized across version, frequency).
    • Examples: 
      curl "http://localhost:6920/api/pensions/schedules?file_no=F-1&sort_by=vests_on&sort_dir=asc&limit=20&include_total=true"
curl "http://localhost:6920/api/pensions/schedules?file_no=F-1&version=02&vests_at_min=10&vests_at_max=50"
  • GET /api/pensions/marriages - List marriage history for a file
    • Query params: file_no (required), skip, limit, sort_by (id,file_no,version,married_from,married_to,marital_percent,service_from,service_to), sort_dir, include_total, filters: start, end, version, numeric ranges: married_years_min/_max, service_years_min/_max, marital_percent_min/_max, search: search (tokenized across version, spouse_name, notes).
    • Example: 
      curl "http://localhost:6920/api/pensions/marriages?file_no=F-1&search=Jane%20Doe&sort_by=married_from&sort_dir=desc"
  • GET /api/pensions/death-benefits - List death benefits for a file
    • Query params: file_no (required), skip, limit, sort_by (id,file_no,version,lump1,lump2,growth1,growth2,disc1,disc2,created), sort_dir, include_total, filters: start, end, version, numeric ranges: lump1_min/_max, lump2_min/_max, growth1_min/_max, growth2_min/_max, disc1_min/_max, disc2_min/_max, search: search (tokenized across version, beneficiary_name, benefit_type, notes).
    • Example: 
      curl "http://localhost:6920/api/pensions/death-benefits?file_no=F-1&lump1_min=100&sort_by=lump1&sort_dir=desc"
  • GET /api/pensions/separations - List separation agreements for a file
    • Query params: file_no (required), skip, limit, sort_by (id,file_no,version,agreement_date), sort_dir, include_total, filters: start, end, version, search: search (tokenized across version, terms, notes).
    • Example: 
      curl "http://localhost:6920/api/pensions/separations?file_no=F-1&start=2024-01-01&end=2024-12-31&sort_by=agreement_date"
  • GET /api/pensions/{file_no}/detail - Detail view for a file's pension data with nested, independently paginated lists
    • Each nested list accepts its own paging/sorting/filtering query prefixes: s_* (schedules), m_* (marriages), d_* (death benefits), sep_* (separations)
    • Example: 
      curl "http://localhost:6920/api/pensions/F-1/detail?s_sort_by=vests_on&s_limit=10&m_sort_by=married_from&d_sort_by=lump1&sep_sort_by=agreement_date"
  • POST /api/pensions/ - Create a main Pension record
    • Body (JSON, selected fields): 
      {
  "file_no": "F-1",
  "version": "01",
  "plan_id": "PID1",
  "plan_name": "Plan A",
  "vested_per": 50,
  "tax_rate": 25
}
    • Notes: numeric validation enforced (e.g., vested_per 0–100; tax_rate 0–100; monetary values non-negative)
  • GET /api/pensions/{pension_id} - Get a Pension by id
  • PUT /api/pensions/{pension_id} - Update a Pension (partial fields accepted)
    • Example: 
      curl -X PUT "http://localhost:6920/api/pensions/123" -H 'Content-Type: application/json' -d '{"plan_name":"Plan B","vested_per":75}'
  • DELETE /api/pensions/{pension_id} - Delete a Pension

Templates

  • GET /api/templates/search - Search document templates

    • Query params:
      • q (str, optional): partial match on template name or description
      • category (str[] or CSV, optional): filter by one or more categories. Repeat the parameter (?category=A&category=B) or pass a comma-separated list (?category=A,B).
      • keywords (str[], optional, repeatable): keyword tags assigned to templates
      • keywords_mode (str, optional): any (default) returns templates that match any of the provided keywords; all returns only templates that contain all the provided keywords
      • has_keywords (bool, optional): when true, only templates that have one or more keywords are returned; when false, only templates with no keywords are returned
      • skip (int, optional): pagination offset, default 0
      • limit (int, optional): page size, default 50, max 200
      • sort_by (str, optional): name (default) | category | updated
      • sort_dir (str, optional): asc (default) | desc
      • active_only (bool, optional): when true (default), only active templates are returned
    • Examples: 
      # Any of the keywords (default)
curl "http://localhost:6920/api/templates/search?keywords=qdro&keywords=divorce"

# Must contain all keywords
curl "http://localhost:6920/api/templates/search?keywords=qdro&keywords=divorce&keywords_mode=all"

# Sorted by name descending with pagination
curl "http://localhost:6920/api/templates/search?sort_by=name&sort_dir=desc&skip=10&limit=10"

# Include inactive templates
curl "http://localhost:6920/api/templates/search?active_only=false"

  • GET /api/templates/categories - List distinct template categories with counts

    • Query params:
      • active_only (bool, optional): when true (default), only counts active templates
    • Example: 
      curl "http://localhost:6920/api/templates/categories?active_only=false"

Support

  • POST /api/support/tickets - Create support ticket (public; auth optional)
  • GET /api/support/tickets - List tickets (admin; supports filters, search, pagination, sorting, include_total)
  • GET /api/support/tickets/{id} - Get ticket details (admin)
  • PUT /api/support/tickets/{id} - Update ticket (admin)
  • POST /api/support/tickets/{id}/responses - Add response (admin)
  • GET /api/support/my-tickets - List current user's tickets (supports status filter, search, pagination, sorting, include_total)
  • GET /api/support/stats - Ticket statistics (admin)

Search

  • GET /api/search/customers?q={query} - Search customers
  • GET /api/search/files?q={query} - Search files
  • GET /api/search/global?q={query} - Global search

Admin

  • GET /api/admin/health - System health check
  • GET /api/admin/stats - System statistics
  • POST /api/admin/import/csv - Import CSV data
  • GET /api/admin/export/{table} - Export table data
  • GET /api/admin/backup/download - Download database backup

🔒 Authentication

  • Session-based JWT authentication
  • Role-based access (User/Admin)
  • Password hashing with bcrypt
  • Token expiration and refresh

JWT details:

  • Access token: returned by POST /api/auth/login, use in Authorization: Bearer header
  • Refresh token: also returned on login; use POST /api/auth/refresh with body { "refresh_token": "..." } to obtain a new access token. On refresh, the provided refresh token is revoked and a new one is issued.
  • Legacy compatibility: POST /api/auth/refresh called without a body (but with Authorization header) will issue a new access token only.

🗄️ Data Management

  • CSV import/export functionality
  • Database backup and restore
  • Data validation and error handling
  • Automatic financial calculations (matching legacy system)

⚙️ Configuration

Environment variables (create .env file). Real environment variables override .env which override defaults:

# Database
DATABASE_URL=sqlite:///./delphi_database.db

# Security
SECRET_KEY=your-secret-key-change-in-production
# Optional previous key to allow rotation
PREVIOUS_SECRET_KEY=
ACCESS_TOKEN_EXPIRE_MINUTES=240
REFRESH_TOKEN_EXPIRE_MINUTES=43200

# Application
DEBUG=False
APP_NAME=Delphi Consulting Group Database System

📋 Development Tasks

Phase 1 - Foundation ✅

  • Project structure setup
  • SQLAlchemy models based on legacy system
  • Authentication system
  • Core FastAPI application

Phase 2 - API Development ✅

  • Customer management endpoints
  • File management endpoints
  • Financial/ledger endpoints
  • Document management endpoints
  • Search functionality
  • Admin endpoints

Phase 3 - Frontend (In Progress)

  • Base HTML template with keyboard shortcuts
  • Dashboard interface
  • Customer management UI
  • File management UI
  • Financial interface
  • Document management UI
  • Search interface
  • Admin interface

Phase 4 - Data Migration

  • Legacy .SC file parser
  • Data cleaning and validation
  • Migration scripts
  • Data verification tools

Phase 5 - Advanced Features

  • Financial calculations (matching legacy Tally_Ledger)
  • Report generation
  • Document templates
  • Time tracking system
  • Backup automation

🧪 Testing

# Run tests (when implemented)
pytest

# Run with coverage
pytest --cov=app tests/

🚢 Deployment

Docker Deployment (Recommended)

# Build and start services
docker-compose up -d

# With Nginx reverse proxy
docker-compose --profile production up -d

# Check status
docker-compose ps

HTTP client usage in the frontend

  • Prefer calling window.http.wrappedFetch(url, options) instead of fetch directly.
  • The wrapper automatically adds:
    • Authorization: Bearer <token> when available
    • X-Correlation-ID on every request
    • Content-Type: application/json when you pass a string body (e.g., from JSON.stringify)
  • It also exposes helpers: window.http.parseErrorEnvelope, window.http.toError, window.http.formatAlert.
  • The global fetch remains wrapped for compatibility, but will log a one-time deprecation warning. New code should use window.http.wrappedFetch.

Traditional Deployment

# With gunicorn for production
gunicorn app.main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000

📖 Complete deployment guide: DOCKER.md

🛡️ Security & Git Best Practices

🚨 NEVER Commit These Files:

  • .env - Contains secrets, passwords, API keys
  • Database files - *.db, *.sqlite, delphi_database.db
  • Backup files - backups/, *.backup, *.dump
  • Upload files - uploads/, user documents
  • SSL certificates - *.pem, *.key, *.crt
  • Local configs - *-local.*, config.local.py

✅ Repository Security:

  • Use python scripts/setup-security.py for secure configuration
  • Install Git hooks: ./scripts/install-git-hooks.sh
  • Review .gitignore before committing
  • Never commit real customer data
  • Rotate secrets if accidentally committed
  • Use environment variables for all sensitive data

🔒 Git Hooks Protection:

The pre-commit hook automatically blocks commits containing:

  • Environment files (.env)
  • Database files (*.db, *.sqlite)
  • Backup files (backups/, *.backup)
  • SSL certificates and keys (*.pem, *.key)
  • Upload directories with user files
  • Large files that may contain sensitive data

🤝 Contributing

  1. Follow the modular architecture principles
  2. Maintain keyboard shortcut compatibility
  3. Preserve legacy system workflows
  4. Ensure comprehensive error handling
  5. Document all API changes
  6. Review security checklist before commits

📄 License

Proprietary software for Delphi Consulting Group Inc.

📞 Support

For technical support or questions about this system, contact the development team.

Built with ❤️ for Delphi Consulting Group Inc.

Reference in New Issue View Git Blame Copy Permalink