# Delphi Consulting Group Database System

A modern Python web application built with FastAPI to replace the legacy Pascal-based database system. This system maintains the familiar keyboard shortcuts and workflows while providing a robust, modular backend with a clean web interface.

## 🏢 Company Information
**Delphi Consulting Group Inc.**  
Modern database system for legal practice management, financial tracking, and document management.

## 🎯 Project Goals
- Replace legacy Pascal system with modern, maintainable technology
- Preserve keyboard shortcuts for user familiarity  
- Fully modular system - easy to add/remove sections
- Backend-first approach with solid API endpoints
- Simple HTML with minimal JavaScript - prioritize reliability
- Single SQLite file for easy backup/restore

## 🛠️ Technology Stack
- **Backend**: Python 3.12, FastAPI, SQLAlchemy 2.0+
- **Database**: SQLite (single file)
- **Frontend**: Jinja2 templates, Tailwind CSS 3, vanilla JavaScript
- **Authentication**: JWT with bcrypt password hashing
- **Validation**: Pydantic v2

## ⚡ Search Performance (FTS + Cache)

- Full-text search is enabled via SQLite FTS5 for Customers (`rolodex`), Files, Ledger, and QDRO.
  - The app creates virtual FTS tables and sync triggers at startup.
  - On engines without FTS5, search falls back to standard `ILIKE` queries.
- Common filter columns are indexed for faster filtering: `files(status, file_type, empl_num)` and `ledger(t_type, empl_num)`.
- Response caching (optional) uses Redis for global search and suggestions.
  - Cache TTL: ~90s for global search, ~60s for suggestions.
  - Cache is auto-invalidated on create/update/delete affecting customers, files, ledger, or QDROs.

Enable cache:
```bash
export CACHE_ENABLED=true
export REDIS_URL=redis://localhost:6379/0
```

Diagnostics:
- `GET /api/search/_debug` reports whether FTS tables exist and Redis is available (requires auth).

## 📊 Database Structure
Based on analysis of legacy Pascal system:

### Core Tables
1. **ROLODEX** - Customer/client information and contact details
2. **PHONE** - Phone numbers linked to customers
3. **FILES** - Legal cases/files with financial tracking
4. **LEDGER** - Financial transactions per case
5. **QDROS** - Legal documents (Qualified Domestic Relations Orders)
6. **USERS** - System authentication and authorization

## ⌨️ Keyboard Shortcuts
Maintains legacy system shortcuts for user familiarity:

### Navigation
- `Alt+C` - Customers/Rolodex
- `Alt+F` - File Cabinet  
- `Alt+L` - Ledger/Financial
- `Alt+D` - Documents/QDROs
- `Alt+A` - Admin Panel
- `Ctrl+F` - Global Search

### Forms
- `Ctrl+N` - New Record
- `Ctrl+S` - Save
- `F9` - Edit Mode
- `F2` - Complete/Save  
- `F8` - Clear/Cancel
- `Del` - Delete Record
- `Esc` - Cancel/Close

### Legacy Functions
- `F1` - Help/Shortcuts
- `F10` - Menu
- `Alt+M` - Memo/Notes
- `Alt+T` - Time Tracker
- `Alt+B` - Balance Summary
- `+/-` - Change dates by day

## 🚀 Quick Start

### Option 1: Docker (Recommended)
```bash
# Clone repository
git clone <repository-url>
cd delphi-database

# Set up secure configuration
python scripts/setup-security.py

# Development mode
docker-compose -f docker-compose.dev.yml up

# Production mode
docker-compose up -d
```

### Option 2: Local Installation
```bash
# Install dependencies
pip install -r requirements.txt

# Create database and admin user
python create_admin.py

# Run application
python -m uvicorn app.main:app --reload
```

The application will be available at: http://localhost:6920

📖 **For detailed Docker deployment instructions, see [DOCKER.md](DOCKER.md)**

## 📁 Project Structure
```
delphi-database/
├── app/
│   ├── main.py              # FastAPI application entry point
│   ├── config.py            # Configuration settings
│   ├── models/              # SQLAlchemy database models
│   │   ├── user.py          # User authentication
│   │   ├── rolodex.py       # Customer/phone models  
│   │   ├── files.py         # File cabinet model
│   │   ├── ledger.py        # Financial transactions
│   │   └── qdro.py          # Legal documents
│   ├── api/                 # API route handlers
│   │   ├── auth.py          # Authentication endpoints
│   │   ├── customers.py     # Customer management
│   │   ├── files.py         # File management
│   │   ├── financial.py     # Ledger/financial
│   │   ├── documents.py     # Document management
│   │   ├── search.py        # Search functionality
│   │   └── admin.py         # Admin functions
│   ├── auth/                # Authentication system
│   ├── database/            # Database configuration
│   └── import_export/       # Data import/export utilities
├── templates/               # Jinja2 HTML templates
│   ├── base.html           # Base template with nav/shortcuts
│   └── dashboard.html      # Main dashboard
├── static/                 # Static files (CSS, JS, images)
│   ├── css/main.css        # Main stylesheet
│   ├── js/keyboard-shortcuts.js  # Keyboard shortcut system
│   └── js/main.js          # Main JavaScript utilities
├── old database/           # Legacy Pascal files (reference)
├── uploads/                # File uploads
├── backups/                # Database backups
├── requirements.txt        # Python dependencies
├── create_admin.py         # Admin user creation script
└── README.md              # This file
```

## 🔧 API Endpoints

### Common pagination, sorting, and totals
- Many list endpoints support the same query parameters:
  - `skip` (int): offset for pagination. Default varies per endpoint.
  - `limit` (int): page size. Most endpoints cap at 200–1000.
  - `sort_by` (str): whitelisted field name per endpoint.
  - `sort_dir` (str): `asc` or `desc`.
  - `include_total` (bool): when `true`, the response is an object `{ items, total }`; otherwise a plain list is returned for backwards compatibility.
- Some endpoints also support `search` (tokenized across multiple columns with AND semantics) for simple text filtering.

Examples:
```bash
# Support tickets (admin)
curl \
  "http://localhost:6920/api/support/tickets?include_total=true&limit=10&sort_by=created&sort_dir=desc"

# My support tickets (current user)
curl \
  "http://localhost:6920/api/support/my-tickets?include_total=true&limit=10&sort_by=updated&sort_dir=desc"

# QDROs for a file
curl \
  "http://localhost:6920/api/documents/qdros/FILE-123?include_total=true&sort_by=updated&sort_dir=desc"

# Ledger entries for a file
curl \
  "http://localhost:6920/api/financial/ledger/FILE-123?include_total=true&sort_by=date&sort_dir=desc"

# Customer phones
curl \
  "http://localhost:6920/api/customers/CUST-1/phones?include_total=true&sort_by=location&sort_dir=asc"
```

Allowed sort fields (high level):
- Support tickets: `created`, `updated`, `resolved`, `priority`, `status`, `subject`
- My tickets: `created`, `updated`, `resolved`, `priority`, `status`, `subject`
- QDROs (list and by file): `file_no`, `version`, `status`, `created`, `updated`
- Ledger by file: `date`, `item_no`, `amount`, `billed`
- Templates: `form_id`, `form_name`, `category`, `created`, `updated`
- Files: `file_no`, `client`, `opened`, `closed`, `status`, `amount_owing`, `total_charges`
- Admin users: `username`, `email`, `first_name`, `last_name`, `created`, `updated`
- Customer phones: `location`, `phone`

### Authentication
- `POST /api/auth/login` - User login
- `POST /api/auth/register` - Register user (admin only)
- `GET /api/auth/me` - Current user info

### Customers (Rolodex)
- `GET /api/customers/` - List customers
- `POST /api/customers/` - Create customer
- `GET /api/customers/{id}` - Get customer details
- `PUT /api/customers/{id}` - Update customer  
- `DELETE /api/customers/{id}` - Delete customer
- `GET /api/customers/{id}/phones` - Get phone numbers
- `POST /api/customers/{id}/phones` - Add phone number

### Files
- `GET /api/files/` - List files
- `POST /api/files/` - Create file
- `GET /api/files/{file_no}` - Get file details
- `PUT /api/files/{file_no}` - Update file
- `DELETE /api/files/{file_no}` - Delete file

### Financial (Ledger)
- `GET /api/financial/ledger/{file_no}` - Get ledger entries (supports pagination, sorting, `include_total`)
- `POST /api/financial/ledger/` - Create transaction
- `PUT /api/financial/ledger/{id}` - Update transaction
- `DELETE /api/financial/ledger/{id}` - Delete transaction
- `GET /api/financial/reports/{file_no}` - Financial reports

### Documents (QDROs)
- `GET /api/documents/qdros/{file_no}` - Get QDROs for file (supports pagination, sorting, `include_total`)
- `POST /api/documents/qdros/` - Create QDRO
- `GET /api/documents/qdros/{file_no}/{id}` - Get specific QDRO
- `PUT /api/documents/qdros/{file_no}/{id}` - Update QDRO
- `DELETE /api/documents/qdros/{file_no}/{id}` - Delete QDRO

### Support
- `POST /api/support/tickets` - Create support ticket (public; auth optional)
- `GET /api/support/tickets` - List tickets (admin; supports filters, search, pagination, sorting, `include_total`)
- `GET /api/support/tickets/{id}` - Get ticket details (admin)
- `PUT /api/support/tickets/{id}` - Update ticket (admin)
- `POST /api/support/tickets/{id}/responses` - Add response (admin)
- `GET /api/support/my-tickets` - List current user's tickets (supports status filter, search, pagination, sorting, `include_total`)
- `GET /api/support/stats` - Ticket statistics (admin)

### Search
- `GET /api/search/customers?q={query}` - Search customers
- `GET /api/search/files?q={query}` - Search files
- `GET /api/search/global?q={query}` - Global search

### Admin
- `GET /api/admin/health` - System health check
- `GET /api/admin/stats` - System statistics  
- `POST /api/admin/import/csv` - Import CSV data
- `GET /api/admin/export/{table}` - Export table data
- `GET /api/admin/backup/download` - Download database backup

## 🔒 Authentication
- Session-based JWT authentication
- Role-based access (User/Admin)
- Password hashing with bcrypt
- Token expiration and refresh

JWT details:

- Access token: returned by `POST /api/auth/login`, use in `Authorization: Bearer` header
- Refresh token: also returned on login; use `POST /api/auth/refresh` with body `{ "refresh_token": "..." }` to obtain a new access token. On refresh, the provided refresh token is revoked and a new one is issued.
- Legacy compatibility: `POST /api/auth/refresh` called without a body (but with Authorization header) will issue a new access token only.

## 🗄️ Data Management
- CSV import/export functionality
- Database backup and restore
- Data validation and error handling
- Automatic financial calculations (matching legacy system)

## ⚙️ Configuration
Environment variables (create `.env` file). Real environment variables override `.env` which override defaults:
```bash
# Database
DATABASE_URL=sqlite:///./delphi_database.db

# Security
SECRET_KEY=your-secret-key-change-in-production
# Optional previous key to allow rotation
PREVIOUS_SECRET_KEY=
ACCESS_TOKEN_EXPIRE_MINUTES=240
REFRESH_TOKEN_EXPIRE_MINUTES=43200

# Application
DEBUG=False
APP_NAME=Delphi Consulting Group Database System
```

## 📋 Development Tasks

### Phase 1 - Foundation ✅
- [x] Project structure setup
- [x] SQLAlchemy models based on legacy system
- [x] Authentication system
- [x] Core FastAPI application

### Phase 2 - API Development ✅  
- [x] Customer management endpoints
- [x] File management endpoints
- [x] Financial/ledger endpoints
- [x] Document management endpoints
- [x] Search functionality
- [x] Admin endpoints

### Phase 3 - Frontend (In Progress)
- [x] Base HTML template with keyboard shortcuts
- [x] Dashboard interface
- [ ] Customer management UI
- [ ] File management UI  
- [ ] Financial interface
- [ ] Document management UI
- [ ] Search interface
- [ ] Admin interface

### Phase 4 - Data Migration
- [ ] Legacy .SC file parser
- [ ] Data cleaning and validation
- [ ] Migration scripts
- [ ] Data verification tools

### Phase 5 - Advanced Features
- [ ] Financial calculations (matching legacy Tally_Ledger)
- [ ] Report generation
- [ ] Document templates
- [ ] Time tracking system
- [ ] Backup automation

## 🧪 Testing
```bash
# Run tests (when implemented)
pytest

# Run with coverage
pytest --cov=app tests/
```

## 🚢 Deployment

### Docker Deployment (Recommended)
```bash
# Build and start services
docker-compose up -d

# With Nginx reverse proxy
docker-compose --profile production up -d

# Check status
docker-compose ps
```

## HTTP client usage in the frontend

- Prefer calling `window.http.wrappedFetch(url, options)` instead of `fetch` directly.
- The wrapper automatically adds:
  - `Authorization: Bearer <token>` when available
  - `X-Correlation-ID` on every request
  - `Content-Type: application/json` when you pass a string body (e.g., from `JSON.stringify`)
- It also exposes helpers: `window.http.parseErrorEnvelope`, `window.http.toError`, `window.http.formatAlert`.
- The global `fetch` remains wrapped for compatibility, but will log a one-time deprecation warning. New code should use `window.http.wrappedFetch`.

### Traditional Deployment
```bash
# With gunicorn for production
gunicorn app.main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000
```

📖 **Complete deployment guide:** [DOCKER.md](DOCKER.md)

## 🛡️ Security & Git Best Practices

### 🚨 NEVER Commit These Files:
- **`.env`** - Contains secrets, passwords, API keys
- **Database files** - `*.db`, `*.sqlite`, `delphi_database.db`
- **Backup files** - `backups/`, `*.backup`, `*.dump`
- **Upload files** - `uploads/`, user documents
- **SSL certificates** - `*.pem`, `*.key`, `*.crt`
- **Local configs** - `*-local.*`, `config.local.py`

### ✅ Repository Security:
- Use `python scripts/setup-security.py` for secure configuration
- Install Git hooks: `./scripts/install-git-hooks.sh` 
- Review `.gitignore` before committing
- Never commit real customer data
- Rotate secrets if accidentally committed
- Use environment variables for all sensitive data

### 🔒 Git Hooks Protection:
The pre-commit hook automatically blocks commits containing:
- Environment files (`.env`)
- Database files (`*.db`, `*.sqlite`)
- Backup files (`backups/`, `*.backup`)
- SSL certificates and keys (`*.pem`, `*.key`)
- Upload directories with user files
- Large files that may contain sensitive data

## 🤝 Contributing
1. Follow the modular architecture principles
2. Maintain keyboard shortcut compatibility
3. Preserve legacy system workflows
4. Ensure comprehensive error handling
5. Document all API changes
6. **Review security checklist before commits**

## 📄 License
Proprietary software for Delphi Consulting Group Inc.

## 📞 Support
For technical support or questions about this system, contact the development team.

---
**Built with ❤️ for Delphi Consulting Group Inc.**