412 lines
15 KiB
Markdown
412 lines
15 KiB
Markdown
# Delphi Consulting Group Database System
|
||
|
||
A modern Python web application built with FastAPI to replace the legacy Pascal-based database system. This system maintains the familiar keyboard shortcuts and workflows while providing a robust, modular backend with a clean web interface.
|
||
|
||
## 🏢 Company Information
|
||
**Delphi Consulting Group Inc.**
|
||
Modern database system for legal practice management, financial tracking, and document management.
|
||
|
||
## 🎯 Project Goals
|
||
- Replace legacy Pascal system with modern, maintainable technology
|
||
- Preserve keyboard shortcuts for user familiarity
|
||
- Fully modular system - easy to add/remove sections
|
||
- Backend-first approach with solid API endpoints
|
||
- Simple HTML with minimal JavaScript - prioritize reliability
|
||
- Single SQLite file for easy backup/restore
|
||
|
||
## 🛠️ Technology Stack
|
||
- **Backend**: Python 3.12, FastAPI, SQLAlchemy 2.0+
|
||
- **Database**: SQLite (single file)
|
||
- **Frontend**: Jinja2 templates, Tailwind CSS 3, vanilla JavaScript
|
||
- **Authentication**: JWT with bcrypt password hashing
|
||
- **Validation**: Pydantic v2
|
||
|
||
## ⚡ Search Performance (FTS + Cache)
|
||
|
||
- Full-text search is enabled via SQLite FTS5 for Customers (`rolodex`), Files, Ledger, and QDRO.
|
||
- The app creates virtual FTS tables and sync triggers at startup.
|
||
- On engines without FTS5, search falls back to standard `ILIKE` queries.
|
||
- Common filter columns are indexed for faster filtering: `files(status, file_type, empl_num)` and `ledger(t_type, empl_num)`.
|
||
- Response caching (optional) uses Redis for global search and suggestions.
|
||
- Cache TTL: ~90s for global search, ~60s for suggestions.
|
||
- Cache is auto-invalidated on create/update/delete affecting customers, files, ledger, or QDROs.
|
||
|
||
Enable cache:
|
||
```bash
|
||
export CACHE_ENABLED=true
|
||
export REDIS_URL=redis://localhost:6379/0
|
||
```
|
||
|
||
Diagnostics:
|
||
- `GET /api/search/_debug` reports whether FTS tables exist and Redis is available (requires auth).
|
||
|
||
## 📊 Database Structure
|
||
Based on analysis of legacy Pascal system:
|
||
|
||
### Core Tables
|
||
1. **ROLODEX** - Customer/client information and contact details
|
||
2. **PHONE** - Phone numbers linked to customers
|
||
3. **FILES** - Legal cases/files with financial tracking
|
||
4. **LEDGER** - Financial transactions per case
|
||
5. **QDROS** - Legal documents (Qualified Domestic Relations Orders)
|
||
6. **USERS** - System authentication and authorization
|
||
|
||
## ⌨️ Keyboard Shortcuts
|
||
Maintains legacy system shortcuts for user familiarity:
|
||
|
||
### Navigation
|
||
- `Alt+C` - Customers/Rolodex
|
||
- `Alt+F` - File Cabinet
|
||
- `Alt+L` - Ledger/Financial
|
||
- `Alt+D` - Documents/QDROs
|
||
- `Alt+A` - Admin Panel
|
||
- `Ctrl+F` - Global Search
|
||
|
||
### Forms
|
||
- `Ctrl+N` - New Record
|
||
- `Ctrl+S` - Save
|
||
- `F9` - Edit Mode
|
||
- `F2` - Complete/Save
|
||
- `F8` - Clear/Cancel
|
||
- `Del` - Delete Record
|
||
- `Esc` - Cancel/Close
|
||
|
||
### Legacy Functions
|
||
- `F1` - Help/Shortcuts
|
||
- `F10` - Menu
|
||
- `Alt+M` - Memo/Notes
|
||
- `Alt+T` - Time Tracker
|
||
- `Alt+B` - Balance Summary
|
||
- `+/-` - Change dates by day
|
||
|
||
## 🚀 Quick Start
|
||
|
||
### Option 1: Docker (Recommended)
|
||
```bash
|
||
# Clone repository
|
||
git clone <repository-url>
|
||
cd delphi-database
|
||
|
||
# Set up secure configuration
|
||
python scripts/setup-security.py
|
||
|
||
# Development mode
|
||
docker-compose -f docker-compose.dev.yml up
|
||
|
||
# Production mode
|
||
docker-compose up -d
|
||
```
|
||
|
||
### Option 2: Local Installation
|
||
```bash
|
||
# Install dependencies
|
||
pip install -r requirements.txt
|
||
|
||
# Create database and admin user
|
||
python create_admin.py
|
||
|
||
# Run application
|
||
python -m uvicorn app.main:app --reload
|
||
```
|
||
|
||
The application will be available at: http://localhost:6920
|
||
|
||
📖 **For detailed Docker deployment instructions, see [DOCKER.md](DOCKER.md)**
|
||
|
||
## 📁 Project Structure
|
||
```
|
||
delphi-database/
|
||
├── app/
|
||
│ ├── main.py # FastAPI application entry point
|
||
│ ├── config.py # Configuration settings
|
||
│ ├── models/ # SQLAlchemy database models
|
||
│ │ ├── user.py # User authentication
|
||
│ │ ├── rolodex.py # Customer/phone models
|
||
│ │ ├── files.py # File cabinet model
|
||
│ │ ├── ledger.py # Financial transactions
|
||
│ │ └── qdro.py # Legal documents
|
||
│ ├── api/ # API route handlers
|
||
│ │ ├── auth.py # Authentication endpoints
|
||
│ │ ├── customers.py # Customer management
|
||
│ │ ├── files.py # File management
|
||
│ │ ├── financial.py # Ledger/financial
|
||
│ │ ├── documents.py # Document management
|
||
│ │ ├── search.py # Search functionality
|
||
│ │ └── admin.py # Admin functions
|
||
│ ├── auth/ # Authentication system
|
||
│ ├── database/ # Database configuration
|
||
│ └── import_export/ # Data import/export utilities
|
||
├── templates/ # Jinja2 HTML templates
|
||
│ ├── base.html # Base template with nav/shortcuts
|
||
│ └── dashboard.html # Main dashboard
|
||
├── static/ # Static files (CSS, JS, images)
|
||
│ ├── css/main.css # Main stylesheet
|
||
│ ├── js/keyboard-shortcuts.js # Keyboard shortcut system
|
||
│ └── js/main.js # Main JavaScript utilities
|
||
├── old database/ # Legacy Pascal files (reference)
|
||
├── uploads/ # File uploads
|
||
├── backups/ # Database backups
|
||
├── requirements.txt # Python dependencies
|
||
├── create_admin.py # Admin user creation script
|
||
└── README.md # This file
|
||
```
|
||
|
||
## 🔧 API Endpoints
|
||
|
||
### Common pagination, sorting, and totals
|
||
- Many list endpoints support the same query parameters:
|
||
- `skip` (int): offset for pagination. Default varies per endpoint.
|
||
- `limit` (int): page size. Most endpoints cap at 200–1000.
|
||
- `sort_by` (str): whitelisted field name per endpoint.
|
||
- `sort_dir` (str): `asc` or `desc`.
|
||
- `include_total` (bool): when `true`, the response is an object `{ items, total }`; otherwise a plain list is returned for backwards compatibility.
|
||
- Some endpoints also support `search` (tokenized across multiple columns with AND semantics) for simple text filtering.
|
||
|
||
Examples:
|
||
```bash
|
||
# Support tickets (admin)
|
||
curl \
|
||
"http://localhost:6920/api/support/tickets?include_total=true&limit=10&sort_by=created&sort_dir=desc"
|
||
|
||
# My support tickets (current user)
|
||
curl \
|
||
"http://localhost:6920/api/support/my-tickets?include_total=true&limit=10&sort_by=updated&sort_dir=desc"
|
||
|
||
# QDROs for a file
|
||
curl \
|
||
"http://localhost:6920/api/documents/qdros/FILE-123?include_total=true&sort_by=updated&sort_dir=desc"
|
||
|
||
# Ledger entries for a file
|
||
curl \
|
||
"http://localhost:6920/api/financial/ledger/FILE-123?include_total=true&sort_by=date&sort_dir=desc"
|
||
|
||
# Customer phones
|
||
curl \
|
||
"http://localhost:6920/api/customers/CUST-1/phones?include_total=true&sort_by=location&sort_dir=asc"
|
||
```
|
||
|
||
Allowed sort fields (high level):
|
||
- Support tickets: `created`, `updated`, `resolved`, `priority`, `status`, `subject`
|
||
- My tickets: `created`, `updated`, `resolved`, `priority`, `status`, `subject`
|
||
- QDROs (list and by file): `file_no`, `version`, `status`, `created`, `updated`
|
||
- Ledger by file: `date`, `item_no`, `amount`, `billed`
|
||
- Templates: `form_id`, `form_name`, `category`, `created`, `updated`
|
||
- Files: `file_no`, `client`, `opened`, `closed`, `status`, `amount_owing`, `total_charges`
|
||
- Admin users: `username`, `email`, `first_name`, `last_name`, `created`, `updated`
|
||
- Customer phones: `location`, `phone`
|
||
|
||
### Authentication
|
||
- `POST /api/auth/login` - User login
|
||
- `POST /api/auth/register` - Register user (admin only)
|
||
- `GET /api/auth/me` - Current user info
|
||
|
||
### Customers (Rolodex)
|
||
- `GET /api/customers/` - List customers
|
||
- `POST /api/customers/` - Create customer
|
||
- `GET /api/customers/{id}` - Get customer details
|
||
- `PUT /api/customers/{id}` - Update customer
|
||
- `DELETE /api/customers/{id}` - Delete customer
|
||
- `GET /api/customers/{id}/phones` - Get phone numbers
|
||
- `POST /api/customers/{id}/phones` - Add phone number
|
||
|
||
### Files
|
||
- `GET /api/files/` - List files
|
||
- `POST /api/files/` - Create file
|
||
- `GET /api/files/{file_no}` - Get file details
|
||
- `PUT /api/files/{file_no}` - Update file
|
||
- `DELETE /api/files/{file_no}` - Delete file
|
||
|
||
### Financial (Ledger)
|
||
- `GET /api/financial/ledger/{file_no}` - Get ledger entries (supports pagination, sorting, `include_total`)
|
||
- `POST /api/financial/ledger/` - Create transaction
|
||
- `PUT /api/financial/ledger/{id}` - Update transaction
|
||
- `DELETE /api/financial/ledger/{id}` - Delete transaction
|
||
- `GET /api/financial/reports/{file_no}` - Financial reports
|
||
|
||
### Documents (QDROs)
|
||
- `GET /api/documents/qdros/{file_no}` - Get QDROs for file (supports pagination, sorting, `include_total`)
|
||
- `POST /api/documents/qdros/` - Create QDRO
|
||
- `GET /api/documents/qdros/{file_no}/{id}` - Get specific QDRO
|
||
- `PUT /api/documents/qdros/{file_no}/{id}` - Update QDRO
|
||
- `DELETE /api/documents/qdros/{file_no}/{id}` - Delete QDRO
|
||
|
||
### Support
|
||
- `POST /api/support/tickets` - Create support ticket (public; auth optional)
|
||
- `GET /api/support/tickets` - List tickets (admin; supports filters, search, pagination, sorting, `include_total`)
|
||
- `GET /api/support/tickets/{id}` - Get ticket details (admin)
|
||
- `PUT /api/support/tickets/{id}` - Update ticket (admin)
|
||
- `POST /api/support/tickets/{id}/responses` - Add response (admin)
|
||
- `GET /api/support/my-tickets` - List current user's tickets (supports status filter, search, pagination, sorting, `include_total`)
|
||
- `GET /api/support/stats` - Ticket statistics (admin)
|
||
|
||
### Search
|
||
- `GET /api/search/customers?q={query}` - Search customers
|
||
- `GET /api/search/files?q={query}` - Search files
|
||
- `GET /api/search/global?q={query}` - Global search
|
||
|
||
### Admin
|
||
- `GET /api/admin/health` - System health check
|
||
- `GET /api/admin/stats` - System statistics
|
||
- `POST /api/admin/import/csv` - Import CSV data
|
||
- `GET /api/admin/export/{table}` - Export table data
|
||
- `GET /api/admin/backup/download` - Download database backup
|
||
|
||
## 🔒 Authentication
|
||
- Session-based JWT authentication
|
||
- Role-based access (User/Admin)
|
||
- Password hashing with bcrypt
|
||
- Token expiration and refresh
|
||
|
||
JWT details:
|
||
|
||
- Access token: returned by `POST /api/auth/login`, use in `Authorization: Bearer` header
|
||
- Refresh token: also returned on login; use `POST /api/auth/refresh` with body `{ "refresh_token": "..." }` to obtain a new access token. On refresh, the provided refresh token is revoked and a new one is issued.
|
||
- Legacy compatibility: `POST /api/auth/refresh` called without a body (but with Authorization header) will issue a new access token only.
|
||
|
||
## 🗄️ Data Management
|
||
- CSV import/export functionality
|
||
- Database backup and restore
|
||
- Data validation and error handling
|
||
- Automatic financial calculations (matching legacy system)
|
||
|
||
## ⚙️ Configuration
|
||
Environment variables (create `.env` file). Real environment variables override `.env` which override defaults:
|
||
```bash
|
||
# Database
|
||
DATABASE_URL=sqlite:///./delphi_database.db
|
||
|
||
# Security
|
||
SECRET_KEY=your-secret-key-change-in-production
|
||
# Optional previous key to allow rotation
|
||
PREVIOUS_SECRET_KEY=
|
||
ACCESS_TOKEN_EXPIRE_MINUTES=240
|
||
REFRESH_TOKEN_EXPIRE_MINUTES=43200
|
||
|
||
# Application
|
||
DEBUG=False
|
||
APP_NAME=Delphi Consulting Group Database System
|
||
```
|
||
|
||
## 📋 Development Tasks
|
||
|
||
### Phase 1 - Foundation ✅
|
||
- [x] Project structure setup
|
||
- [x] SQLAlchemy models based on legacy system
|
||
- [x] Authentication system
|
||
- [x] Core FastAPI application
|
||
|
||
### Phase 2 - API Development ✅
|
||
- [x] Customer management endpoints
|
||
- [x] File management endpoints
|
||
- [x] Financial/ledger endpoints
|
||
- [x] Document management endpoints
|
||
- [x] Search functionality
|
||
- [x] Admin endpoints
|
||
|
||
### Phase 3 - Frontend (In Progress)
|
||
- [x] Base HTML template with keyboard shortcuts
|
||
- [x] Dashboard interface
|
||
- [ ] Customer management UI
|
||
- [ ] File management UI
|
||
- [ ] Financial interface
|
||
- [ ] Document management UI
|
||
- [ ] Search interface
|
||
- [ ] Admin interface
|
||
|
||
### Phase 4 - Data Migration
|
||
- [ ] Legacy .SC file parser
|
||
- [ ] Data cleaning and validation
|
||
- [ ] Migration scripts
|
||
- [ ] Data verification tools
|
||
|
||
### Phase 5 - Advanced Features
|
||
- [ ] Financial calculations (matching legacy Tally_Ledger)
|
||
- [ ] Report generation
|
||
- [ ] Document templates
|
||
- [ ] Time tracking system
|
||
- [ ] Backup automation
|
||
|
||
## 🧪 Testing
|
||
```bash
|
||
# Run tests (when implemented)
|
||
pytest
|
||
|
||
# Run with coverage
|
||
pytest --cov=app tests/
|
||
```
|
||
|
||
## 🚢 Deployment
|
||
|
||
### Docker Deployment (Recommended)
|
||
```bash
|
||
# Build and start services
|
||
docker-compose up -d
|
||
|
||
# With Nginx reverse proxy
|
||
docker-compose --profile production up -d
|
||
|
||
# Check status
|
||
docker-compose ps
|
||
```
|
||
|
||
## HTTP client usage in the frontend
|
||
|
||
- Prefer calling `window.http.wrappedFetch(url, options)` instead of `fetch` directly.
|
||
- The wrapper automatically adds:
|
||
- `Authorization: Bearer <token>` when available
|
||
- `X-Correlation-ID` on every request
|
||
- `Content-Type: application/json` when you pass a string body (e.g., from `JSON.stringify`)
|
||
- It also exposes helpers: `window.http.parseErrorEnvelope`, `window.http.toError`, `window.http.formatAlert`.
|
||
- The global `fetch` remains wrapped for compatibility, but will log a one-time deprecation warning. New code should use `window.http.wrappedFetch`.
|
||
|
||
### Traditional Deployment
|
||
```bash
|
||
# With gunicorn for production
|
||
gunicorn app.main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000
|
||
```
|
||
|
||
📖 **Complete deployment guide:** [DOCKER.md](DOCKER.md)
|
||
|
||
## 🛡️ Security & Git Best Practices
|
||
|
||
### 🚨 NEVER Commit These Files:
|
||
- **`.env`** - Contains secrets, passwords, API keys
|
||
- **Database files** - `*.db`, `*.sqlite`, `delphi_database.db`
|
||
- **Backup files** - `backups/`, `*.backup`, `*.dump`
|
||
- **Upload files** - `uploads/`, user documents
|
||
- **SSL certificates** - `*.pem`, `*.key`, `*.crt`
|
||
- **Local configs** - `*-local.*`, `config.local.py`
|
||
|
||
### ✅ Repository Security:
|
||
- Use `python scripts/setup-security.py` for secure configuration
|
||
- Install Git hooks: `./scripts/install-git-hooks.sh`
|
||
- Review `.gitignore` before committing
|
||
- Never commit real customer data
|
||
- Rotate secrets if accidentally committed
|
||
- Use environment variables for all sensitive data
|
||
|
||
### 🔒 Git Hooks Protection:
|
||
The pre-commit hook automatically blocks commits containing:
|
||
- Environment files (`.env`)
|
||
- Database files (`*.db`, `*.sqlite`)
|
||
- Backup files (`backups/`, `*.backup`)
|
||
- SSL certificates and keys (`*.pem`, `*.key`)
|
||
- Upload directories with user files
|
||
- Large files that may contain sensitive data
|
||
|
||
## 🤝 Contributing
|
||
1. Follow the modular architecture principles
|
||
2. Maintain keyboard shortcut compatibility
|
||
3. Preserve legacy system workflows
|
||
4. Ensure comprehensive error handling
|
||
5. Document all API changes
|
||
6. **Review security checklist before commits**
|
||
|
||
## 📄 License
|
||
Proprietary software for Delphi Consulting Group Inc.
|
||
|
||
## 📞 Support
|
||
For technical support or questions about this system, contact the development team.
|
||
|
||
---
|
||
**Built with ❤️ for Delphi Consulting Group Inc.** |