clean up docs

2025-08-14 21:40:49 -05:00
parent 679ab4446a
commit 21c6b285d6
6 changed files with 0 additions and 0 deletions
--- a/docs/README.md
+++ b/docs/README.md
@@ -0,0 +1,412 @@
+# Delphi Consulting Group Database System
+
+A modern Python web application built with FastAPI to replace the legacy Pascal-based database system. This system maintains the familiar keyboard shortcuts and workflows while providing a robust, modular backend with a clean web interface.
+
+## 🏢 Company Information
+**Delphi Consulting Group Inc.**  
+Modern database system for legal practice management, financial tracking, and document management.
+
+## 🎯 Project Goals
+- Replace legacy Pascal system with modern, maintainable technology
+- Preserve keyboard shortcuts for user familiarity  
+- Fully modular system - easy to add/remove sections
+- Backend-first approach with solid API endpoints
+- Simple HTML with minimal JavaScript - prioritize reliability
+- Single SQLite file for easy backup/restore
+
+## 🛠️ Technology Stack
+- **Backend**: Python 3.12, FastAPI, SQLAlchemy 2.0+
+- **Database**: SQLite (single file)
+- **Frontend**: Jinja2 templates, Tailwind CSS 3, vanilla JavaScript
+- **Authentication**: JWT with bcrypt password hashing
+- **Validation**: Pydantic v2
+
+## ⚡ Search Performance (FTS + Cache)
+
+- Full-text search is enabled via SQLite FTS5 for Customers (`rolodex`), Files, Ledger, and QDRO.
+  - The app creates virtual FTS tables and sync triggers at startup.
+  - On engines without FTS5, search falls back to standard `ILIKE` queries.
+- Common filter columns are indexed for faster filtering: `files(status, file_type, empl_num)` and `ledger(t_type, empl_num)`.
+- Response caching (optional) uses Redis for global search and suggestions.
+  - Cache TTL: ~90s for global search, ~60s for suggestions.
+  - Cache is auto-invalidated on create/update/delete affecting customers, files, ledger, or QDROs.
+
+Enable cache:
+```bash
+export CACHE_ENABLED=true
+export REDIS_URL=redis://localhost:6379/0
+```
+
+Diagnostics:
+- `GET /api/search/_debug` reports whether FTS tables exist and Redis is available (requires auth).
+
+## 📊 Database Structure
+Based on analysis of legacy Pascal system:
+
+### Core Tables
+1. **ROLODEX** - Customer/client information and contact details
+2. **PHONE** - Phone numbers linked to customers
+3. **FILES** - Legal cases/files with financial tracking
+4. **LEDGER** - Financial transactions per case
+5. **QDROS** - Legal documents (Qualified Domestic Relations Orders)
+6. **USERS** - System authentication and authorization
+
+## ⌨️ Keyboard Shortcuts
+Maintains legacy system shortcuts for user familiarity:
+
+### Navigation
+- `Alt+C` - Customers/Rolodex
+- `Alt+F` - File Cabinet  
+- `Alt+L` - Ledger/Financial
+- `Alt+D` - Documents/QDROs
+- `Alt+A` - Admin Panel
+- `Ctrl+F` - Global Search
+
+### Forms
+- `Ctrl+N` - New Record
+- `Ctrl+S` - Save
+- `F9` - Edit Mode
+- `F2` - Complete/Save  
+- `F8` - Clear/Cancel
+- `Del` - Delete Record
+- `Esc` - Cancel/Close
+
+### Legacy Functions
+- `F1` - Help/Shortcuts
+- `F10` - Menu
+- `Alt+M` - Memo/Notes
+- `Alt+T` - Time Tracker
+- `Alt+B` - Balance Summary
+- `+/-` - Change dates by day
+
+## 🚀 Quick Start
+
+### Option 1: Docker (Recommended)
+```bash
+# Clone repository
+git clone <repository-url>
+cd delphi-database
+
+# Set up secure configuration
+python scripts/setup-security.py
+
+# Development mode
+docker-compose -f docker-compose.dev.yml up
+
+# Production mode
+docker-compose up -d
+```
+
+### Option 2: Local Installation
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Create database and admin user
+python create_admin.py
+
+# Run application
+python -m uvicorn app.main:app --reload
+```
+
+The application will be available at: http://localhost:6920
+
+📖 **For detailed Docker deployment instructions, see [DOCKER.md](DOCKER.md)**
+
+## 📁 Project Structure
+```
+delphi-database/
+├── app/
+│   ├── main.py              # FastAPI application entry point
+│   ├── config.py            # Configuration settings
+│   ├── models/              # SQLAlchemy database models
+│   │   ├── user.py          # User authentication
+│   │   ├── rolodex.py       # Customer/phone models  
+│   │   ├── files.py         # File cabinet model
+│   │   ├── ledger.py        # Financial transactions
+│   │   └── qdro.py          # Legal documents
+│   ├── api/                 # API route handlers
+│   │   ├── auth.py          # Authentication endpoints
+│   │   ├── customers.py     # Customer management
+│   │   ├── files.py         # File management
+│   │   ├── financial.py     # Ledger/financial
+│   │   ├── documents.py     # Document management
+│   │   ├── search.py        # Search functionality
+│   │   └── admin.py         # Admin functions
+│   ├── auth/                # Authentication system
+│   ├── database/            # Database configuration
+│   └── import_export/       # Data import/export utilities
+├── templates/               # Jinja2 HTML templates
+│   ├── base.html           # Base template with nav/shortcuts
+│   └── dashboard.html      # Main dashboard
+├── static/                 # Static files (CSS, JS, images)
+│   ├── css/main.css        # Main stylesheet
+│   ├── js/keyboard-shortcuts.js  # Keyboard shortcut system
+│   └── js/main.js          # Main JavaScript utilities
+├── old database/           # Legacy Pascal files (reference)
+├── uploads/                # File uploads
+├── backups/                # Database backups
+├── requirements.txt        # Python dependencies
+├── create_admin.py         # Admin user creation script
+└── README.md              # This file
+```
+
+## 🔧 API Endpoints
+
+### Common pagination, sorting, and totals
+- Many list endpoints support the same query parameters:
+  - `skip` (int): offset for pagination. Default varies per endpoint.
+  - `limit` (int): page size. Most endpoints cap at 200–1000.
+  - `sort_by` (str): whitelisted field name per endpoint.
+  - `sort_dir` (str): `asc` or `desc`.
+  - `include_total` (bool): when `true`, the response is an object `{ items, total }`; otherwise a plain list is returned for backwards compatibility.
+- Some endpoints also support `search` (tokenized across multiple columns with AND semantics) for simple text filtering.
+
+Examples:
+```bash
+# Support tickets (admin)
+curl \
+  "http://localhost:6920/api/support/tickets?include_total=true&limit=10&sort_by=created&sort_dir=desc"
+
+# My support tickets (current user)
+curl \
+  "http://localhost:6920/api/support/my-tickets?include_total=true&limit=10&sort_by=updated&sort_dir=desc"
+
+# QDROs for a file
+curl \
+  "http://localhost:6920/api/documents/qdros/FILE-123?include_total=true&sort_by=updated&sort_dir=desc"
+
+# Ledger entries for a file
+curl \
+  "http://localhost:6920/api/financial/ledger/FILE-123?include_total=true&sort_by=date&sort_dir=desc"
+
+# Customer phones
+curl \
+  "http://localhost:6920/api/customers/CUST-1/phones?include_total=true&sort_by=location&sort_dir=asc"
+```
+
+Allowed sort fields (high level):
+- Support tickets: `created`, `updated`, `resolved`, `priority`, `status`, `subject`
+- My tickets: `created`, `updated`, `resolved`, `priority`, `status`, `subject`
+- QDROs (list and by file): `file_no`, `version`, `status`, `created`, `updated`
+- Ledger by file: `date`, `item_no`, `amount`, `billed`
+- Templates: `form_id`, `form_name`, `category`, `created`, `updated`
+- Files: `file_no`, `client`, `opened`, `closed`, `status`, `amount_owing`, `total_charges`
+- Admin users: `username`, `email`, `first_name`, `last_name`, `created`, `updated`
+- Customer phones: `location`, `phone`
+
+### Authentication
+- `POST /api/auth/login` - User login
+- `POST /api/auth/register` - Register user (admin only)
+- `GET /api/auth/me` - Current user info
+
+### Customers (Rolodex)
+- `GET /api/customers/` - List customers
+- `POST /api/customers/` - Create customer
+- `GET /api/customers/{id}` - Get customer details
+- `PUT /api/customers/{id}` - Update customer  
+- `DELETE /api/customers/{id}` - Delete customer
+- `GET /api/customers/{id}/phones` - Get phone numbers
+- `POST /api/customers/{id}/phones` - Add phone number
+
+### Files
+- `GET /api/files/` - List files
+- `POST /api/files/` - Create file
+- `GET /api/files/{file_no}` - Get file details
+- `PUT /api/files/{file_no}` - Update file
+- `DELETE /api/files/{file_no}` - Delete file
+
+### Financial (Ledger)
+- `GET /api/financial/ledger/{file_no}` - Get ledger entries (supports pagination, sorting, `include_total`)
+- `POST /api/financial/ledger/` - Create transaction
+- `PUT /api/financial/ledger/{id}` - Update transaction
+- `DELETE /api/financial/ledger/{id}` - Delete transaction
+- `GET /api/financial/reports/{file_no}` - Financial reports
+
+### Documents (QDROs)
+- `GET /api/documents/qdros/{file_no}` - Get QDROs for file (supports pagination, sorting, `include_total`)
+- `POST /api/documents/qdros/` - Create QDRO
+- `GET /api/documents/qdros/{file_no}/{id}` - Get specific QDRO
+- `PUT /api/documents/qdros/{file_no}/{id}` - Update QDRO
+- `DELETE /api/documents/qdros/{file_no}/{id}` - Delete QDRO
+
+### Support
+- `POST /api/support/tickets` - Create support ticket (public; auth optional)
+- `GET /api/support/tickets` - List tickets (admin; supports filters, search, pagination, sorting, `include_total`)
+- `GET /api/support/tickets/{id}` - Get ticket details (admin)
+- `PUT /api/support/tickets/{id}` - Update ticket (admin)
+- `POST /api/support/tickets/{id}/responses` - Add response (admin)
+- `GET /api/support/my-tickets` - List current user's tickets (supports status filter, search, pagination, sorting, `include_total`)
+- `GET /api/support/stats` - Ticket statistics (admin)
+
+### Search
+- `GET /api/search/customers?q={query}` - Search customers
+- `GET /api/search/files?q={query}` - Search files
+- `GET /api/search/global?q={query}` - Global search
+
+### Admin
+- `GET /api/admin/health` - System health check
+- `GET /api/admin/stats` - System statistics  
+- `POST /api/admin/import/csv` - Import CSV data
+- `GET /api/admin/export/{table}` - Export table data
+- `GET /api/admin/backup/download` - Download database backup
+
+## 🔒 Authentication
+- Session-based JWT authentication
+- Role-based access (User/Admin)
+- Password hashing with bcrypt
+- Token expiration and refresh
+
+JWT details:
+
+- Access token: returned by `POST /api/auth/login`, use in `Authorization: Bearer` header
+- Refresh token: also returned on login; use `POST /api/auth/refresh` with body `{ "refresh_token": "..." }` to obtain a new access token. On refresh, the provided refresh token is revoked and a new one is issued.
+- Legacy compatibility: `POST /api/auth/refresh` called without a body (but with Authorization header) will issue a new access token only.
+
+## 🗄️ Data Management
+- CSV import/export functionality
+- Database backup and restore
+- Data validation and error handling
+- Automatic financial calculations (matching legacy system)
+
+## ⚙️ Configuration
+Environment variables (create `.env` file). Real environment variables override `.env` which override defaults:
+```bash
+# Database
+DATABASE_URL=sqlite:///./delphi_database.db
+
+# Security
+SECRET_KEY=your-secret-key-change-in-production
+# Optional previous key to allow rotation
+PREVIOUS_SECRET_KEY=
+ACCESS_TOKEN_EXPIRE_MINUTES=240
+REFRESH_TOKEN_EXPIRE_MINUTES=43200
+
+# Application
+DEBUG=False
+APP_NAME=Delphi Consulting Group Database System
+```
+
+## 📋 Development Tasks
+
+### Phase 1 - Foundation ✅
+- [x] Project structure setup
+- [x] SQLAlchemy models based on legacy system
+- [x] Authentication system
+- [x] Core FastAPI application
+
+### Phase 2 - API Development ✅  
+- [x] Customer management endpoints
+- [x] File management endpoints
+- [x] Financial/ledger endpoints
+- [x] Document management endpoints
+- [x] Search functionality
+- [x] Admin endpoints
+
+### Phase 3 - Frontend (In Progress)
+- [x] Base HTML template with keyboard shortcuts
+- [x] Dashboard interface
+- [ ] Customer management UI
+- [ ] File management UI  
+- [ ] Financial interface
+- [ ] Document management UI
+- [ ] Search interface
+- [ ] Admin interface
+
+### Phase 4 - Data Migration
+- [ ] Legacy .SC file parser
+- [ ] Data cleaning and validation
+- [ ] Migration scripts
+- [ ] Data verification tools
+
+### Phase 5 - Advanced Features
+- [ ] Financial calculations (matching legacy Tally_Ledger)
+- [ ] Report generation
+- [ ] Document templates
+- [ ] Time tracking system
+- [ ] Backup automation
+
+## 🧪 Testing
+```bash
+# Run tests (when implemented)
+pytest
+
+# Run with coverage
+pytest --cov=app tests/
+```
+
+## 🚢 Deployment
+
+### Docker Deployment (Recommended)
+```bash
+# Build and start services
+docker-compose up -d
+
+# With Nginx reverse proxy
+docker-compose --profile production up -d
+
+# Check status
+docker-compose ps
+```
+
+## HTTP client usage in the frontend
+
+- Prefer calling `window.http.wrappedFetch(url, options)` instead of `fetch` directly.
+- The wrapper automatically adds:
+  - `Authorization: Bearer <token>` when available
+  - `X-Correlation-ID` on every request
+  - `Content-Type: application/json` when you pass a string body (e.g., from `JSON.stringify`)
+- It also exposes helpers: `window.http.parseErrorEnvelope`, `window.http.toError`, `window.http.formatAlert`.
+- The global `fetch` remains wrapped for compatibility, but will log a one-time deprecation warning. New code should use `window.http.wrappedFetch`.
+
+### Traditional Deployment
+```bash
+# With gunicorn for production
+gunicorn app.main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000
+```
+
+📖 **Complete deployment guide:** [DOCKER.md](DOCKER.md)
+
+## 🛡️ Security & Git Best Practices
+
+### 🚨 NEVER Commit These Files:
+- **`.env`** - Contains secrets, passwords, API keys
+- **Database files** - `*.db`, `*.sqlite`, `delphi_database.db`
+- **Backup files** - `backups/`, `*.backup`, `*.dump`
+- **Upload files** - `uploads/`, user documents
+- **SSL certificates** - `*.pem`, `*.key`, `*.crt`
+- **Local configs** - `*-local.*`, `config.local.py`
+
+### ✅ Repository Security:
+- Use `python scripts/setup-security.py` for secure configuration
+- Install Git hooks: `./scripts/install-git-hooks.sh` 
+- Review `.gitignore` before committing
+- Never commit real customer data
+- Rotate secrets if accidentally committed
+- Use environment variables for all sensitive data
+
+### 🔒 Git Hooks Protection:
+The pre-commit hook automatically blocks commits containing:
+- Environment files (`.env`)
+- Database files (`*.db`, `*.sqlite`)
+- Backup files (`backups/`, `*.backup`)
+- SSL certificates and keys (`*.pem`, `*.key`)
+- Upload directories with user files
+- Large files that may contain sensitive data
+
+## 🤝 Contributing
+1. Follow the modular architecture principles
+2. Maintain keyboard shortcut compatibility
+3. Preserve legacy system workflows
+4. Ensure comprehensive error handling
+5. Document all API changes
+6. **Review security checklist before commits**
+
+## 📄 License
+Proprietary software for Delphi Consulting Group Inc.
+
+## 📞 Support
+For technical support or questions about this system, contact the development team.
+
+---
+**Built with ❤️ for Delphi Consulting Group Inc.**