Add terminal session management with heartbeat monitoring, idle timeout detection, session reuse logic, and command history panel UI with search and filtering capabilities

documentation/TERMINAL_COMMAND_HISTORY.md

@@ -0,0 +1,323 @@
+# Terminal Command History - Documentation
+
+## Overview
+
+The Terminal Command History feature logs validated terminal commands executed via the web SSH terminal, providing:
+- **Command history** per host and user
+- **Search functionality** with instant filtering
+- **Security-first approach** - only safe commands are logged, sensitive data is blocked or masked
+
+## Architecture
+
+```
+Browser <--> ttyd <--> Terminal Gateway (PTY proxy) <--> SSH <--> Remote Host
+                              |
+                              v
+                    Command Logger (buffer)
+                              |
+                              v
+                    Command Policy Engine
+                              |
+                              v
+                    Database (terminal_command_logs)
+```
+
+## Components
+
+### Backend
+
+| File | Description |
+|------|-------------|
+| `app/security/command_policy.py` | Security policy engine (blocklist/allowlist/masking) |
+| `app/services/terminal_command_logger.py` | Buffer management and Enter detection |
+| `app/services/terminal_gateway.py` | PTY gateway for command interception |
+| `app/models/terminal_command_log.py` | SQLAlchemy model for command logs |
+| `app/crud/terminal_command_log.py` | CRUD operations for command logs |
+| `app/routes/terminal.py` | API endpoints for command history |
+| `app/schemas/terminal.py` | Pydantic schemas for API responses |
+
+### Frontend
+
+| File | Description |
+|------|-------------|
+| `app/main.js` | Terminal history UI logic |
+| `app/index.html` | CSS styles for history panel |
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `TERMINAL_COMMAND_RETENTION_DAYS` | `30` | Days to retain command logs |
+| `COMMAND_POLICY_CONFIG` | (none) | Path to custom policy YAML file |
+
+### Example .env
+
+```env
+# Terminal command history retention (days)
+TERMINAL_COMMAND_RETENTION_DAYS=30
+
+# Custom command policy (optional)
+# COMMAND_POLICY_CONFIG=/path/to/policy.yaml
+```
+
+## Security Policy
+
+### Blocklist (Commands NEVER Logged)
+
+Commands matching these patterns are **blocked** and never stored:
+
+- Password/secret keywords: `password`, `passwd`, `token`, `apikey`, `secret`
+- Docker login with credentials
+- `curl`/`wget` with Authorization headers
+- Export of sensitive environment variables
+- Access to SSH keys (`~/.ssh/id_rsa`, etc.)
+- Access to `/etc/shadow`
+- MySQL/PostgreSQL with passwords on command line
+- Cloud CLI authentication commands
+- Kubernetes secrets
+- Ansible vault operations
+
+### Allowlist (Commands That ARE Logged)
+
+Only commands matching these patterns are logged:
+
+- Basic commands: `ls`, `cd`, `pwd`, `whoami`, `id`, `uname`
+- System info: `df`, `du`, `free`, `uptime`
+- Process info: `ps`, `top`, `htop`
+- Network info: `ip addr`, `ifconfig`, `netstat`, `ping`
+- Systemd: `systemctl status|start|stop|restart`
+- Logs: `journalctl`, `tail`, `less`
+- Docker: `docker ps|logs|images|inspect|stats`
+- Package managers: `apt list`, `dnf list` (read operations)
+- Git: `git status|log|diff|branch`
+- File operations: `cat`, `grep`, `find` (non-sensitive paths)
+
+### Masking
+
+Even for allowed commands, sensitive values are masked:
+- `--password=secret` → `--password=***`
+- `--token=abc123` → `--token=***`
+- `Authorization: Bearer xyz` → `Authorization: Bearer ***`
+
+## Extending the Policy
+
+### Custom Policy File (YAML)
+
+Create a custom policy file:
+
+```yaml
+# /etc/homelab/command_policy.yaml
+
+# Policy mode: 'strict' (allowlist only) or 'permissive' (log everything not blocked)
+mode: strict
+
+# Additional blocklist patterns (regex)
+blocklist:
+  - '\bmy-custom-secret\b'
+  - '\bcompany-token\b'
+
+# Additional allowlist patterns (regex)
+allowlist:
+  - '^my-safe-tool\b'
+  - '^company-cli\s+(status|info)\b'
+
+# Additional masking patterns (regex, replacement)
+mask:
+  - ['(--company-key[=\s]+)\S+', '\1***']
+```
+
+Then set the environment variable:
+
+```env
+COMMAND_POLICY_CONFIG=/etc/homelab/command_policy.yaml
+```
+
+### Adding Patterns Programmatically
+
+```python
+from app.security.command_policy import get_command_policy
+
+policy = get_command_policy()
+
+# Add to blocklist
+policy.blocklist_patterns.append(re.compile(r'\bmy-secret\b', re.IGNORECASE))
+
+# Add to allowlist
+policy.allowlist_patterns.append(re.compile(r'^my-safe-command\b', re.IGNORECASE))
+```
+
+## API Endpoints
+
+### Get Command History for Host
+
+```http
+GET /api/terminal/{host_id}/command-history?query=&limit=50&offset=0
+Authorization: Bearer <token>
+```
+
+Response:
+```json
+{
+  "commands": [
+    {
+      "id": 1,
+      "command": "systemctl status nginx",
+      "created_at": "2024-12-18T10:30:00Z",
+      "host_name": "web-server-1",
+      "username": "admin"
+    }
+  ],
+  "total": 42,
+  "host_id": "host-123",
+  "query": null
+}
+```
+
+### Get Unique Commands for Host
+
+```http
+GET /api/terminal/{host_id}/command-history/unique?query=&limit=50
+Authorization: Bearer <token>
+```
+
+Response:
+```json
+{
+  "commands": [
+    {
+      "command": "docker ps",
+      "command_hash": "abc123...",
+      "last_used": "2024-12-18T10:30:00Z",
+      "execution_count": 15
+    }
+  ],
+  "total": 10,
+  "host_id": "host-123"
+}
+```
+
+### Get Global Command History
+
+```http
+GET /api/terminal/command-history?query=&host_id=&limit=50&offset=0
+Authorization: Bearer <token>
+```
+
+### Clear Host Command History (Admin Only)
+
+```http
+DELETE /api/terminal/{host_id}/command-history
+Authorization: Bearer <token>
+```
+
+### Purge Old Command History (Admin Only)
+
+```http
+POST /api/terminal/command-history/purge?days=30
+Authorization: Bearer <token>
+```
+
+## Database Schema
+
+### Table: `terminal_command_logs`
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | INTEGER | Primary key |
+| `created_at` | DATETIME | Timestamp (UTC) |
+| `host_id` | VARCHAR | Foreign key to hosts |
+| `user_id` | VARCHAR | Foreign key to users (nullable) |
+| `terminal_session_id` | VARCHAR(64) | Session reference |
+| `command` | TEXT | Masked command text |
+| `command_hash` | VARCHAR(64) | SHA-256 hash for dedup |
+| `source` | VARCHAR(20) | Always "terminal" |
+| `is_blocked` | BOOLEAN | Whether command was blocked |
+| `blocked_reason` | VARCHAR(255) | Reason for blocking |
+| `username` | VARCHAR(100) | User display name |
+| `host_name` | VARCHAR(100) | Host display name |
+
+### Indexes
+
+- `ix_terminal_cmd_host_created` (host_id, created_at)
+- `ix_terminal_cmd_user_created` (user_id, created_at)
+- `ix_terminal_cmd_host_hash` (host_id, command_hash)
+
+## Frontend Usage
+
+### Opening Command History
+
+1. Open a terminal session for a host
+2. Click the **History** button (clock icon) in the terminal header
+3. Search for commands or browse recent ones
+4. Click a command to copy it to clipboard
+5. Paste into terminal with `Ctrl+Shift+V`
+
+### Search Features
+
+- **Instant filtering**: Type to filter commands
+- **All hosts toggle**: Search across all hosts or just current
+- **Execution count**: See how often a command was used
+- **Relative timestamps**: "Il y a 5min", "Il y a 2h", etc.
+
+## Limitations
+
+1. **No real-time interception**: Currently, commands are not captured in real-time from the terminal. This would require a WebSocket proxy enhancement.
+
+2. **Buffer reconstruction**: Complex terminal interactions (cursor movement, tab completion) may not be perfectly reconstructed.
+
+3. **Platform dependency**: Full PTY interception requires Unix-like systems. Windows has limited support.
+
+4. **ttyd integration**: The current implementation works alongside ttyd but doesn't intercept its WebSocket traffic directly.
+
+## Future Enhancements
+
+1. **WebSocket proxy**: Intercept ttyd WebSocket to capture commands in real-time
+2. **Command favorites**: Allow users to star frequently used commands
+3. **Command aliases**: Create shortcuts for common commands
+4. **Export history**: Download command history as CSV/JSON
+5. **Audit log integration**: Link to broader audit logging system
+
+## Troubleshooting
+
+### Commands Not Being Logged
+
+1. Check if the command matches the allowlist patterns
+2. Verify the command isn't being blocked by the blocklist
+3. Check the application logs for policy decisions
+
+### History Panel Empty
+
+1. Ensure the host has had terminal sessions
+2. Check database connectivity
+3. Verify the retention policy hasn't purged old logs
+
+### Performance Issues
+
+1. Increase `TERMINAL_COMMAND_RETENTION_DAYS` carefully
+2. Monitor database size
+3. Run periodic purge jobs
+
+## Running Tests
+
+```bash
+# Run command policy tests
+pytest tests/backend/test_command_policy.py -v
+
+# Run buffer/logger tests
+pytest tests/backend/test_terminal_command_logger.py -v
+
+# Run API endpoint tests
+pytest tests/backend/test_terminal_command_history.py -v
+```
+
+## Migration
+
+Apply the database migration:
+
+```bash
+alembic upgrade head
+```
+
+This creates the `terminal_command_logs` table with all required indexes.