homelab_automation/documentation/TERMINAL_COMMAND_HISTORY.md

# Terminal Command History - Documentation

## Overview

The Terminal Command History feature logs validated terminal commands executed via the web SSH terminal, providing:
- **Command history** per host and user
- **Search functionality** with instant filtering
- **Security-first approach** - only safe commands are logged, sensitive data is blocked or masked

## Architecture

```
Browser <--> ttyd <--> Terminal Gateway (PTY proxy) <--> SSH <--> Remote Host
                              |
                              v
                    Command Logger (buffer)
                              |
                              v
                    Command Policy Engine
                              |
                              v
                    Database (terminal_command_logs)
```

## Components

### Backend

| File | Description |
|------|-------------|
| `app/security/command_policy.py` | Security policy engine (blocklist/allowlist/masking) |
| `app/services/terminal_command_logger.py` | Buffer management and Enter detection |
| `app/services/terminal_gateway.py` | PTY gateway for command interception |
| `app/models/terminal_command_log.py` | SQLAlchemy model for command logs |
| `app/crud/terminal_command_log.py` | CRUD operations for command logs |
| `app/routes/terminal.py` | API endpoints for command history |
| `app/schemas/terminal.py` | Pydantic schemas for API responses |

### Frontend

| File | Description |
|------|-------------|
| `app/main.js` | Terminal history UI logic |
| `app/index.html` | CSS styles for history panel |

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `TERMINAL_COMMAND_RETENTION_DAYS` | `30` | Days to retain command logs |
| `COMMAND_POLICY_CONFIG` | (none) | Path to custom policy YAML file |

### Example .env

```env
# Terminal command history retention (days)
TERMINAL_COMMAND_RETENTION_DAYS=30

# Custom command policy (optional)
# COMMAND_POLICY_CONFIG=/path/to/policy.yaml
```

## Security Policy

### Blocklist (Commands NEVER Logged)

Commands matching these patterns are **blocked** and never stored:

- Password/secret keywords: `password`, `passwd`, `token`, `apikey`, `secret`
- Docker login with credentials
- `curl`/`wget` with Authorization headers
- Export of sensitive environment variables
- Access to SSH keys (`~/.ssh/id_rsa`, etc.)
- Access to `/etc/shadow`
- MySQL/PostgreSQL with passwords on command line
- Cloud CLI authentication commands
- Kubernetes secrets
- Ansible vault operations

### Allowlist (Commands That ARE Logged)

Only commands matching these patterns are logged:

- Basic commands: `ls`, `cd`, `pwd`, `whoami`, `id`, `uname`
- System info: `df`, `du`, `free`, `uptime`
- Process info: `ps`, `top`, `htop`
- Network info: `ip addr`, `ifconfig`, `netstat`, `ping`
- Systemd: `systemctl status|start|stop|restart`
- Logs: `journalctl`, `tail`, `less`
- Docker: `docker ps|logs|images|inspect|stats`
- Package managers: `apt list`, `dnf list` (read operations)
- Git: `git status|log|diff|branch`
- File operations: `cat`, `grep`, `find` (non-sensitive paths)

### Masking

Even for allowed commands, sensitive values are masked:
- `--password=secret` → `--password=***`
- `--token=abc123` → `--token=***`
- `Authorization: Bearer xyz` → `Authorization: Bearer ***`

## Extending the Policy

### Custom Policy File (YAML)

Create a custom policy file:

```yaml
# /etc/homelab/command_policy.yaml

# Policy mode: 'strict' (allowlist only) or 'permissive' (log everything not blocked)
mode: strict

# Additional blocklist patterns (regex)
blocklist:
  - '\bmy-custom-secret\b'
  - '\bcompany-token\b'

# Additional allowlist patterns (regex)
allowlist:
  - '^my-safe-tool\b'
  - '^company-cli\s+(status|info)\b'

# Additional masking patterns (regex, replacement)
mask:
  - ['(--company-key[=\s]+)\S+', '\1***']
```

Then set the environment variable:

```env
COMMAND_POLICY_CONFIG=/etc/homelab/command_policy.yaml
```

### Adding Patterns Programmatically

```python
from app.security.command_policy import get_command_policy

policy = get_command_policy()

# Add to blocklist
policy.blocklist_patterns.append(re.compile(r'\bmy-secret\b', re.IGNORECASE))

# Add to allowlist
policy.allowlist_patterns.append(re.compile(r'^my-safe-command\b', re.IGNORECASE))
```

## API Endpoints

### Get Command History for Host

```http
GET /api/terminal/{host_id}/command-history?query=&limit=50&offset=0
Authorization: Bearer <token>
```

Response:
```json
{
  "commands": [
    {
      "id": 1,
      "command": "systemctl status nginx",
      "created_at": "2024-12-18T10:30:00Z",
      "host_name": "web-server-1",
      "username": "admin"
    }
  ],
  "total": 42,
  "host_id": "host-123",
  "query": null
}
```

### Get Unique Commands for Host

```http
GET /api/terminal/{host_id}/command-history/unique?query=&limit=50
Authorization: Bearer <token>
```

Response:
```json
{
  "commands": [
    {
      "command": "docker ps",
      "command_hash": "abc123...",
      "last_used": "2024-12-18T10:30:00Z",
      "execution_count": 15
    }
  ],
  "total": 10,
  "host_id": "host-123"
}
```

### Get Global Command History

```http
GET /api/terminal/command-history?query=&host_id=&limit=50&offset=0
Authorization: Bearer <token>
```

### Clear Host Command History (Admin Only)

```http
DELETE /api/terminal/{host_id}/command-history
Authorization: Bearer <token>
```

### Purge Old Command History (Admin Only)

```http
POST /api/terminal/command-history/purge?days=30
Authorization: Bearer <token>
```

## Database Schema

### Table: `terminal_command_logs`

| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER | Primary key |
| `created_at` | DATETIME | Timestamp (UTC) |
| `host_id` | VARCHAR | Foreign key to hosts |
| `user_id` | VARCHAR | Foreign key to users (nullable) |
| `terminal_session_id` | VARCHAR(64) | Session reference |
| `command` | TEXT | Masked command text |
| `command_hash` | VARCHAR(64) | SHA-256 hash for dedup |
| `source` | VARCHAR(20) | Always "terminal" |
| `is_blocked` | BOOLEAN | Whether command was blocked |
| `blocked_reason` | VARCHAR(255) | Reason for blocking |
| `username` | VARCHAR(100) | User display name |
| `host_name` | VARCHAR(100) | Host display name |

### Indexes

- `ix_terminal_cmd_host_created` (host_id, created_at)
- `ix_terminal_cmd_user_created` (user_id, created_at)
- `ix_terminal_cmd_host_hash` (host_id, command_hash)

## Frontend Usage

### Opening Command History

1. Open a terminal session for a host
2. Click the **History** button (clock icon) in the terminal header
3. Search for commands or browse recent ones
4. Click a command to copy it to clipboard
5. Paste into terminal with `Ctrl+Shift+V`

### Search Features

- **Instant filtering**: Type to filter commands
- **All hosts toggle**: Search across all hosts or just current
- **Execution count**: See how often a command was used
- **Relative timestamps**: "Il y a 5min", "Il y a 2h", etc.

## Limitations

1. **No real-time interception**: Currently, commands are not captured in real-time from the terminal. This would require a WebSocket proxy enhancement.

2. **Buffer reconstruction**: Complex terminal interactions (cursor movement, tab completion) may not be perfectly reconstructed.

3. **Platform dependency**: Full PTY interception requires Unix-like systems. Windows has limited support.

4. **ttyd integration**: The current implementation works alongside ttyd but doesn't intercept its WebSocket traffic directly.

## Future Enhancements

1. **WebSocket proxy**: Intercept ttyd WebSocket to capture commands in real-time
2. **Command favorites**: Allow users to star frequently used commands
3. **Command aliases**: Create shortcuts for common commands
4. **Export history**: Download command history as CSV/JSON
5. **Audit log integration**: Link to broader audit logging system

## Troubleshooting

### Commands Not Being Logged

1. Check if the command matches the allowlist patterns
2. Verify the command isn't being blocked by the blocklist
3. Check the application logs for policy decisions

### History Panel Empty

1. Ensure the host has had terminal sessions
2. Check database connectivity
3. Verify the retention policy hasn't purged old logs

### Performance Issues

1. Increase `TERMINAL_COMMAND_RETENTION_DAYS` carefully
2. Monitor database size
3. Run periodic purge jobs

## Running Tests

```bash
# Run command policy tests
pytest tests/backend/test_command_policy.py -v

# Run buffer/logger tests
pytest tests/backend/test_terminal_command_logger.py -v

# Run API endpoint tests
pytest tests/backend/test_terminal_command_history.py -v
```

## Migration

Apply the database migration:

```bash
alembic upgrade head
```

This creates the `terminal_command_logs` table with all required indexes.