path: root/ARCHITECTURE.md

# JCHAT Architecture Document

**Version:** 1.0  
**Date:** August 15, 2025  
**Status:** Living Document

## Executive Summary

JCHAT is a real-time chat system built on the JMAP (JSON Meta Application Protocol) standard. It provides a scalable, extensible foundation for messaging with a clean separation between protocol, server implementation, and client interfaces.

## Current Architecture Overview

### System Components

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Web Client    │────│   JMAP Server    │────│   Database      │
│  (HTML/JS/CSS)  │    │   (Erlang/OTP)   │    │   (Mnesia)      │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         │              ┌────────────────┐               │
         └──────────────│  HTTP/REST     │───────────────┘
                        │  JSON/JMAP     │
                        └────────────────┘
```

### Protocol Layer: JMAP Extension

**Base Protocol:** RFC 8620 - JSON Meta Application Protocol  
**Extension:** JCHAT Capability (`urn:ietf:params:jmap:chat`)

**Core Objects:**
- `Conversation`: Chat rooms/threads with metadata and participants
- `Message`: Individual messages with content, reactions, and delivery status  
- `Participant`: User membership and roles within conversations
- `Presence`: User availability and status information

**Methods Implemented:**
- `Conversation/get`, `Conversation/set`, `Conversation/query`, `Conversation/changes`
- `Message/get`, `Message/set`, `Message/query`, `Message/changes`
- `Participant/get`, `Participant/set`, `Participant/changes`
- `Presence/get`, `Presence/set`

### Server Architecture (Erlang/OTP)

```
Application Layer (jchat_app)
    │
Supervisor Layer (jchat_sup)
    │
┌─────────────────────────────────────────────────────────────┐
│                    Worker Processes                          │
├─────────────────┬─────────────────┬─────────────────────────┤
│   HTTP Server   │   Push Manager  │   Presence Manager      │
│   (jchat_http)  │  (jchat_push)   │   (jchat_presence)      │
└─────────────────┴─────────────────┴─────────────────────────┘
    │                       │                       │
┌─────────────────────────────────────────────────────────────┐
│                    Business Logic Layer                      │
├─────────────────┬─────────────────┬─────────────────────────┤
│  JMAP Methods   │   Utilities     │    Database Layer       │
│(jchat_methods)  │(jchat_utils)    │    (jchat_db)           │
└─────────────────┴─────────────────┴─────────────────────────┘
```

**Key Modules:**
- `jchat_http`: Cowboy-based HTTP server, CORS, request routing
- `jchat_methods`: JMAP method implementations, business logic
- `jchat_db`: Mnesia database operations, CRUD, queries
- `jchat_utils`: Shared utilities (UUID generation, timestamps, formatting)
- `jchat_push`: Server-sent events for real-time updates
- `jchat_presence`: User status and availability management

### Client Architecture

**Technology Stack:** Vanilla HTML5, CSS3, JavaScript (ES6+)  
**Deployment:** Static files served by lightweight HTTP server (shttpd, Python, nginx)

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   UI Layer      │────│  Application     │────│  JMAP Client    │
│  (index.html)   │    │  Logic           │    │  Library        │
│                 │    │  (app.js)        │    │ (jmap-client.js)│
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │                       │
                                │              ┌────────────────┐
                                └──────────────│  Server API    │
                                               │ (HTTP/JSON)    │
                                               └────────────────┘
```

**Features:**
- Real-time message synchronization (5-second polling)
- Responsive design for mobile/desktop
- Local state management and caching
- User settings persistence (localStorage)
- Modal-based UI for settings and new conversations

### Database Schema (Mnesia)

**Tables:**
- `user`: User accounts and profile information
- `conversation`: Chat rooms with metadata, participants, settings
- `message`: Individual messages with content, reactions, delivery status
- `participant`: User membership and roles in conversations  
- `presence`: User availability and status
- `state_counter`: JMAP state tracking for real-time synchronization

**Storage:** In-memory with optional disk persistence (ram_copies)  
**Consistency:** ACID transactions, distributed capabilities

## Current Implementation Status

### ✅ Completed Features
- [x] JMAP protocol foundation and session management
- [x] Basic conversation management (create, list, select)
- [x] Message sending and real-time synchronization  
- [x] User display name settings with persistence
- [x] Cross-origin resource sharing (CORS) support
- [x] Responsive web UI with modern design
- [x] Static file deployment (no build process)

### 🔄 Partially Implemented
- [ ] Message history loading (implemented but needs optimization)
- [ ] Real-time updates (polling works, changes detection needs improvement)
- [ ] Error handling and user feedback (basic implementation)

### ❌ Not Yet Implemented
- [ ] User authentication and sessions
- [ ] Role-based access control (RBAC)
- [ ] Private vs public conversations
- [ ] Message attachments and file uploads
- [ ] Message reactions and threading
- [ ] Push notifications
- [ ] Persistent database storage

---

## Future Architecture Considerations

### 1. User Authentication & Authorization

#### Current State
- **Authentication:** None - users set display names locally
- **Session Management:** No server-side sessions
- **User Identity:** Display names stored in client localStorage

#### Proposed Architecture

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Client Auth   │────│   Auth Service   │────│  User Database  │
│  (Login Form)   │    │ (OAuth2/JWT/etc) │    │   (Accounts)    │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         │              ┌────────────────┐               │
         └──────────────│  Session Mgmt  │───────────────┘
                        │ (Redis/Mnesia) │
                        └────────────────┘
```

**Options to Consider:**
1. **JWT-based Authentication**
   - Stateless tokens
   - Self-contained user info
   - Easy to scale horizontally

2. **OAuth2/OpenID Connect**
   - External identity providers (Google, GitHub, etc.)
   - Standardized protocols
   - Reduced authentication complexity

3. **Traditional Session-Based**
   - Server-side session storage
   - Cookie-based authentication
   - More control over session lifecycle

**Recommended Approach:**
```erlang
%% User record with authentication
-record(user, {
    id,              % binary() - Unique user ID
    username,        % binary() - Unique username
    email,           % binary() - Email address
    password_hash,   % binary() - Hashed password (if local auth)
    display_name,    % binary() - Display name
    avatar_url,      % binary() | null - Avatar image URL
    created_at,      % binary() - ISO8601 timestamp
    last_login_at,   % binary() | null - Last login time
    is_active,       % boolean() - Account status
    auth_provider,   % binary() - 'local' | 'google' | 'github' etc
    auth_provider_id % binary() | null - External provider user ID
}).

%% Session record
-record(session, {
    id,           % binary() - Session token/ID
    user_id,      % binary() - User ID
    created_at,   % binary() - ISO8601 timestamp
    expires_at,   % binary() - ISO8601 timestamp
    ip_address,   % binary() - Client IP
    user_agent    % binary() - Client user agent
}).
```

### 2. Role-Based Access Control (RBAC)

#### Proposed Permission Model

```
System Level Permissions:
├── admin.system          (Full system administration)
├── admin.users           (User management)
├── admin.conversations   (Global conversation management)
├── user.create           (Create new conversations)
└── user.invite           (Invite others to conversations)

Conversation Level Permissions:
├── conversation.admin    (Full conversation control)
├── conversation.moderate (Moderation capabilities)
├── conversation.write    (Send messages)
├── conversation.read     (View messages)
└── conversation.invite   (Invite new participants)

Message Level Permissions:
├── message.delete.own    (Delete own messages)
├── message.delete.any    (Delete any message - moderators)
├── message.edit.own      (Edit own messages)
└── message.react         (Add reactions to messages)
```

#### Database Schema for RBAC

```erlang
%% Role definition
-record(role, {
    id,           % binary() - Role ID
    name,         % binary() - Human readable name
    description,  % binary() - Role description
    permissions,  % [binary()] - List of permission strings
    is_system,    % boolean() - System role vs user-defined
    created_at    % binary() - ISO8601 timestamp
}).

%% User role assignments (system-wide)
-record(user_role, {
    user_id,      % binary() - User ID
    role_id,      % binary() - Role ID
    granted_by,   % binary() - User ID who granted the role
    granted_at    % binary() - ISO8601 timestamp
}).

%% Conversation-specific role assignments
-record(conversation_role, {
    user_id,         % binary() - User ID
    conversation_id, % binary() - Conversation ID
    role_id,         % binary() - Role ID
    granted_by,      % binary() - User ID who granted the role
    granted_at       % binary() - ISO8601 timestamp
}).
```

#### Permission Checking Architecture

```erlang
%% Permission checking API
jchat_auth:check_permission(UserId, Permission) -> boolean().
jchat_auth:check_permission(UserId, Permission, ConversationId) -> boolean().

%% Example usage in JMAP methods
handle_message_set(Args, AccountId) ->
    ConversationId = get_conversation_from_args(Args),
    case jchat_auth:check_permission(AccountId, <<"message.write">>, ConversationId) of
        true -> 
            %% Proceed with message creation
            proceed_with_message_creation(Args, AccountId);
        false ->
            {error, #{type => <<"forbidden">>, description => <<"Insufficient permissions">>}}
    end.
```

### 3. Conversation Types & Privacy

#### Current State
- All conversations are essentially "public" (no access control)
- No distinction between different conversation types

#### Proposed Conversation Types

```erlang
%% Enhanced conversation record
-record(conversation, {
    id,                  % binary() - Conversation ID
    title,              % binary() | null - Conversation title
    description,        % binary() | null - Description
    conversation_type,   % binary() - Type of conversation
    privacy_level,      % binary() - Privacy setting
    created_at,         % binary() - ISO8601 timestamp
    updated_at,         % binary() - ISO8601 timestamp
    created_by,         % binary() - Creator user ID
    is_archived,        % boolean() - Archived status
    is_muted,          % boolean() - Muted status
    participant_ids,    % [binary()] - Participant IDs
    last_message_id,    % binary() | null - Last message ID
    last_message_at,    % binary() | null - Last message timestamp
    unread_count,       % integer() - Unread message count
    message_count,      % integer() - Total message count
    settings,          % map() - Conversation settings
    metadata           % map() | null - Additional metadata
}).
```

**Conversation Types:**
- `<<"direct">>` - Direct message between 2 users
- `<<"group">>` - Private group chat (invite-only)
- `<<"channel">>` - Public channel (discoverable, open join)
- `<<"announcement">>` - Broadcast channel (read-only for most users)

**Privacy Levels:**
- `<<"private">>` - Invite-only, not discoverable
- `<<"public">>` - Anyone can join, publicly discoverable  
- `<<"restricted">>` - Discoverable but requires approval to join

#### Channel/Room Discovery

```erlang
%% Public channel discovery
-record(channel_directory, {
    conversation_id,  % binary() - Conversation ID
    title,           % binary() - Channel title
    description,     % binary() - Channel description
    category,        % binary() - Category/topic
    member_count,    % integer() - Number of members
    activity_score,  % float() - Activity ranking
    is_featured,     % boolean() - Featured channel
    created_at       % binary() - ISO8601 timestamp
}).
```

### 4. Database Architecture Evolution

#### Current State: Mnesia
**Pros:**
- Built into Erlang/OTP
- ACID transactions
- Distributed capabilities
- No external dependencies
- RAM-based performance

**Cons:**
- Memory limitations for large datasets
- Limited query capabilities
- No full-text search
- Backup/restore complexity

#### Future Database Considerations

##### Option 1: Enhanced Mnesia Setup
```erlang
%% Hybrid storage approach
Tables = [
    {user, disc_copies},           % Users - persistent
    {session, ram_copies},         % Sessions - temporary
    {conversation, disc_copies},   % Conversations - persistent
    {message, disc_only_copies},   % Messages - disk-based for large volume
    {presence, ram_copies},        % Presence - temporary
    {state_counter, disc_copies}   % State - persistent
].
```

##### Option 2: PostgreSQL Backend
```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Erlang App    │────│   Database Pool  │────│   PostgreSQL    │
│                 │    │    (Poolboy)     │    │                 │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │
                       ┌────────────────┐
                       │  SQL Queries   │
                       │  (epgsql/pgapp)│
                       └────────────────┘
```

**Benefits:**
- Mature, battle-tested
- Full-text search capabilities
- JSON support for flexible schema
- Excellent tooling and monitoring
- Horizontal scaling options (read replicas)

**Schema Example:**
```sql
-- Users table
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    password_hash VARCHAR(255),
    display_name VARCHAR(100),
    avatar_url TEXT,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    last_login_at TIMESTAMP WITH TIME ZONE,
    is_active BOOLEAN DEFAULT true,
    auth_provider VARCHAR(50) DEFAULT 'local',
    auth_provider_id VARCHAR(255),
    metadata JSONB DEFAULT '{}'
);

-- Conversations table
CREATE TABLE conversations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    title VARCHAR(255),
    description TEXT,
    conversation_type VARCHAR(20) NOT NULL DEFAULT 'group',
    privacy_level VARCHAR(20) NOT NULL DEFAULT 'private',
    created_by UUID REFERENCES users(id),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_archived BOOLEAN DEFAULT false,
    settings JSONB DEFAULT '{}',
    metadata JSONB DEFAULT '{}'
);

-- Messages table with partitioning for scale
CREATE TABLE messages (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    conversation_id UUID REFERENCES conversations(id),
    sender_id UUID REFERENCES users(id),
    body TEXT NOT NULL,
    body_type VARCHAR(50) DEFAULT 'text/plain',
    sent_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    edited_at TIMESTAMP WITH TIME ZONE,
    is_deleted BOOLEAN DEFAULT false,
    reply_to_message_id UUID REFERENCES messages(id),
    attachments JSONB DEFAULT '[]',
    reactions JSONB DEFAULT '[]',
    metadata JSONB DEFAULT '{}'
) PARTITION BY RANGE (sent_at);

-- Full-text search
CREATE INDEX messages_body_fts ON messages USING GIN(to_tsvector('english', body));
```

##### Option 3: Hybrid Approach
- **Mnesia**: Real-time data (sessions, presence, caching)
- **PostgreSQL**: Persistent data (users, messages, conversations)
- **Redis**: Caching and pub/sub for real-time features

#### Message Storage Scaling Strategy

For high-volume messaging, consider:

1. **Message Archival**
   ```erlang
   %% Archive old messages to separate storage
   -record(message_archive, {
       id,
       original_message,
       archived_at,
       archive_reason
   }).
   ```

2. **Message Partitioning**
   - Partition by time (monthly tables)
   - Partition by conversation ID
   - Hot/warm/cold data tiers

3. **Search Integration**
   - Elasticsearch for full-text search
   - Index message content asynchronously
   - Search API separate from chat API

---

## Implementation Roadmap

### Phase 1: Authentication & Authorization (Weeks 1-2)
- [ ] Implement JWT-based authentication
- [ ] Add user registration/login endpoints
- [ ] Basic role system (admin, user, moderator)
- [ ] Secure existing JMAP endpoints

### Phase 2: Enhanced Conversations (Weeks 3-4)
- [ ] Conversation types (direct, group, channel)
- [ ] Privacy levels and access control
- [ ] Channel discovery and joining
- [ ] Participant management UI

### Phase 3: Database Evolution (Weeks 5-6)
- [ ] Evaluate PostgreSQL migration
- [ ] Implement database abstraction layer
- [ ] Message archival and pagination
- [ ] Full-text search capabilities

### Phase 4: Advanced Features (Weeks 7-8)
- [ ] File attachments and media
- [ ] Message reactions and threading
- [ ] Push notifications
- [ ] Presence and typing indicators

### Phase 5: Scaling & Production (Weeks 9-10)
- [ ] Horizontal scaling architecture
- [ ] Monitoring and observability
- [ ] Performance optimization
- [ ] Security hardening and audit

---

## Security Considerations

### Current Security Status
- **Authentication:** ❌ None implemented
- **Authorization:** ❌ No access control
- **Input Validation:** ⚠️ Basic validation
- **CORS:** ✅ Properly configured
- **HTTPS:** ❌ Not enforced (development)

### Security Roadmap

#### Authentication Security
- [ ] Secure password hashing (bcrypt/scrypt)
- [ ] JWT token security (short expiry, refresh tokens)
- [ ] Rate limiting on authentication endpoints
- [ ] Account lockout on failed attempts
- [ ] Two-factor authentication (TOTP)

#### API Security
- [ ] Input validation and sanitization
- [ ] SQL injection prevention (parameterized queries)
- [ ] XSS prevention (content escaping)
- [ ] CSRF protection for state-changing operations
- [ ] Rate limiting per user/IP

#### Transport Security
- [ ] HTTPS enforcement (TLS 1.3)
- [ ] HTTP Strict Transport Security (HSTS)
- [ ] Certificate pinning for mobile apps
- [ ] Secure cookie settings

#### Data Protection
- [ ] Encryption at rest (database encryption)
- [ ] PII data handling compliance
- [ ] Message retention policies
- [ ] User data export/deletion (GDPR)

---

## Monitoring & Observability

### Current Monitoring
- **Logging:** ✅ Basic Erlang logging
- **Metrics:** ❌ None implemented
- **Health Checks:** ❌ None implemented
- **Error Tracking:** ⚠️ Console errors only

### Proposed Monitoring Stack

```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Application   │────│    Telemetry     │────│   Prometheus    │
│                 │    │   (Metrics)      │    │                 │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                       │                       │
         │              ┌────────────────┐               │
         └──────────────│     Logging    │───────────────┘
                        │  (ELK Stack)   │
                        └────────────────┘
```

#### Key Metrics to Track
- **Performance**: Response times, throughput, error rates
- **Business**: Active users, messages sent, conversations created
- **Infrastructure**: CPU, memory, disk usage, connection pools
- **Security**: Failed login attempts, rate limit hits, suspicious activity

#### Alerting Strategy
- **Critical**: System down, database connection lost
- **Warning**: High error rate, memory usage above threshold
- **Info**: New user registrations, system updates

---

## Deployment & DevOps

### Current Deployment
- **Development**: Local rebar3 shell, static file server
- **Production**: ❌ Not implemented

### Proposed Deployment Architecture

#### Containerization
```dockerfile
# Erlang application container
FROM erlang:26-alpine
COPY _build/prod/rel/jchat /opt/jchat
EXPOSE 8080
CMD ["/opt/jchat/bin/jchat", "foreground"]
```

#### Container Orchestration
```yaml
# docker-compose.yml for local development
version: '3.8'
services:
  jchat-server:
    build: ./server
    ports:
      - "8080:8080"
    environment:
      - DATABASE_URL=postgresql://user:pass@db:5432/jchat
    depends_on:
      - db
      - redis
  
  jchat-client:
    image: nginx:alpine
    volumes:
      - ./client:/usr/share/nginx/html
    ports:
      - "3000:80"
  
  db:
    image: postgres:15
    environment:
      POSTGRES_DB: jchat
      POSTGRES_USER: jchat
      POSTGRES_PASSWORD: password
    volumes:
      - postgres_data:/var/lib/postgresql/data
  
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  postgres_data:
```

#### Production Considerations
- **Load Balancing**: HAProxy/nginx for horizontal scaling
- **Database**: Managed PostgreSQL (AWS RDS, Google Cloud SQL)
- **Caching**: Redis cluster for session management
- **CDN**: CloudFlare/AWS CloudFront for static assets
- **Monitoring**: Datadog/New Relic for APM

---

## Testing Strategy

### Current Testing
- **Unit Tests**: ⚠️ Basic test suite structure
- **Integration Tests**: ❌ Not implemented
- **Performance Tests**: ❌ Not implemented
- **Security Tests**: ❌ Not implemented

### Comprehensive Testing Plan

#### Unit Testing (Erlang)
```erlang
%% Example test structure
-module(jchat_methods_tests).
-include_lib("eunit/include/eunit.hrl").

conversation_create_test() ->
    %% Test conversation creation logic
    Args = #{<<"title">> => <<"Test Conv">>, <<"description">> => <<"Test">>},
    {ok, Result} = jchat_methods:handle_conversation_set(Args, <<"user1">>),
    ?assertMatch(#{<<"created">> := CreatedMap} when map_size(CreatedMap) > 0, Result).
```

#### Integration Testing
- JMAP protocol compliance tests
- Database operations end-to-end
- HTTP API endpoint testing
- WebSocket/SSE real-time features

#### Performance Testing
- Message throughput benchmarks
- Concurrent user simulation
- Database query optimization
- Memory usage profiling

#### Security Testing
- Authentication bypass attempts
- Input validation testing
- Rate limiting verification
- Session management security

---

This architecture document provides a comprehensive roadmap for evolving JCHAT from a proof-of-concept to a production-ready system. Each section can be expanded based on specific implementation decisions and requirements.