Lesson 05: Designing a Signaling Protocol — JSON over WebSocket

Prerequisites: Lesson 02: Peer-to-Peer Networking, Lesson 03: WebRTC Fundamentals

Key source files:

• sidecar/src/protocol.rs — SignalingMessage struct definition
• registry/src/signaling.rs — handle_message function, message relay logic
• sidecar/src/signaling.rs — handle_signaling_message, sidecar-side processing
• registry/src/main.rs — WebSocket upgrade handler, sender verification

---

What is a Protocol?

A protocol is a contract between two or more systems about how they will communicate. It defines three things:

1. Message format — the structure and encoding of data on the wire
2. Message sequence — the order in which messages are exchanged
3. Message semantics — what each message means and what the receiver should do

chatixia-mesh defines three application protocols:

Protocol	Transport	Purpose
Signaling	JSON over WebSocket	Coordinate WebRTC connection setup between peers
Mesh	JSON over WebRTC DataChannel	Agent-to-agent communication (tasks, prompts, status)
IPC	JSON lines over Unix socket	Bridge between sidecar and Python agent

This lesson focuses on the signaling protocol — the smallest and most critical of the three. Without it, no two peers can establish a direct connection.

JSON over WebSocket was chosen because signaling involves fewer than 10 high-importance messages per connection setup. It provides persistent connections, low overhead, human-readable messages for debugging, and the ability to add new message types without infrastructure changes. The SDP payloads are already text, so binary encoding saves almost nothing.

---

The SignalingMessage Struct

The signaling protocol is built on a single message type with four fields (sidecar/src/protocol.rs):

#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct SignalingMessage {
    #[serde(rename = "type")]
    pub msg_type: String,
    pub peer_id: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub target_id: Option<String>,
    #[serde(default)]
    pub payload: serde_json::Value,
}

On the wire:

{
  "type": "offer",
  "peer_id": "agent-001",
  "target_id": "agent-002",
  "payload": { "sdp": "v=0\r\no=- 123456 ..." }
}

Message Types

Type	Direction	Purpose
register	Sidecar -> Registry	”I’m online, tell me who else is here”
peer_list	Registry -> Sidecar	”Here are the other peers you can connect to”
offer	Sidecar -> Registry -> Sidecar	WebRTC SDP offer, relayed to target peer
answer	Sidecar -> Registry -> Sidecar	WebRTC SDP answer, relayed back to offerer
ice_candidate	Sidecar -> Registry -> Sidecar	ICE candidate, relayed to target peer

There is also heartbeat (keep-alive, no action) and the catch-all for unknown types (logged and ignored).

Field Details

• type determines how the message is processed via match statements. The #[serde(rename = "type")] annotation maps Rust’s msg_type field to the JSON key "type" (since type is a reserved keyword).
• peer_id identifies the sender, used for routing responses and sender verification.
• target_id is optional — present on directed messages (offer, answer, ice_candidate), absent on broadcasts (register). skip_serializing_if = "Option::is_none" omits the field entirely when None.
• payload carries type-specific data as serde_json::Value — a dynamically-typed JSON value. The signaling layer does not need to understand SDP or ICE contents; it just delivers them.

---

Sequence: Two Peers Connecting

  Sidecar A                    Registry                    Sidecar B
     |                            |                            |
     |--- POST /api/token ------->|                            |
     |<-- { token, peer_id } ----|                            |
     |--- WebSocket /ws?token --->|                            |
     |                            |                            |
     |  (1) register              |                            |
     |--- { type: "register" } -->|                            |
     |  (2) peer_list: []         |                            |
     |<---------------------------|                            |
     |                            |<--- register --------------|
     |                            |--- (4) peer_list: ["A"] -->|
     |                            |                            |
     |  (5) offer                 |<-- offer from B ----------|
     |<-- offer relayed ---------|                            |
     |--- (6) answer ----------->|--- answer relayed -------->|
     |                            |                            |
     |  (7-N) ice_candidates exchanged bidirectionally         |
     |============= ICE checks, DTLS handshake ===============|
     |<========= DataChannel open ============================>|

Token exchange: Each sidecar exchanges an API key for a JWT via POST /api/token. The JWT’s sub claim contains the peer_id, bound to the WebSocket for sender verification.

Registration: The sidecar sends register; the registry responds with a peer_list of authorized, connected peers. When B registers and sees A in its list, it initiates a WebRTC connection.

Offer/Answer: B creates an RTCPeerConnection, generates an SDP offer, and sends it through signaling. A receives it, creates its own connection, generates an answer, and sends it back. The registry relays both messages without parsing their SDP payloads.

ICE Candidates: Both sides gather and exchange ICE candidates (possible network paths) through signaling. Each candidate contains candidate, sdpMid, and sdpMLineIndex fields per the WebRTC standard.

After signaling: Once ICE finds a path and DTLS encrypts it, the DataChannel opens. The signaling server is no longer involved — signaling is transient, data flow is direct. The registry can restart without interrupting active peer-to-peer conversations.

---

Sender Verification

Every incoming message’s peer_id must match the identity established during WebSocket authentication (registry/src/main.rs):

if sm.peer_id != peer_id {
    error!("[WS] peer_id mismatch: expected={}, got={}", peer_id, sm.peer_id);
    continue;
}

The trust chain: (1) the sidecar exchanges an API key for a JWT — the registry assigns the peer_id; (2) on WebSocket connect, the JWT is validated and peer_id is bound to the connection; (3) every message is checked against that bound identity. This prevents a compromised sidecar from sending messages with a forged peer_id.

The registry is a trusted relay — it can see all peers, deliver messages, and verify identities. The signaling protocol does not include end-to-end encryption between peers; it assumes the registry is honest. This is reasonable because the registry operator controls the mesh — if compromised, far more powerful attacks (forging JWTs) are available.

---

Peer List Filtering

Not every connected peer appears in every peer_list. The registry filters based on pairing status.

Authorization Model

Two categories of peers are authorized:

• Approved peers — went through the pairing system and were admin-approved
• Legacy peers — have static API keys (from api_keys.json), auto-authorized

A peer must be in either set to appear in peer lists.

Filtering Rules

1. Authorized peers see only other authorized peers. Pending, rejected, or revoked peers are invisible.
2. Unauthorized peers see nobody. They receive an empty peer_list but can maintain the WebSocket (useful for the pairing flow).

Authorization sets are recomputed on every incoming message, not cached:

let approved = state.pairing.approved_peer_ids();
let legacy = state.auth.api_key_peer_ids();

This means revocation takes effect immediately. A revoked agent is excluded from the next peer list. Its existing DataChannels remain open (signaling cannot close them), but no new connections will be established.

---

Protocol Evolution

The Unknown-Type Pattern

Both registry and sidecar handle unrecognized message types by logging and ignoring them:

// Registry
other => { warn!("[SIG] unknown message type: {}", other); }

// Sidecar
_ => { warn!("[SIG] unhandled message type: {}", msg.msg_type); }

No panic, no error, no disconnection. This is the backward compatibility mechanism: a new message type is invisible to old clients. Adding a ping/pong health check, for example, only requires adding handlers in updated code — old sidecars simply log “unknown type” and continue.

Limitations

This pattern handles additive changes but not: removing required types (old clients hang waiting), changing existing type semantics (old parsers fail), or changing the envelope structure (everything breaks). For those, you would need version negotiation. chatixia-mesh does not implement this yet because all components are deployed together.

The heartbeat type demonstrates graceful placeholder design — accepted and ignored, ready for future liveness-tracking logic.

---

How the Pieces Fit Together

The signaling protocol is a thin layer that solves the bootstrapping problem: two peers on different networks cannot exchange SDP directly because they do not know each other’s addresses. The registry acts as a rendezvous point.

Component	Responsibility	Does NOT do
sidecar/src/protocol.rs	Defines the message struct	Process or route messages
registry/src/signaling.rs	Relays messages between peers	Parse payloads, manage WebRTC state
sidecar/src/signaling.rs	Processes received messages, drives WebRTC	Relay to other peers
registry/src/main.rs	Authenticates and verifies senders	Understand message semantics

Each component handles one concern. The protocol struct is defined once and used identically on both sides — there is no client-vs-server version mismatch risk.

---

Previous: Lesson 04: Async Programming Patterns | Next: Lesson 06: Inter-Process Communication