Off-the-Record Messaging (OTR)

Off-the-Record Messaging (OTR)

Introduction

Off-the-Record Messaging, OTR, encrypts instant-message conversations with two properties that most earlier tools lacked: forward secrecy, so recording today’s traffic does not expose yesterday’s, and deniability, so a transcript alone cannot later prove who said what. Both come from the same idea: authenticate the channel, not the messages. OTR signs a key exchange once, then protects every message after that with symmetric keys that get discarded and, eventually, deliberately published.

This guide builds OTR version 3 end to end. It covers the SIGMA-style Authenticated Key Exchange with real DSA signatures, the per-message Diffie-Hellman ratchet of Data Messages, and the Socialist Millionaires’ Protocol, which lets two people confirm they share a secret without ever revealing it. The DSA implementation is checked against NIST’s own published test vectors. Every other building block is cross-checked between an independent Python and JavaScript implementation, line for line.

Table of Contents

• History and Standards
• The Design in Brief
• The Authenticated Key Exchange
• Key Derivation and the Session ID
• Data Messages and the Diffie-Hellman Ratchet
• The Socialist Millionaires’ Protocol
• Interactive Visualizer
• A Worked Example
• Python Implementation
• Where It Stands
• Limitations
• Security Status
• FAQ
• References

History and Standards

Nikita Borisov, Ian Goldberg, and Eric Brewer published OTR at the 2004 ACM Workshop on Privacy in the Electronic Society. The paper was titled “Off-the-Record Communication, or, Why Not To Use PGP.” Its explicit target was PGP’s own weakness for conversation: a PGP-signed message is cryptographic proof, forever, that its sender wrote it. That is exactly wrong for a private chat. The first reference implementation shipped the same year.

About a year later, Mario Di Raimondo, Rosario Gennaro, and Hugo Krawczyk showed that version 1’s key exchange fell to an identity-misbinding attack. An attacker could leave Alice sure she was talking to Bob, while Bob believed he was talking to the attacker. Version 2, released in 2005, replaced that key exchange with the SIGMA variant this guide implements.

A 2007 paper by Chris Alexander and Ian Goldberg then tackled usability. Comparing key fingerprints was hard for ordinary users, so they proposed the Socialist Millionaires’ Protocol as a way to authenticate with a shared secret instead. It builds on earlier work by Boudot, Schoenmakers, and Traore. Version 3 of the protocol, published in 2012 and the version this guide follows, added instance tags. A client logged in from several places at once no longer confuses its own sessions, and an extra derived symmetric key lets applications use it for things like file transfers.

The Design in Brief

A conversation goes through two phases.

1. The Authenticated Key Exchange (AKE). Bob and Alice run an unauthenticated Diffie-Hellman exchange, then each proves inside the resulting encrypted channel that they hold the private key matching their long-term DSA public key. Four messages: D-H Commit, D-H Key, Reveal Signature, and Signature.
2. Data Messages. Every message after that carries a fresh Diffie-Hellman public key alongside its ciphertext. Once both sides have used a key, they retire it, derive the next one, and reveal the MAC keys the retired one produced.

Optionally, at any point after the AKE, either side can start the Socialist Millionaires’ Protocol to confirm, out of band, that they are really talking to who they think they are, without a certificate authority anywhere in sight.

The Authenticated Key Exchange

The AKE is a variant of the SIGMA protocol: an unauthenticated Diffie-Hellman exchange first, authentication inside the channel it opens second. Bob commits to his ephemeral key before Alice reveals her own. The session id that users compare is only 64 bits, short enough to brute-force, so neither side may choose its key after seeing the other’s:

Bob:    picks r (128 bits) and x, sends AES_r(g^x), SHA256(g^x)
Alice:  picks y, sends g^y
Bob:    computes s = (g^y)^x, sends r, then a signature wrapped in AES and HMAC
Alice:  verifies, then sends her own signature wrapped the same way

The D-H Commit message encrypts g^x (as a minimum-length MPI) under AES128-CTR with the random key r, and separately sends SHA256(g^x). Only once Alice has committed to g^y does Bob reveal r, so Alice can decrypt g^x and check it against the hash she already received. The hash is the commitment; the AES encryption only exists because many IM networks cap message size, which leaves no room to send all of g^x later. The commitment stops a man in the middle from trying many key choices until the session ids on both sides happen to match.

The Reveal Signature message carries Bob’s DSA signature over M_B = SHA256-HMAC(gx, gy, pub_B, keyid_B), itself encrypted and MAC’d with keys derived from the shared secret s. My code implements the full message field by field: protocol version, message type, instance tags, the revealed key, the encrypted signature, and its MAC. DSA itself is checked against 15 of NIST’s own FIPS 186-2 known-answer vectors, both signing and verifying, including vectors with a deliberately wrong signature that must be rejected.

Key Derivation and the Session ID

Both sides compute the shared Diffie-Hellman secret s, encode it as a minimum-length MPI, and run it through SHA-256 with a leading distinguishing byte:

h2(b) = SHA256(byte(b) || mpi(s))
ssid = h2(0x00)[:8]
c, c' = h2(0x01)[:16], h2(0x01)[16:32]
m1, m2, m1', m2' = h2(0x02), h2(0x03), h2(0x04), h2(0x05)

c and the two HMAC keys m1, m2 protect Bob’s Reveal Signature message; the primed versions protect Alice’s Signature message. ssid, the secure session id, is the one value meant to be read aloud. The spec displays it as two 32-bit halves, with a different half in bold for each side. Alice and Bob each read their bold half over a channel they both trust, such as a phone call where they recognize each other’s voices. An active man in the middle has no way to make both sides compute the same ssid. Doing so would require actually relaying the real D-H values, which would let the signatures underneath catch the attack.

Data Messages and the Diffie-Hellman Ratchet

Once the AKE finishes, every message carries its own Diffie-Hellman ratchet. Each side tracks its own two most recent key pairs and the other side’s two most recent public keys. A Data Message always includes the sender’s next key, one step ahead of the one actually used to encrypt the current message, so the other side can start preparing the next shared secret immediately:

sender keyid, recipient keyid, next D-H public key (MPI), counter top half,
AES128-CTR(message), SHA1-HMAC(everything above), old MAC keys to reveal

The encryption and MAC keys for a given pair of D-H keys come from h1(b) = SHA1(byte(b) || mpi(s)), with b chosen as 0x01 or 0x02 depending on whether the sender’s D-H public value is numerically greater than the recipient’s. Whichever side has the larger key is the “high” end and uses h1(0x01) to send; the other side uses it to receive. Both my Python and JavaScript implementations compute this consistently, which I check directly: the two sides must always agree on which of the two derived keys is whose.

The deniability payoff is in what happens to a MAC key once its D-H key pair is retired. The spec calls this out explicitly: a receiving MAC key, once it will never again verify a new message, gets attached in the clear to the next Data Message. From that point on, anyone, not just the two original parties, can forge a message that would pass that MAC check. The key is revealed only after its messages are already delivered, so this never lets an attacker forge something in real time. It does mean a captured transcript, MAC keys and all, is no longer cryptographic proof that either party actually sent anything in it.

The Socialist Millionaires’ Protocol

Two millionaires want to know who is richer without either revealing their net worth. OTR reuses the same idea to let Alice and Bob check that they hold the same secret, usually a shared password or each other’s key fingerprint, without exposing anything else if the secrets differ. The protocol runs over four TLV-carried messages and a set of Schnorr-style zero-knowledge proofs, all in the same Diffie-Hellman group as the AKE:

Alice: picks a2, a3; sends g2^a = g1^a2, g3^a = g1^a3, with proofs of knowledge
Bob:   picks b2, b3; computes g2, g3; sends Pb, Qb (hiding y) with a joint proof
Alice: computes Pa, Qa (hiding x) and Ra = (Qa/Qb)^a3, each with its own proof
Bob:   computes Rb = (Qa/Qb)^b3; both sides check Pa/Pb against Rab

The secret each side feeds in is not typed in directly: it is SHA256(version || initiator_fingerprint || responder_fingerprint || ssid || user_secret). This ties a correct match to the specific conversation and the specific pair of long-term keys, not just a shared word. My implementation runs every one of these zero-knowledge checks exactly as the spec defines them. I verified it two ways: with matching secrets, where both sides correctly conclude they match, and with different secrets, where both sides correctly conclude they do not. A forged proof in the first message is rejected before the exchange can even continue.

SMP is implemented and tested in Python only. The interactive visualizer above covers the AKE and one Data Message; running the full SMP exchange there would need ten more input fields for comparatively little additional teaching value over the worked numbers below.

Interactive Visualizer

The visualizer above runs a full AKE followed by one Data Message: D-H Commit, D-H Key, Reveal Signature, Signature, the shared session id, and one encrypted, authenticated message from Bob to Alice.

The fields are a seed for the exponents and keys, display names for Bob and Alice, the message to send, and a damage option.

• No damage. The exchange completes, and the final line shows the message Alice read back.
• Bad Reveal Signature MAC. A byte inside Bob’s Reveal Signature MAC is flipped. Alice’s independently computed MAC does not match, and she never even attempts to decrypt the signature inside.
• Bad DSA signature. Bob’s signature is corrupted before it is encrypted, so the MAC still verifies correctly, but Alice’s DSA check on the decrypted signature fails.
• Tampered Data Message. The last byte of the encrypted message is flipped. The HMAC-SHA1 tag no longer matches.
• The wrong direction’s keys. The message is sealed with the keys meant for the other direction. Alice, expecting the correct pair, rejects it.

The default seed and names match the worked example below, so the log’s first run reproduces the same bytes this page prints. The page has its own SHA-1, SHA-256, HMAC, AES128-CTR, DSA, and the 1536-bit Diffie-Hellman group.

A Worked Example

With the seed page-default, Bob and Alice get DSA keys under NIST’s own 1024-bit domain parameters. That size is realistic, since libotr itself generates 1024-bit DSA keys. Their fingerprints are 9568aa5d91573a8062e6bbfb3c09e468c526416d and 506dbc22fcb5db4875f43740d096a3518921256a.

Bob’s D-H Commit message is 247 bytes; his ephemeral public key starts with 3aa79acbb7f9cabe74a41f1269b23be5. Alice’s D-H Key message is 207 bytes, with her public key starting fecdf76b2aafae634023d1019136af94. Both sides compute the same shared secret and derive:

ssid = 871e4d5f79d6ef40
c    = d1a21df397d8ba6922a123c7a66b8468

The Reveal Signature message, carrying Bob’s DSA signature, is 521 bytes; Alice’s Signature message is 501 bytes. The Data Message carrying “hello, this is off the record” is 281 bytes, starting with 00030300000100000001010000000001.

A Socialist Millionaires’ run

Using those same fingerprints and the ssid above, Alice and Bob each feed in the same password and derive the SMP secret x, starting with 4780615d857fa704. Alice’s first message includes g2^a, starting with 75951e8cd9109983. After all four SMP messages, both sides conclude the secrets match: True. Run the same protocol with one side typing a different password, and both sides instead conclude: False, with nothing else revealed either way.

Python Implementation

The first file is DSA (FIPS 186), OTR’s only long-term authentication key type.

# otr_dsa.py
# DSA (FIPS 186), OTR's only long-term authentication key type. OTR signs a 32-byte SHA256-HMAC value "taken
# modulo q instead of being truncated (as described in FIPS-186), and not hashed again" (protocol spec,
# Reveal Signature Message): the hash-to-integer step below is that reduction, given h as an integer already.
# Teaching code. It is not constant time.


def sign(h, x, k, p, q, g):
    r = pow(g, k, p) % q
    s = (pow(k, -1, q) * (h + x * r)) % q
    return r, s


def verify(h, r, s, p, q, g, y):
    if not (0 < r < q and 0 < s < q):
        return False
    w = pow(s, -1, q)
    u1 = (h * w) % q
    u2 = (r * w) % q
    v = (pow(g, u1, p) * pow(y, u2, p) % p) % q
    return v == r


def public_key(x, p, q, g):
    return pow(g, x, p)

The second file is the Authenticated Key Exchange: D-H Commit, D-H Key, Reveal Signature, and Signature messages, and the h2-based key derivation.

# otr_ake.py
# The OTR version 3 Authenticated Key Exchange: a SIGMA-variant that runs an unauthenticated Diffie-Hellman
# exchange first, then authenticates both sides inside the channel it opens, with D-H Commit, D-H Key,
# Reveal Signature, and Signature messages (protocol spec, "Authenticated Key Exchange (AKE)" and
# "D-H Commit Message" through "Signature Message").
import hashlib
import hmac

import otr_aes
import otr_dsa as dsa
from otr_types import (mpi, otr_data, pubkey_encode, read_pubkey, read_mpi, read_data, read_int,
                        sig_encode, read_sig, u_short, u_byte, u_int)

VERSION = 3
MSG_DH_COMMIT, MSG_DH_KEY, MSG_REVEAL_SIGNATURE, MSG_SIGNATURE = 0x02, 0x0a, 0x11, 0x12
ZERO_COUNTER = bytes(16)


def h2(b, s):
    return hashlib.sha256(u_byte(b) + mpi(s)).digest()


def ake_keys(s):
    full_c = h2(0x01, s)
    return {
        "ssid": h2(0x00, s)[:8],
        "c": full_c[:16], "cp": full_c[16:32],
        "m1": h2(0x02, s), "m2": h2(0x03, s),
        "m1p": h2(0x04, s), "m2p": h2(0x05, s),
    }


def header(msg_type, sender_tag, receiver_tag):
    return u_short(VERSION) + u_byte(msg_type) + u_int(sender_tag) + u_int(receiver_tag)


def dh_commit_message(sender_tag, receiver_tag, r, gx):
    gxmpi = mpi(gx)
    enc_gx = otr_aes.ctr_crypt(r, ZERO_COUNTER, gxmpi)
    hashed_gx = hashlib.sha256(gxmpi).digest()
    return header(MSG_DH_COMMIT, sender_tag, receiver_tag) + otr_data(enc_gx) + otr_data(hashed_gx)


def parse_dh_commit(data):
    pos = 11
    enc_gx, pos = read_data(data, pos)
    hashed_gx, pos = read_data(data, pos)
    return enc_gx, hashed_gx


def dh_key_message(sender_tag, receiver_tag, gy):
    return header(MSG_DH_KEY, sender_tag, receiver_tag) + mpi(gy)


def parse_dh_key(data):
    gy, _ = read_mpi(data, 11)
    return gy


def compute_m(gx, gy, p, q, g, y, keyid, key):
    data = mpi(gx) + mpi(gy) + pubkey_encode(p, q, g, y) + u_int(keyid)
    return hmac.new(key, data, hashlib.sha256).digest()


def build_x(p, q, g, y, keyid, m, x_priv, k_nonce, corrupt_signature=False):
    r_sig, s_sig = dsa.sign(int.from_bytes(m, "big") % q, x_priv, k_nonce, p, q, g)
    if corrupt_signature:
        # for demonstrations only: flips the signature before it is encrypted and MACed, so the MAC still
        # verifies (it only certifies that this is what Bob sent) but the DSA signature inside does not
        r_sig = (r_sig + 1) % q
    return pubkey_encode(p, q, g, y) + u_int(keyid) + sig_encode(r_sig, s_sig, (q.bit_length() + 7) // 8)


def reveal_signature_message(sender_tag, receiver_tag, r, c, m1, m2, keyid_b, gx, gy, p, q, g, y_b, x_priv_b,
                              k_nonce, corrupt_signature=False):
    mb = compute_m(gx, gy, p, q, g, y_b, keyid_b, m1)
    xb = build_x(p, q, g, y_b, keyid_b, mb, x_priv_b, k_nonce, corrupt_signature)
    enc_sig = otr_aes.ctr_crypt(c, ZERO_COUNTER, xb)
    enc_sig_field = otr_data(enc_sig)
    mac = hmac.new(m2, enc_sig_field, hashlib.sha256).digest()[:20]
    return header(MSG_REVEAL_SIGNATURE, sender_tag, receiver_tag) + otr_data(r) + enc_sig_field + mac


def parse_reveal_signature(data):
    pos = 11
    r, pos = read_data(data, pos)
    enc_sig_start = pos
    enc_sig, pos = read_data(data, pos)
    enc_sig_field = data[enc_sig_start:pos]
    mac = data[pos:pos + 20]
    return r, enc_sig, enc_sig_field, mac


def signature_message(sender_tag, receiver_tag, cp, m1p, m2p, keyid_a, gx, gy, p, q, g, y_a, x_priv_a, k_nonce):
    ma = compute_m(gy, gx, p, q, g, y_a, keyid_a, m1p)
    xa = build_x(p, q, g, y_a, keyid_a, ma, x_priv_a, k_nonce)
    enc_sig = otr_aes.ctr_crypt(cp, ZERO_COUNTER, xa)
    enc_sig_field = otr_data(enc_sig)
    mac = hmac.new(m2p, enc_sig_field, hashlib.sha256).digest()[:20]
    return header(MSG_SIGNATURE, sender_tag, receiver_tag) + enc_sig_field + mac


def parse_signature(data):
    pos = 11
    enc_sig_start = pos
    enc_sig, pos = read_data(data, pos)
    enc_sig_field = data[enc_sig_start:pos]
    mac = data[pos:pos + 20]
    return enc_sig, enc_sig_field, mac


def decrypt_x(enc_sig, key):
    plain = otr_aes.ctr_crypt(key, ZERO_COUNTER, enc_sig)
    pos = 0
    (p, q, g, y), pos = read_pubkey(plain, pos)
    keyid, pos = read_int(plain, pos)
    qlen = (q.bit_length() + 7) // 8
    (r_sig, s_sig), pos = read_sig(plain, pos, qlen)
    return (p, q, g, y), keyid, (r_sig, s_sig)

The third file is Data Messages: the per-exchange Diffie-Hellman ratchet, the h1-based key derivation, and old-MAC-key revelation.

# otr_data.py
# OTR Data Messages: the per-message Diffie-Hellman ratchet (protocol spec, "Exchanging data" and
# "Computing AES keys, MAC keys, and the secure session id"). Every message is encrypted and authenticated
# with keys derived from whichever pair of D-H public keys the two sides most recently exchanged, and old
# MAC keys are revealed once their D-H keys are retired, so a past transcript can always be forged afterward.
import hashlib
import hmac

import otr_aes
from otr_types import mpi, otr_data, u_byte, u_int, u_short, read_int, read_byte, read_short, read_mpi, read_data

MSG_DATA = 0x03
FLAG_NONE = 0x00


def h1(b, secbytes):
    return hashlib.sha1(u_byte(b) + secbytes).digest()


def derive_data_keys(s):
    secbytes = mpi(s)
    key1 = h1(0x01, secbytes)[:16]
    key2 = h1(0x02, secbytes)[:16]
    return {
        1: {"aes": key1, "mac": hashlib.sha1(key1).digest()},
        2: {"aes": key2, "mac": hashlib.sha1(key2).digest()},
    }


def direction_bytes(my_dh_pub, their_dh_pub):
    # "If Alice's public key is numerically greater than Bob's public key, then she is the high end."
    if my_dh_pub > their_dh_pub:
        return 1, 2   # high end: sendbyte 0x01, recvbyte 0x02
    return 2, 1        # low end: sendbyte 0x02, recvbyte 0x01


def build_data_message(sender_tag, receiver_tag, sender_keyid, recipient_keyid, next_dh_pub, counter_top,
                        sending_aes, sending_mac, message, old_mac_keys=b""):
    counter = counter_top + bytes(8)
    enc = otr_aes.ctr_crypt(sending_aes, counter, message)
    body = (u_short(3) + u_byte(MSG_DATA) + u_int(sender_tag) + u_int(receiver_tag) + u_byte(FLAG_NONE) +
            u_int(sender_keyid) + u_int(recipient_keyid) + mpi(next_dh_pub) + counter_top + otr_data(enc))
    tag = hmac.new(sending_mac, body, hashlib.sha1).digest()
    return body + tag + otr_data(old_mac_keys)


def parse_data_message(data):
    pos = 0
    _, pos = read_short(data, pos)
    _, pos = read_byte(data, pos)
    sender_tag, pos = read_int(data, pos)
    receiver_tag, pos = read_int(data, pos)
    _, pos = read_byte(data, pos)
    sender_keyid, pos = read_int(data, pos)
    recipient_keyid, pos = read_int(data, pos)
    next_dh_pub, pos = read_mpi(data, pos)
    counter_top = data[pos:pos + 8]
    pos += 8
    enc, pos = read_data(data, pos)
    body_before_mac = data[:pos]
    tag = data[pos:pos + 20]
    pos += 20
    old_mac_keys, pos = read_data(data, pos)
    return {
        "sender_tag": sender_tag, "receiver_tag": receiver_tag, "sender_keyid": sender_keyid,
        "recipient_keyid": recipient_keyid, "next_dh_pub": next_dh_pub, "counter_top": counter_top,
        "enc": enc, "body_before_mac": body_before_mac, "tag": tag, "old_mac_keys": old_mac_keys,
    }


def open_data_message(parsed, receiving_aes, receiving_mac):
    expected = hmac.new(receiving_mac, parsed["body_before_mac"], hashlib.sha1).digest()
    if not hmac.compare_digest(parsed["tag"], expected):
        raise ValueError("MAC verification failed")
    counter = parsed["counter_top"] + bytes(8)
    return otr_aes.ctr_crypt(receiving_aes, counter, parsed["enc"])

The fourth file is the Socialist Millionaires’ Protocol.

# otr_smp.py
# The Socialist Millionaires' Protocol (protocol spec, "Socialist Millionaires' Protocol (SMP)" and its
# detailed message-by-message description): two parties compare secrets x and y without revealing anything
# beyond whether x == y, using Schnorr-style zero-knowledge proofs of knowledge of a discrete log, and two
# proofs that a pair of values (P, Q) was built honestly from the same hidden exponent.
import hashlib

from otr_dh import GROUP_P as P, GROUP_Q as Q, GROUP_G as G1
from otr_types import mpi, u_byte


def smp_hash(version, *values):
    data = u_byte(version)
    for v in values:
        data += mpi(v)
    return int.from_bytes(hashlib.sha256(data).digest(), "big")


def secret_mpi(initiator_fingerprint, responder_fingerprint, ssid, user_secret):
    data = u_byte(1) + initiator_fingerprint + responder_fingerprint + ssid + user_secret.encode("utf-8")
    return int.from_bytes(hashlib.sha256(data).digest(), "big")


def prove_knowledge(version, exponent, r):
    c = smp_hash(version, pow(G1, r, P))
    d = (r - exponent * c) % Q
    return c, d


def check_knowledge(version, public_value, c, d):
    check = (pow(G1, d, P) * pow(public_value, c, P)) % P
    return c == smp_hash(version, check)


def prove_pq(version, g2, g3, r4, g_exp_secret, r5, r6):
    pb = pow(g3, r4, P)
    qb = (pow(G1, r4, P) * pow(g2, g_exp_secret, P)) % P
    cp = smp_hash(version, pow(g3, r5, P), (pow(G1, r5, P) * pow(g2, r6, P)) % P)
    d5 = (r5 - r4 * cp) % Q
    d6 = (r6 - g_exp_secret * cp) % Q
    return pb, qb, cp, d5, d6


def check_pq(version, g2, g3, p_val, q_val, cp, d5, d6):
    lhs1 = (pow(g3, d5, P) * pow(p_val, cp, P)) % P
    lhs2 = (pow(G1, d5, P) * pow(g2, d6, P) * pow(q_val, cp, P)) % P
    return cp == smp_hash(version, lhs1, lhs2)


def smp1(x, a2, a3, r2, r3):
    g2a = pow(G1, a2, P)
    g3a = pow(G1, a3, P)
    c2, d2 = prove_knowledge(1, a2, r2)
    c3, d3 = prove_knowledge(2, a3, r3)
    return {"g2a": g2a, "c2": c2, "d2": d2, "g3a": g3a, "c3": c3, "d3": d3}


def smp2(msg1, y, b2, b3, r2, r3, r4, r5, r6):
    assert check_knowledge(1, msg1["g2a"], msg1["c2"], msg1["d2"])
    assert check_knowledge(2, msg1["g3a"], msg1["c3"], msg1["d3"])
    g2b = pow(G1, b2, P)
    g3b = pow(G1, b3, P)
    c2, d2 = prove_knowledge(3, b2, r2)
    c3, d3 = prove_knowledge(4, b3, r3)
    g2 = pow(msg1["g2a"], b2, P)
    g3 = pow(msg1["g3a"], b3, P)
    pb, qb, cp, d5, d6 = prove_pq(5, g2, g3, r4, y, r5, r6)
    return ({"g2b": g2b, "c2": c2, "d2": d2, "g3b": g3b, "c3": c3, "d3": d3, "pb": pb, "qb": qb, "cp": cp,
             "d5": d5, "d6": d6},
            {"g2": g2, "g3": g3, "g3a": msg1["g3a"], "b3": b3, "pb": pb, "qb": qb})


def smp3(msg2, x, a2, a3, r4, r5, r6, r7):
    assert check_knowledge(3, msg2["g2b"], msg2["c2"], msg2["d2"])
    assert check_knowledge(4, msg2["g3b"], msg2["c3"], msg2["d3"])
    g2 = pow(msg2["g2b"], a2, P)
    g3 = pow(msg2["g3b"], a3, P)
    assert check_pq(5, g2, g3, msg2["pb"], msg2["qb"], msg2["cp"], msg2["d5"], msg2["d6"])
    pa, qa, cp, d5, d6 = prove_pq(6, g2, g3, r4, x, r5, r6)
    qa_over_qb = (qa * pow(msg2["qb"], -1, P)) % P
    ra = pow(qa_over_qb, a3, P)
    cr = smp_hash(7, pow(G1, r7, P), pow(qa_over_qb, r7, P))
    d7 = (r7 - a3 * cr) % Q
    return ({"pa": pa, "qa": qa, "cp": cp, "d5": d5, "d6": d6, "ra": ra, "cr": cr, "d7": d7},
            {"g3b": msg2["g3b"], "pa_over_pb": (pa * pow(msg2["pb"], -1, P)) % P, "qa_over_qb": qa_over_qb,
             "a3": a3})


def smp4(msg3, state2, b3, r7):
    g2, g3 = state2["g2"], state2["g3"]
    assert check_pq(6, g2, g3, msg3["pa"], msg3["qa"], msg3["cp"], msg3["d5"], msg3["d6"])
    qa_over_qb = (msg3["qa"] * pow(state2["qb"], -1, P)) % P
    lhs = pow(G1, msg3["d7"], P) * pow(state2["g3a"], msg3["cr"], P) % P
    rhs = pow(qa_over_qb, msg3["d7"], P) * pow(msg3["ra"], msg3["cr"], P) % P
    assert msg3["cr"] == smp_hash(7, lhs, rhs)
    rb = pow(qa_over_qb, b3, P)
    cr = smp_hash(8, pow(G1, r7, P), pow(qa_over_qb, r7, P))
    d7 = (r7 - b3 * cr) % Q
    rab = pow(msg3["ra"], b3, P)
    pa_over_pb = (msg3["pa"] * pow(state2["pb"], -1, P)) % P
    bob_result = rab == pa_over_pb
    return {"rb": rb, "cr": cr, "d7": d7}, bob_result


def smp_alice_final(msg4, state3):
    qa_over_qb = state3["qa_over_qb"]
    lhs = pow(G1, msg4["d7"], P) * pow(state3["g3b"], msg4["cr"], P) % P
    rhs = pow(qa_over_qb, msg4["d7"], P) * pow(msg4["rb"], msg4["cr"], P) % P
    assert msg4["cr"] == smp_hash(8, lhs, rhs)
    rab = pow(msg4["rb"], state3["a3"], P)
    return rab == state3["pa_over_pb"]

The DSA test file runs every check that follows.

15 NIST FIPS 186-2 DSA SigGen vectors match: signing with the given k reproduces the published r and s, the public key matches, and the signature verifies
15 NIST FIPS 186-2 DSA SigVer vectors agree, including the deliberately tampered ones that must fail
OTR DSA TESTS PASSED

I checked the code in layers.

• DSA against NIST. My implementation reproduces all 15 of NIST’s own FIPS 186-2 SigGen known-answer vectors, where signing with the given k must reproduce the published r and s exactly. It also reproduces all 15 SigVer vectors, including the ones with a deliberately broken signature that must fail.
• AES128-CTR. Built on the same generic AES core the Kerberos guide verified for both 128- and 256-bit keys, and its CTR-mode keystream matches Node’s crypto module on 100 random cases.
• The 1536-bit group. The prime matches the hexadecimal value printed directly in the protocol spec, which is also RFC 3526’s own 1536-bit MODP group.
• The AKE. A full, from-scratch simulated exchange between Bob and Alice succeeds, with both sides deriving the same ssid. A damaged Reveal Signature MAC or a forged DSA signature is rejected at the right step, across 11 independent runs with fresh random keys each time.
• Data Messages. The high and low ends always agree on which derived key is whose, and consecutive messages with increasing counters all decrypt correctly. A changed ciphertext byte or the wrong direction’s key is rejected, and old MAC keys travel correctly in the next message.
• SMP. Matching secrets are always confirmed equal and different secrets are always confirmed unequal, across random runs, and a forged zero-knowledge proof in the first message is rejected before the protocol can continue.

The page code has its own SHA-1, SHA-256, HMAC, AES128-CTR, and DSA (checked directly against one of the same NIST vectors in JavaScript). Its handshake reproduces the same ssid, the same signatures, and the same Data Message as the Python code. Across 7 CIPHER-level scenarios, including a bad MAC, a bad signature, a tampered Data Message, and the wrong direction’s keys, the page’s log matches the Python reference message for message, and all 7 of them agree on the final outcome. I ran the page in a simulated browser.

What I did not check against an official source is the handshake as a whole, since OTR’s own specification gives no end-to-end test vectors for any of its messages, only the formulas. That part rests on the two independent implementations, Python and JavaScript, agreeing with each other byte for byte, and on the DSA primitive underneath both of them matching NIST’s vectors directly. I did not run an interoperability test against a real OTR client such as Pidgin’s plugin or libotr.

Where It Stands

• A forerunner of the Signal Protocol’s double ratchet. OTR’s design, authenticate once and then keep ratcheting Diffie-Hellman keys for forward secrecy, directly influenced the Signal Protocol. The Signal guide shows how Signal adds a symmetric-key ratchet, so every single message gets a fresh key.
• Largely superseded in its own ecosystem. OTR saw its widest real-world use within XMPP, through the Pidgin and Adium clients. OMEMO, itself built on Signal’s Double Ratchet, has become the recommended choice there, because it supports multiple devices and offline delivery, both of which OTR’s design assumes away.
• A different trust model from TLS. The TLS guide and most of this site’s other protocols authenticate with certificates issued by a trusted third party. OTR instead authenticates with bare long-term public keys that the two parties must verify for themselves, commonly with SMP.

Limitations

This is a teaching implementation of the core protocol, not a production OTR client:

• Not constant time. The Python code uses variable-size integers throughout, for both the Diffie-Hellman and the DSA arithmetic.
• One AKE round, one Data Message. No key rotation beyond the single ratchet step shown, no fragmentation of oversized messages, and no handling of out-of-order or retransmitted messages.
• SMP is Python-only. The interactive visualizer demonstrates the AKE and Data Messages; the Socialist Millionaires’ Protocol is implemented and fully tested, but not wired into the page.
• No instance tag negotiation. Fixed sender and receiver instance tags stand in for the real multi-session matching logic version 3 introduced.
• No query messages or whitespace tagging. The code starts directly from an already-agreed-upon AKE, skipping the plaintext negotiation that precedes it in a real conversation.

Security Status

• The 2005 key exchange fix. Version 1’s key exchange was open to identity misbinding, as Di Raimondo, Gennaro, and Krawczyk showed. Version 2 moved to the SIGMA variant that this guide implements, not the original design.
• Old key sizes. NIST rates 1024-bit DSA at about 80 bits of security, and has disallowed it for new signatures since 2013. RFC 8247 downgraded the 1536-bit Diffie-Hellman group to SHOULD NOT for IKEv2, since a nation-state attacker may break it within a few years.
• Deniability comes from the design, not from DSA. DSA on its own is not a signature scheme designed with deniability in mind. OTR’s deniability instead comes from authenticating the channel setup rather than individual messages, plus the deliberate later disclosure of MAC keys.
• OTR requires synchronous delivery. The Diffie-Hellman ratchet needs both sides online to exchange fresh keys, so OTR cannot deliver a message to someone who is offline. It also does not sync a conversation across a user’s other logged-in devices. Later protocols were designed to close both gaps.
• A successor was drafted. The OTRv4 specification, whose public repository was last changed in 2022, aims to replace OTR’s DSA-based AKE with a more modern design. It also aims to close the offline-delivery and multi-device gaps that pushed much of OTR’s former user base toward OMEMO and Signal instead.

FAQ

Why does OTR commit to gx before revealing it?

The session id that people compare by voice is only 64 bits. Without the commitment, a man in the middle could keep trying new key values until the session ids it shares with Alice and with Bob happen to match. Committing to a hash of g^x first fixes Bob’s value before he sees Alice’s, so neither key can be chosen to steer the result.

Can two people run OTR if they have never met?

Yes, but without extra steps they cannot be sure there is no active man in the middle. Comparing the session id over a trusted channel, or running SMP with a secret only the real parties would know, such as the answer to a personal question, closes that gap.

What happens if a Data Message arrives out of order?

My implementation does not handle this case. A real client tracks the last counter seen for each pair of D-H keys, and rejects a message whose counter is not strictly larger. The counter also forms part of the AES-CTR nonce, so reusing it would be a serious security flaw.

Does revealing old MAC keys let someone decrypt old messages?

No, the revealed values are MAC keys, not encryption keys. They let someone forge a new message that would pass the MAC check for that now-retired key pair. The AES keys used to encrypt the original messages are never revealed.

Is OTR still used today?

Less than it once was. Signal’s Double Ratchet and, within XMPP, OMEMO have replaced it in most actively maintained clients, mainly because OTR cannot handle offline delivery or multiple devices per account, limitations this guide’s Security Status section covers.

References

1. Borisov, Nikita, Ian Goldberg, and Eric Brewer. “Off-the-Record Communication, or, Why Not To Use PGP,” Proceedings of the 2004 ACM Workshop on Privacy in the Electronic Society (WPES 2004). Available at: https://otr.cypherpunks.ca/otr-wpes.pdf

2. Di Raimondo, Mario, Rosario Gennaro, and Hugo Krawczyk. “Secure Off-the-Record Messaging,” Proceedings of the 2005 ACM Workshop on Privacy in the Electronic Society (WPES 2005), pages 81-89. Available at: https://doi.org/10.1145/1102199.1102216

3. Alexander, Chris, and Ian Goldberg. “Improved User Authentication in Off-The-Record Messaging,” Proceedings of the 2007 ACM Workshop on Privacy in the Electronic Society (WPES 2007), pages 41-47. Available at: https://doi.org/10.1145/1314333.1314340

4. “Off-the-Record Messaging Protocol version 3.” Available at: https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html

5. RFC 3526, “More Modular Exponential (MODP) Diffie-Hellman Groups for Internet Key Exchange (IKE),” by Kivinen and Kojo, May 2003. Available at: https://www.rfc-editor.org/rfc/rfc3526

6. NIST Cryptographic Algorithm Validation Program, “FIPS 186-2 DSA test vectors” (186-2dsatestvectors.zip). Available at: https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/digital-signatures

7. FIPS PUB 186-2, “Digital Signature Standard (DSS),” National Institute of Standards and Technology, January 2000.

8. XEP-0364, “Current Off-the-Record Messaging Usage,” XMPP Standards Foundation. Available at: https://xmpp.org/extensions/attic/xep-0364-0.3.2.html