Holding a Trading Session Together

Pattern: Concern-Sorted Live State  ·  Source: quickfix-j/quickfixj · SessionState.java (commit 700b798). Every [src] link opens the exact lines on GitHub.

New to FIX? FIX is the messaging protocol that brokerages, banks, and exchanges use to talk to each other across the trading day. A FIX session is a single long-running connection over that protocol, kept open so messages can flow either way at any moment. Each message carries a sequence number; both sides keep a running expected-next, which is how they agree on what’s been said and what’s been missed. That’s enough to follow the rest.

A FIX session is a long-running, stateful conversation between two trading parties. Once it’s open, it’s expected to stay open — through pauses in trading, through brief network blips, through silence when one side has nothing to say. Either side can disconnect any time. But neither side should be in any doubt about what’s been said when the line goes quiet.

To stay open, the session has two distinct jobs running side by side.

Process the conversation. Receive the next message, check it’s in sequence, accept it or ask for a resend, possibly reply. This happens whenever a message arrives — which is unpredictable. Bursts during the trading day. Long quiet stretches overnight.

Keep the connection alive. Periodically check: have I sent anything recently, or do I owe them a heartbeat? Have I heard anything recently, or should I start worrying that the line has dropped? This happens on a wall-clock timer, every few seconds, regardless of whether any messages are flowing.

Those two jobs can’t share a thread. The conversation handler is busy when messages are arriving; the liveness watcher has to fire on schedule even when nothing’s happening. So you get two threads — and both of them need to read and write the same shared bookkeeping.

What does the bookkeeping have to track?

Five distinct kinds of state, each with a different job:

• Where we are in the conversation — the next outbound sequence number, the next inbound expected.
• Vital signs — when did I last send, when did I last receive, what’s the expected heartbeat interval.
• Where we are in the lifecycle — did we both log on, has either side requested a reset, are we mid-logout.
• What we’re recovering from — are we currently re-syncing a gap of missed messages, and which range.
• What we agreed at the start — an extra recovery agreement made during the logon handshake, recording the sequence number from which the conversation should resume after a re-connect [src].

Two threads. Five kinds of state. Different access patterns. The wrong organization here is either too slow to keep up with line-rate messages, or too unsafe to trust with sequence numbers that govern real orders booked into real markets.

So: how do you organize this state?

The natural reaches an experienced designer would consider, roughly in the order they come to mind:

• One big lock around the whole bookkeeping. Simple, correct, slow. Every cross-thread access blocks every other one, including the trivial field reads that have no business waiting. At line rate, this becomes “the message thread can’t keep up.”
• One queue, one thread for everything. Both jobs send work to a single worker via a queue. Clean, common in modern designs. But the liveness watcher needs to fire on a wall-clock schedule even when nothing’s queued, and a single thread can’t both poll its timer and process inbound at line rate. You end up with a watchdog thread anyway — and now it has the same shared-state problem you started with.
• Lock-free everywhere. Atomic compare-and-swap loops, immutable snapshots, copy-on-write. Beautiful in theory. Hard when the operation involves read a value, decide based on it, then maybe update. The compare-and-swap version of “accept this message if it’s the next expected, otherwise enqueue it for resend” is a maintenance nightmare nobody wants in safety-critical code.
• One lock per field, mapped by name. Flexible, but every cross-field operation now needs careful ordering, and you’ve effectively rebuilt a small database engine inside your session.

None of those is what twenty years of production code chose.

The class is real

It lives in QuickFIX/J, the open-source FIX engine that runs at half the sell-side banks on the planet. The class is SessionState [src]. It holds all five kinds of bookkeeping for one trading session. Two threads access it — the session thread for processing inbound and outbound messages, the timer thread for liveness checks.

Get the bookkeeping wrong and you risk a duplicate order, a dropped fill, or a session that quietly disagrees with the counterparty about what’s been said. The kind of bug that doesn’t crash a process. The kind that just quietly costs the firm money until somebody notices, which might be days.

Here’s how production code organized the state.

What twenty years of production hardening did with it

Three different concurrency primitives, deliberately matched to the three access patterns the five kinds of state fall into.

flowchart TB
    ST["session-thread<br/>writes everything"]
    TT["timer-thread<br/>reads vital signs"]
    subgraph SessionState
      direction TB
      R1["Region 1 · intrinsic lock<br/>vital signs, lifecycle, resend window"]
      R2["Region 2 · ReentrantLock<br/>sequence numbers — compound ops"]
      R3["Region 3 · AtomicInteger<br/>start-of-session latch"]
    end
    ST --> R1
    ST --> R2
    ST --> R3
    TT -. reads .-> R1

SessionState — one class, three concern-sorted regions, each guarded by the cheapest primitive that fits its access pattern.

Five kinds of state, but they fall into three access patterns. Three primitives matched to those three patterns. Let’s walk through each match, why it beats the alternative, and what it costs.

Region 1 — intrinsic lock for vital signs and lifecycle

Most of the bookkeeping is the simple-access-pattern case. Vital signs (last-sent, last-received, heartbeat interval). Lifecycle flags (logon, logout, reset). The resend window state. Single-step writes by the session thread; the timer thread reads.

For this access pattern, the textbook answer is the right answer: intrinsic synchronized(lock) — Java’s built-in per-object lock — wrapped around every access [src]. When no two threads are actually fighting for the lock at the same moment (and they almost never are here), modern JVMs make synchronized nearly free. Contention is almost nil because the timer thread reads briefly and the session thread writes between message arrivals.

The alternative that doesn’t fit: You might reach for volatile on each field. It gives you cross-thread visibility on single reads. For a compound read across multiple fields, though, two separate volatile reads can pick up values from two different moments of the session thread’s updates — so if the caller needed both reads as an atomic snapshot, volatile wouldn’t suffice. Intrinsic lock makes the snapshot achievable: a caller that wraps both reads in one synchronized(state.getLock()) block sees both fields from the same moment. In practice QuickFIX/J’s liveness predicates like isHeartBeatNeeded just call the individual getters sequentially — the intrinsic-lock pattern leaves the wrap-for-snapshot option available; the predicates choose not to use it.

public void setLastReceivedTime(long timestamp) {
    synchronized (lock) {
        lastReceivedTime = timestamp;
    }
}

Aside — the one field both threads write: There is exactly one field in this entire class that both threads write to. Just one. It’s called testRequestCounter [src]. The timer thread increments it when sending a TestRequest — a small FIX message meaning “are you still there?”, sent when the other side has gone quiet long enough that you want positive confirmation they haven’t dropped. The session thread clears the counter on every inbound message (“yes, they’re still talking to us”). Find this one field and you understand the entire liveness mechanism of the engine. Someone thought hard enough about this class to leave it as the only exception in the whole concurrency design — deliberately, because moving the counter somewhere else was worse.

Region 2 — ReentrantLock for advancing the conversation

The sequence numbers are the conversation’s pointer. Advancing them is a compound operation: read the expected next, compare against what arrived, increment if (and only if) it matches. The whole compound has to be atomic — if two threads interleave it, you can acknowledge the same message twice, which is the fireable-offence bug in trading software. And it has to span several method calls because the read and the increment are separate operations on SessionState, called from the orchestrator.

Concretely, here’s the actual production code from Session.java where the orchestrator brackets a compound seq-num operation:

state.lockTargetMsgSeqNum();                       // (1) grab the lock
try {
    // QFJ-557: only advance if we are at the expected number.
    if (!MsgType.LOGON.equals(msgType)
            && !MsgType.SEQUENCE_RESET.equals(msgType)
            && msgSeqNum == getExpectedTargetNum()) {  // (2) read & compare
        state.incrNextTargetMsgSeqNum();           // (3) maybe advance
    }
} finally {
    state.unlockTargetMsgSeqNum();                 // (4) release
}

Intrinsic synchronized is the wrong shape for this API. It’s lexically scoped: the lock is acquired at the start of a synchronized(obj) block and released when the block exits. You can’t expose lock acquisition and release as separate methods on SessionState, because the lock’s lifecycle is tied to the lexical block, not to an object reference.

ReentrantLock — a lock object you can hold across method calls — solves the shape problem exactly. The orchestrator locks in one method, calls the inner mutators in between, releases at the end. Caller-managed lock lifecycle.

The alternatives that don’t fit: Lock-free compare-and-swap on the seq-num integer — works for a single increment, but the actual operation is the compound read-compare-conditional-increment shown above. Not a single CAS-able transition. Or hold the intrinsic lock for the whole compound — but the compound spans multiple method calls in the orchestrator, and intrinsic locks can’t bridge those without redesigning the API into one monster method that destroys the orchestrator/state separation.

Which means the methods that touch the sequence numbers — getNextSenderMsgSeqNum, incrNextSenderMsgSeqNum, setNextTargetMsgSeqNum, incrNextTargetMsgSeqNum — do not acquire the locks internally [src]. They expect the caller to have already locked.

A small asymmetry worth noticing: There is a setNextTargetMsgSeqNum but no setNextSenderMsgSeqNum. You can reset the inbound counter through SessionState; you can’t reset the outbound counter through the same path. The asymmetry is almost certainly deliberate (you should never casually rewrite your own outgoing sequence mid-session — that’s how duplicate orders happen). But it’s never explained anywhere in the code, the Javadoc, or the protocol spec. Add it to the list of design choices buried in production code that everyone learns by osmosis or breaks something.

The first time you read this code, you assume that is a bug. It is not.

The cost of this match: discipline at every call site. Forget to lock, forget to unlock, release too early — the compiler won’t warn you. The runtime won’t either, most of the time. Bugs show up as occasional intermittent symptoms in production, when load is high and the wind is blowing from the right direction. This is the part of the codebase a long-tenured maintainer reviews carefully on every PR.

Region 3 — AtomicInteger for the start-of-session agreement

The third pattern is the simplest. One piece of state — the start-of-session recovery agreement — is established once during the logon handshake and then read for the rest of the session [src]. No compound operation. No lock needed. It just needs atomic visibility across threads: when the session thread sets it during logon, the read paths need to see the new value cleanly — never a mix of half an old write and half a new one.

private final AtomicInteger nextExpectedMsgSeqNum = new AtomicInteger(0);

The alternative that almost fits: The natural alternative is volatile int. It works for set-once-read-many. AtomicInteger is the more careful choice here: it gives you the option of compareAndSet if the protocol ever needs it later, and it’s a clearer signal of intent — “mutable but lock-free atomic” says something stronger to the next reader than “volatile, take it from there.” Cost over volatile is negligible. Clarity is real.

A ReentrantLock would be overkill. The intrinsic lock would block readers for no reason. AtomicInteger is the cheapest correct option for the pattern.

Sometimes the right answer is the boring answer.

Where this design came from

The three-primitive pattern we just walked through isn’t a design that someone planned in 2005 and shipped. It accumulated over seven years from four named contributors, each adding their primitive in response to a specific pressure.

• Oren MillerFeb 200511d3c339 — Wrote the original SessionState. It had the method names — clearTestRequestCounter, the sequence-number accessors — but no concurrency primitives. Just fields, setters, getters.
• Steven BateMay 2007bc892f4d — Added the synchronized(lock) wrapping — retrofitting the intrinsic-lock pattern two years after the class was written. Region 1 was established here.
• Steven BateJun 2007acebdd03 — A month later, added the two ReentrantLocks for sender and target sequence numbers. Region 2 — the caller-managed pattern — dates from this commit. The Region 1 / Region 2 asymmetry was set in a single 30-day window in mid-2007.
• C. John / C. HurstNov 2012a1dbaf30 — Five years later, the AtomicInteger for the start-of-session recovery agreement (Region 3) — applied for QuickFIX/J 1.5.0 because the FIX Session Protocol 1.1 Errata introduced tag 789 (NextExpectedMsgSeqNum). The choice of atomic over volatile int was made here, by one author addressing one protocol change.

What we walked through as a unified pattern is, in production reality, four engineers responding to four distinct pressures across seven years. The class structure (2005) preceded the concurrency primitives (2007), which preceded the lock-free latch (2012). That accumulation is itself the lesson: mature production designs rarely spring fully-formed; they accrete through bug fixes, protocol updates, and contributor decisions over years. The pattern is correct — and it’s retrospectively constructed. A story we can tell about what the code settled into, not a story the original authors set out to tell.

What this design preserves

You could have organized the bookkeeping differently. One big lock around everything. One queue funnelling all access to a single thread. A flat array of volatile fields. Any of those would have worked — they would have produced sessions that processed messages correctly under normal load.

They would also have produced sessions that fail in subtle, hard-to-debug ways under load. Lock contention pushes the message thread into unpredictability — messages arrive late, sequence checks lag, the session disagrees with the counterparty about what’s been said. The single-queue approach starves the timer thread, so heartbeats fire late and the counterparty mistakenly disconnects. The flat-volatile approach removes the option of wrapping a compound read in a snapshot block for the rare cases that need one.

In a trading session, performance is a correctness concern one level up. The session’s correctness depends on its timing, not just on its logic. The design pattern preserves that timing by giving each concern access to its state at the lowest possible cost.

There’s a subtler thing this design preserves, too. The pattern makes the shape of each concern visible in the code. synchronized(lock) means: low-contention bookkeeping. lockSenderMsgSeqNum() at a call site means: this caller is mid-way through advancing the conversation, watch the bracketing. nextExpectedMsgSeqNum.set(...) means: this is something agreed at the start, not changing during the conversation. The primitives are inline documentation of the session’s structure. One lock everywhere would have hidden all of that.

Three primitives, used correctly, recover it — for every engineer who reads this code over the next twenty years.

The pattern: Concern-Sorted Live State

When a long-running stateful component has multiple concerns running on different schedules, all touching the same shared state, sort the state by concern first, then give each subset the cheapest mechanism that protects it for its actual access pattern.

You’ll see this pattern recur across trading infrastructure and well beyond it:

The skill is resisting the urge to protect everything the same way. One lock is simpler to write and worse to live with: it hides the structure and serializes concerns that never needed to wait on each other. Sort first; then reach for the cheapest correct guard per pattern. Done right, the primitives themselves document what each piece of state is for.

1. Name the concerns. What jobs is this component actually running?
2. For each concern, identify the access pattern of the state it touches. Single-step? Compound? Set-once?
3. Match each subset with the cheapest mechanism that protects it correctly.

• FIX session bookkeeping sorts into conversation / liveness / lifecycle / recovery / negotiation — three access patterns, three primitives.
• Market-data feed handlers sort theirs into subscription / heartbeat / sequencing, each with its own guard.
• OS schedulers separate the run-queue lock from per-task accounting from the wall-clock tick — different locks for different rhythms.
• React internals split frequently-mutated fibers from set-once config from atomic lane bitmasks.