Direct Answer
An order in an OMS passes through a formal state machine: New → Pending New → Open → Partially Filled → Filled (terminal), or along cancel/reject branches. Each state transition is triggered by an external event, an acknowledgment, a fill execution report, a cancellation confirmation, and each transition must be validated against the set of legal transitions from the current state. Transitions to invalid states indicate a bug, a message sequencing error, or a protocol violation.
Fills are the events that drive both order state changes and position updates. When a fill arrives, the OMS must perform three operations atomically: record the fill in the fill ledger, update the order's filled quantity and average price, and update the relevant account's position record. Atomicity is not optional, any failure partway through this sequence leaves the OMS in an inconsistent state that is difficult to detect and hard to recover from without manual intervention.
Key Takeaways
- Formal state machines prevent invalid states: Implementing order lifecycle as an explicit state machine with defined valid transitions makes bugs visible, an attempt to move to an invalid state throws an error rather than silently corrupting data.
- FIX ExecType drives transitions: The FIX tag 150 (ExecType) on an execution report tells the OMS what happened: '0'=New acknowledgment, '1'/'F'=Trade (fill), '4'=Cancelled, '8'=Rejected, 'D'=Restated. The OMS must map each ExecType to the correct state transition.
- Fill, order update, position update must be one transaction: All three writes must commit together or roll back together. Partial commit creates ghost fills or phantom positions.
- Terminal states are irreversible, except busts: Once an order is Filled or Cancelled, it does not reopen. The only exception is a bust, which retroactively creates a correcting fill record (negative quantity) rather than reopening the original order.
- In-flight fills must be processed even post-cancel: A fill arriving after a cancel acknowledgment represents a real trade. It must be booked and flagged, not discarded.
- Position state requires both quantity and cost basis: A position record that only tracks quantity is insufficient, average cost basis must be maintained to compute realized and unrealized P&L correctly.
- Long and short must be tracked separately: A position can simultaneously have a long and short leg in systems that allow simultaneous long-short in the same account. Netting them prematurely loses the separate cost basis information needed for P&L and risk calculations.
- Intraday vs. overnight positions have different cash implications: T+1 settlement for equities means an intraday round-trip does not consume settlement cash. The OMS must track which positions are settled vs. unsettled to compute buying power correctly.
Core Concepts
The order state machine
The order state machine defines every valid state an order can be in and every valid transition between states. The minimal set of states for an equity order: New (created in OMS, compliance not yet checked), Compliance Hold (awaiting compliance review), Pending New (sent to EMS/venue, awaiting acknowledgment), Open (acknowledged, live at venue), Partially Filled (some quantity filled, remainder still open), Filled (entire quantity executed, terminal), Pending Cancel (cancel sent, awaiting confirmation), Cancelled (confirmed cancelled, terminal), Pending Replace (amendment sent, awaiting confirmation), and Rejected (declined by EMS or venue, terminal).
Valid transitions define what states can follow each state. From Open: valid transitions are to Partially Filled (on a fill), Pending Cancel (on a cancel request), Filled (on a full fill), or Rejected (on a venue rejection). From Partially Filled: valid transitions are to Filled, Pending Cancel, or Partially Filled again (on a further partial). From Filled: no further transitions are valid. It is terminal. Attempting to process a fill for a Filled order indicates either a duplicate fill or an in-flight fill that arrived after cancellation.
The state machine should be implemented as code that explicitly enumerates each valid transition and throws an exception or returns an error for invalid ones. Systems that implement order state as a set of boolean flags, isOpen, isCancelled, hasPartialFill, invariably drift into states like isOpen=true AND isCancelled=true, which is meaningless and unrecoverable without examining the actual event history.
Evidence to retain: every state transition must be logged with the triggering event (the FIX execution report's ExecID and ExecType), the timestamp, and the operator or system that triggered it. This log is the reconstruction foundation for crash recovery and audit.
Fill records and the fill ledger
A fill is a permanent record that a specific quantity of a specific instrument traded at a specific price at a specific time. Every fill must have: a unique fill identifier (the FIX ExecID), the order it belongs to (ClOrdID and internal order ID), the instrument, the side, the fill quantity, the fill price, the venue that executed it, the execution timestamp, and the account or accounts it will be allocated to.
The fill ledger is the immutable log of all fills. Once written, a fill record is never deleted or modified. If a fill is busted (cancelled by the exchange after execution), a correcting fill record is written, a new fill with negative quantity, rather than editing or deleting the original. This append-only design makes the fill ledger a reliable audit trail and enables deterministic state reconstruction from the log.
Fill deduplication is implemented by the ExecID: before writing a fill to the ledger, the OMS checks whether that ExecID already exists. If it does, the fill is a duplicate (from FIX session recovery or EMS retransmission) and is discarded. This check must be atomic with the write to prevent race conditions when fills arrive in parallel from multiple sources.
Fill records feed multiple downstream consumers: position updates, P&L calculations, allocation engines, settlement instructions, and regulatory reporting. Each consumer reads from the fill ledger independently. The ledger is the single source of truth; no consumer modifies it.
Atomic position updates
When a fill is processed, the position update must happen in the same database transaction as the fill record write. In a relational database. This means both INSERT INTO fills and UPDATE positions execute within a single BEGIN...COMMIT block. In an event-sourced system. It means the fill event and the position projection are updated atomically within the same event processing unit.
The mechanics of a long position update: current position has 10,000 AAPL at average cost $190.50 (cost basis: $1,905,000). A buy fill arrives: 5,000 shares at $192.00 (cost: $960,000). New position: 15,000 shares at average cost ($1,905,000 + $960,000) / 15,000 = $191.00. Remaining open quantity for the relevant order is reduced by 5,000.
For a sell against an existing long: position is 15,000 AAPL at $191.00. A sell fill arrives: 3,000 shares at $195.00. Realized gain = (195.00 - 191.00) × 3,000 = $12,000. New position: 12,000 shares still at $191.00 average cost. The sell does not change the average cost of the remaining position; it realizes a gain on the sold shares. This is a fundamental bookkeeping rule that many simplified systems get wrong, they incorrectly update the cost basis on sells.
Short position mechanics are the mirror: cost basis for a short is the proceeds per share at the time of the short sale. When a buy-to-cover fill arrives, the realized gain or loss is the difference between the short sale proceeds and the cover price, multiplied by quantity. Short positions must be tracked separately from long positions, netting a long and a short into a single signed quantity loses the separate cost basis information.
Position state: settlement and buying power
A position in the OMS has both a quantity dimension and a settlement dimension. For U.S. equities settling T+1 (as of May 2024 under the SEC's amended Rule 15c6-1), a buy today creates an obligation to pay by tomorrow. The position record tracks whether the shares have settled (delivery of shares and cash complete) or are still unsettled (shares held by DTC in a pending settlement status).
Buying power calculations depend on the settlement state. Settled long positions contribute their full market value to margin calculations. Unsettled long positions from buys today consume cash that has not yet left the account, they reduce available cash. Unsettled long positions from buys yesterday (T+1 settling today) are settling today and must be flagged as having their cash committed. Systems that treat all long positions identically regardless of settlement state will produce incorrect buying power calculations.
Worked Scenario
An account holds 5,000 shares of META at average cost $510.00. The portfolio manager places a buy order for 3,000 more and a sell order for 2,000. We trace the state machine for each.
- Buy order created: Order B-001 enters state New. Compliance check passes. Transitions to Pending New when sent to EMS.
- Buy acknowledged: EMS sends ExecReport, OrdStatus=0 (New), ExecType=0. B-001 transitions to Open. Position unchanged (order acknowledged, not filled).
- Sell order created: Order S-001 enters New, passes compliance (5,000 long ≥ 2,000 sell; no short-selling), transitions to Pending New, then Open.
- Partial fill on buy: ExecReport arrives: ExecType=1 (Trade), FillQty=1,000, FillPx=$525.00. OMS writes fill F-001. Atomically: B-001 transitions to Partially Filled (filled=1,000, remaining=2,000, avg=$525.00). Position updates: 5,000 + 1,000 = 6,000 shares, new avg cost = (5,000×$510.00 + 1,000×$525.00) / 6,000 = $512.50.
- Sell fills: S-001 fills 2,000 shares at $528.00. OMS writes fill F-002. Atomically: S-001 transitions to Filled. Realized gain = (528.00 - 512.50) × 2,000 = $31,000. Position: 6,000 - 2,000 = 4,000 shares, still at $512.50 avg cost (sell does not change remaining cost basis).
- Buy completes: Remaining 2,000 shares of B-001 fill at $524.50. OMS writes fill F-003. B-001 transitions to Filled. Position: 4,000 + 2,000 = 6,000 shares, new avg = (4,000×$512.50 + 2,000×$524.50) / 6,000 = $516.50.
- Final state: B-001=Filled, S-001=Filled. Position: 6,000 META at avg cost $516.50. Realized gain: $31,000 booked to the account.
Measurement Framework
| Measurement | Question it answers |
|---|---|
| Invalid state transition count | How many times per day does the OMS attempt an illegal state transition? Should be zero in production; any nonzero count indicates a protocol bug or message sequencing issue. |
| Fill deduplication rate | What fraction of fills are discarded as duplicates? A rate above 0.01% indicates FIX session recovery issues sending more messages than expected. |
| Position update latency | Time between fill timestamp on execution report and position write commit in database. Target: <50ms P99 for equities. |
| Average cost accuracy | Does the OMS-reported average cost for a position match the independently computed value from the fill ledger? Run this audit nightly. |
| Unsettled position tracking accuracy | Do unsettled positions reconcile against prime broker's unsettled position report? Mismatches indicate fills not flowing through to settlement correctly. |
| Ghost order rate | Orders in Pending New or Open state for longer than 30 minutes without a fill or cancel confirmation. Any count above zero needs investigation. |
Common Failure Modes
Non-atomic fill processing creating position/fill divergence
The most dangerous failure is writing a fill to the fill ledger but failing to commit the position update, typically because the position update logic throws an exception after the fill INSERT commits. The fill is now in the ledger but the position has not changed. Every subsequent position calculation will undercount the position; reconciliation against the broker will show a break equal to the fill quantity.
The fix is a database transaction wrapping both writes, or an event-sourced design where the fill event drives a reliable projection update. Any exception inside the processing loop must roll back both writes. Recovery from a non-atomic partial write requires identifying all fills in the ledger that have no corresponding position change, a query that is only possible if the position record tracks the last fill ID that updated it.
State machine bypass via direct database writes
Operations teams under time pressure sometimes fix an order in a bad state by writing directly to the order state column in the database, bypassing the state machine. This is catastrophically dangerous: the state machine records the transition, generates the audit log entry, and triggers downstream events (position update, allocation, notification). A direct database write does none of these. The result is an order whose audit trail shows it never transitioned out of its previous state, while the database shows it in a different one, an inconsistency that will fail both audits and automated reconciliation.
Operational fixes must go through the state machine. If the state machine lacks a correction path (e.g., no way to manually close an orphaned order). That is a gap to fix in the next release, not a reason to bypass it today.
In-flight fill after cancel not booked
When a cancel is sent and a fill arrives for the same order after the cancel has been acknowledged by the OMS, some implementations discard the fill, reasoning that the order is cancelled and should not receive fills. This is incorrect. The fill occurred at the venue before the cancel was processed and is a real trade that must be booked. Discarding it creates a short position (if a sell order) or an undercounted long (if a buy) that will not be discovered until reconciliation.
Correct behavior: process any fill for any order regardless of the order's current state, but log an alert when a fill arrives for a terminal-state order. The operations team investigates; the fill itself is booked.
Average cost corruption on mixed buys and sells
When a sell reduces a long position, the remaining shares must retain their original average cost. If the implementation incorrectly averages the sell price into the cost basis, treating the sell like a buy in the opposite direction, the average cost for remaining shares will be wrong. This error accumulates over many trades and produces systematically incorrect P&L calculations. It is a common bug in simplified OMS implementations that model position as a single signed-quantity field rather than separate long/short records.
Missing fill causing reconciliation break without alert
If a fill execution report is dropped, by a FIX session failure, a bug in the EMS's retransmission logic, or a capacity overflow, the OMS will not record it. The position will be understated relative to what the broker holds. Without automated reconciliation against broker positions, this discrepancy may not surface until end-of-day matching, by which time the position has been used for compliance checks, risk limits, and P&L calculations, all incorrect.
Prevention: the OMS must reconcile its open order quantities and positions against the EMS's live state at regular intervals (every 15-30 minutes during market hours), not just at end of day.
Frequently Asked Questions
What are the valid states in an OMS order state machine?
The canonical states are: New (order created in OMS, not yet sent to EMS), Pending New (sent to EMS, awaiting acknowledgment), Open/Working (acknowledged, active at venue), Partially Filled (some quantity executed), Filled (entire quantity executed), Pending Cancel (cancel sent, awaiting acknowledgment), Cancelled (confirmed cancelled with zero or partial fills), Pending Replace (amendment sent, awaiting acknowledgment), and Rejected (declined by venue or EMS). Each state has defined valid transitions, moving from Filled to Cancelled, for example, is invalid.
Why must fill and position updates be atomic?
If a fill is recorded without the corresponding position update, the OMS will show the trade as complete but the position will remain unchanged, creating an immediate discrepancy that will fail reconciliation. If the position is updated without recording the fill, the position appears to change without a traceable cause. Either failure makes the OMS state non-auditable and potentially non-recoverable without manual intervention. Atomicity guarantees that either both succeed together or neither does, so the OMS is always in a consistent state.
What is the difference between a fill and an execution report?
An execution report (FIX message type 8) is the message that conveys any order state change, acknowledgment, partial fill, full fill, cancellation, rejection. A fill specifically refers to an execution report with ExecType = Trade (value '1' in FIX 4.x, 'F' in FIX 5.x), indicating that shares were actually traded. Not all execution reports are fills; many are state-change notifications with no quantity transacted.
How does the OMS handle a fill that arrives after the order was cancelled?
This is called an in-flight fill or a bust-then-fill sequence. When a cancel request is sent to a venue, fills can still occur before the cancel is processed. When the fill arrives at the OMS after a cancellation, the OMS must process it. It is a real trade that occurred. The OMS marks the order as 'Filled (post-cancel)' or a similar terminal state, books the fill against the position, and flags the discrepancy for operations review. Silently discarding the fill would create a phantom position.
What is a position state model?
A position state model defines what a position record contains and how it changes. A minimal position record includes: account, instrument, long quantity, short quantity, average cost basis, and last-updated timestamp. Fills update long quantity (for buys) or short quantity (for sells) and recalculate average cost basis. The model also tracks whether the position is intraday-only (opened and closed same day) or carries overnight, because T+1 and T+2 settlement create distinct cash implications for each.
What is a 'ghost' order in OMS context?
A ghost order is an order that exists in the OMS with an Open or Pending state but has no corresponding live order at the venue or EMS. This typically happens when an order was sent to the EMS, the EMS processed it but the OMS never received the acknowledgment, and the OMS never timed out and cancelled the order. Ghost orders inflate the OMS's view of open quantity and distort the remaining-to-fill metric for parent orders. Regular reconciliation against the EMS order book catches these.
How does OrdStatus in FIX map to OMS order states?
FIX tag 39 (OrdStatus) carries: 0=New (open), 1=Partially Filled, 2=Filled, 4=Cancelled, 8=Rejected, A=Pending New, E=Pending Replace, 6=Pending Cancel. These map directly to the OMS state machine states. The OMS transitions its internal state based on incoming OrdStatus values in execution reports. A mismatch between the OMS's expected state and the received OrdStatus indicates a sequencing problem requiring investigation.
What is average cost basis and how does it update with fills?
Average cost basis is the weighted average price at which the current position was acquired. For a long position: when a buy fill arrives, new_avg_cost = (old_qty × old_avg_cost + fill_qty × fill_price) / (old_qty + fill_qty). When a sell reduces the position, the remaining shares retain the old average cost, the gain or loss is realized on the sold shares at the difference between fill price and average cost. Short positions use the analogous calculation for short sale proceeds.
How should a short position and a long position in the same instrument be represented?
A single signed quantity is simpler and matches how most brokers report, but it loses information where the two are held for different reasons or in different accounts, and it obscures cost basis when a position crosses through zero. Keeping the signed net as the primary figure while retaining the lot history preserves both. The crossing case deserves explicit handling, because a sell that takes a long position past zero closes one position and opens another with a new basis.
References
- FIX Protocol 4.4 Specification: Execution Report: Defines tag 150 (ExecType) values and tag 39 (OrdStatus) state transitions in authoritative detail.
- SEC Rule 15c6-1 Amendment: T+1 Settlement (2023): Establishes the U.S. T+1 settlement cycle for equities effective May 28, 2024.
- DTCC NSCC Clearing Services: The National Securities Clearing Corporation's netting and settlement process that underlies equity position settlement.
- BIS: Principles for Financial Market Infrastructures: Includes guidance on settlement finality and atomic exchange of value relevant to position settlement models.
- FINRA Rule 4330: Customer Protection: Defines required segregation of customer securities and cash, relevant to how settled positions are held and accounted for.
Educational Disclaimer
This guide is for educational purposes only and does not constitute financial, legal, or compliance advice. OMS state machine designs vary by vendor and jurisdiction. Consult qualified technical and legal professionals before implementing order management systems.