Stale market data is the failure that looks like success. The WebSocket is connected, the dashboard is painting numbers, the alert engine is quiet, and the price on screen is four minutes old. Nothing threw an exception. This post covers how to detect stale market data in a real-time trading application and what to do once you've detected it: which timestamps to compare, how to set market data freshness thresholds per asset class, how to run a WebSocket heartbeat, how to spot missed updates, and how to build a real-time data reconnect that restores state before it resumes the stream.
Two clocks: the venue timestamp and your receive time#
Every tick has at least two ages. The first is how old the market's own print is: the timestamp the venue or aggregator stamped when the trade or quote happened. The second is how long ago your process received anything at all for that symbol. They fail in different ways, so you compare against both.
A large venue age with a small receive age is the dangerous case: the transport is healthy, frames keep arriving, and every frame carries the same old price. That's a frozen upstream, and it's invisible if you only measure time since the last message. The reverse case, no frames at all, is a transport or subscription problem, and the venue timestamp tells you nothing new about it.
SiftingIO live ticks carry the field t as Unix epoch milliseconds, and the fair-price engine exposes three timestamps per tick (source, ingest, publish) plus a quality flag that reads Normal or Degraded, described on the data methodology page. Whatever feed you use, record the receive time yourself with a monotonic clock, keep the venue time separately, and never overwrite one with the other. If your host clock drifts, venue age goes negative or jumps; treat that as an alert in its own right. Timestamp formats and time zone traps are covered in what timezone is market data in.
A single REST call gives you a baseline for both clocks:
curl -H "X-API-Key: $SIFTING_KEY" \
"https://api.sifting.io/v1/last/quote/forex/EURUSD"
If the live value is older than the freshness threshold on the server side, the API returns 503 with error code stale_snapshot and a body carrying last_t and server_now, so you can log the gap rather than guess at it.
Market data freshness thresholds by asset class#
There's no single number. EURUSD during the London and New York overlap updates many times per second, so thirty seconds of silence means something broke. CORNUSD or XAGUSD can go a full minute without a print in a quiet session and be perfectly healthy. A US stock at 03:00 Eastern has simply stopped trading for the session, and an alert at that hour is noise that trains the on-call engineer to ignore the pager.
Set thresholds per asset class, and ideally per symbol and session, from observed inter-arrival times rather than from a guess. A practical method: record the gap between consecutive ticks per symbol for a week, take a high percentile of that gap during open hours, and multiply by a safety factor. Then bake in what you know about the schedule. Check market status before raising a staleness alarm; SiftingIO exposes this as GET /v1/fnd/markets/{market}/status with slugs such as us_equities and forex, and the same schedule is browsable at market hours. Account for the delivery cadence of the feed too: the fair-price update rate on SiftingIO runs from 1 Hz on the Free tier up to 10 Hz on Ultra, so a 500 ms threshold on a Free key will fire constantly and mean nothing.
The check itself is a loop, separate from the receive path, so a stalled socket can't stall the check:
every CHECK_INTERVAL:
now = monotonic_now_ms()
for symbol in subscriptions:
last = state[symbol]
if last is null:
mark(symbol, NO_DATA); continue
if not market_open(symbol):
mark(symbol, CLOSED); continue
receive_age = now - last.received_at # time since any frame arrived
venue_age = now - last.venue_ts # age of the market's own print
th = thresholds[symbol.asset_class][symbol.session]
if receive_age > th.silence:
mark(symbol, SILENT) # transport or subscription problem
else if venue_age > th.venue:
mark(symbol, STALE_AT_SOURCE) # frames arrive, price does not move
else if last.quality == DEGRADED:
mark(symbol, DEGRADED) # the feed says so itself
else:
mark(symbol, FRESH)
What the application does with each state is a product decision, but a reasonable default is: FRESH renders normally, DEGRADED renders with a visible marker, STALE_AT_SOURCE and SILENT grey out the price and block any downstream logic that treats it as current, and NO_DATA after a subscribe is a bug to page on. Two earlier posts cover the price-level checks that complement these time-level ones: flagging a quote whose spread has blown out in bid ask spread: read a quote and flag a wide or stale one, and catching a venue whose print disagrees with the cross-venue consensus in detecting a stale or manipulated quote with a consensus price.
WebSocket heartbeat and detecting missed updates#
A TCP connection can sit half-open for a long time after the other end has gone. The socket library reports it as connected because nothing has told it otherwise. The heartbeat is what turns that silence into a signal. RFC 6455 defines Ping and Pong control frames for this purpose, and many feeds add an application-level equivalent because browser WebSocket APIs don't expose the protocol-level ping. SiftingIO uses the application form: send { "op": "ping" }, expect { "f": "pong" }. The server also closes any connection that has sent nothing for 90 seconds, so a client must send at least one frame every 60 seconds even when it has nothing to say.
Run the heartbeat on its own timer and measure two things: time since the last ping was answered, and time since any frame at all arrived. A quiet market produces no ticks but should still produce pongs. If pongs stop, close the socket yourself rather than waiting for the operating system to notice.
Missed updates are harder to see than a dead socket. If the protocol carries a sequence number, track it per stream and treat any gap or any backwards step as a resync trigger; FIX session layers do this with MsgSeqNum, and the FIX Trading Community publishes the session rules for gap fill and resend. Many WebSocket market data feeds don't number frames, and SiftingIO tick frames carry a timestamp rather than a sequence. In that case, monitor timestamp monotonicity per symbol (a t that goes backwards is a replay or an out-of-order delivery), and use the fact that a fresh subscribe returns the last cached value first: comparing that value's timestamp against what you last saw tells you whether anything happened while you were away.
Real-time data reconnect: backoff, jitter, and state restore#
Reconnecting is the easy part. Reconnecting without hammering the server, and without resuming on stale state, is where most implementations go wrong.
Use exponential backoff with full jitter: pick a delay uniformly between zero and the capped exponential value. The AWS Architecture Blog analysis shows why the jitter matters. After an outage, every client that lost its connection at the same instant will otherwise retry at the same instant, and the retry storm itself keeps the server down. Cap the delay at something like 30 to 60 seconds so a long outage doesn't turn into a client that takes ten minutes to notice the service is back.
Then restore state before resuming. Between the disconnect and the new subscribe, the market moved and your cache didn't. Pull a REST snapshot for every symbol, seed the cache from it, and only then subscribe. Once ticks flow again, discard any frame whose timestamp is older than the snapshot's as_of. The cached value the server emits on subscribe serves the same purpose for one symbol, but a REST snapshot lets you resync and log the gap in one place, and it works the same whether you lost the socket for two seconds or twenty minutes.
state = DISCONNECTED; attempt = 0; healthy_since = null
loop:
switch state:
DISCONNECTED:
delay = random_between(0, min(CAP, BASE * 2 ** attempt)) # full jitter
sleep(delay); attempt += 1
sock = connect("wss://stream.sifting.io/ws/v1?key=" + KEY)
state = AUTHENTICATING
AUTHENTICATING:
frame = recv(timeout = AUTH_TIMEOUT) # ack carries the tier
state = RESYNCING if frame.f == "ack" else DISCONNECTED
RESYNCING:
for symbol in subscriptions:
snap = rest_get("/v1/last/quote/" + venue + "/" + symbol)
cache[symbol] = snap; snapshot_ts[symbol] = snap.meta.as_of
send({ op: "subscribe", product: PRODUCT, symbols: subscriptions })
last_rx = last_ping = now(); healthy_since = null; state = STREAMING
STREAMING:
frame = recv(timeout = RX_POLL)
if frame:
last_rx = now()
if frame.f == "tick" and frame.t >= snapshot_ts[frame.s]: apply(frame)
if frame.f == "error" and frame.code in (auth_failed, max_connections):
close(sock); state = NEEDS_OPERATOR # do not retry blindly
if now() - last_ping > PING_INTERVAL: send({ op: "ping" }); last_ping = now()
if now() - last_rx > SILENCE_TIMEOUT: close(sock); state = DISCONNECTED
if healthy_since is null: healthy_since = now()
if now() - healthy_since > STABLE_WINDOW: attempt = 0 # reset only after a stable run
Two details in that machine matter more than they look. The attempt counter resets only after the stream has been stable for a window, so a server that accepts connections and drops them a second later doesn't put the client in a tight loop. And auth_failed or max_connections stop the retry outright, because retrying a bad key or a connection cap at full speed accomplishes nothing except tripping rate limits. The DEX product has its own wrinkles around chain-prefixed symbols; those are covered in streaming on-chain DEX swaps over WebSocket without reconnect bugs, and the broader question of when to poll REST instead of holding a socket is in REST vs WebSocket for real-time market data.
Production monitoring: what to alert on#
Alert on symptoms the application actually experiences and keep the causes on dashboards. The Google SRE book chapter on monitoring makes this argument in general; for market data it comes down to a short list.
- Freshness by symbol: the maximum venue age and receive age across subscribed symbols, evaluated only during that symbol's open hours. Page when either crosses the per-class threshold for more than one check interval.
- Reconnect rate: connections per hour per process. A rising rate with no incident from the provider usually means a network path or a load balancer idle timeout on your side.
- Heartbeat round trip: time from ping to pong. A steady increase often precedes a disconnect.
- Snapshot resyncs and dropped-as-old frames after each reconnect. Many reconnects with zero resyncs means the restore step isn't running.
- Degraded and 503 stale_snapshot counts per symbol per hour, so a provider-side problem is distinguishable from your own.
- Clock skew between the host and an NTP reference, because every age calculation depends on it.
Common pitfalls#
A pure listener gets disconnected in a quiet market. A client that subscribes and then only reads frames sends nothing, and SiftingIO closes it after 90 seconds of client silence. The reconnect logic then reconnects it, and the log fills with a clean-looking cycle every minute and a half. The fix is the 60-second ping timer, and the tell in the logs is a disconnect interval that is suspiciously regular.
Silence after the close is treated as staleness. A check that doesn't consult the market calendar pages the on-call at every close, every holiday, and every half day. Gate the check on market status, and remember that forex and crypto have different weekend behaviour from US stocks.
The reconnect resumes on old state. Skipping the REST snapshot means the first tick after reconnect is applied on top of a cache that stopped minutes ago. For a last price this only matters until the next tick, but for anything derived from history (a VWAP, a rolling volatility, a position mark) the gap poisons the calculation until the window rolls over.
FAQ#
How old is "stale" for market data It depends on the instrument and the session. Derive the threshold from observed inter-arrival gaps per asset class during open hours, then add a margin. A single global number will be wrong for either your FX majors or your thin commodities.
Should the heartbeat use WebSocket ping frames or an application message Use whatever the feed documents. Browser clients can't send protocol-level pings, so most market data feeds define a JSON ping as well; SiftingIO expects { "op": "ping" } and answers with { "f": "pong" }.
Why compare venue time and receive time instead of one of them Because they detect different failures. Receive age catches a dead transport; venue age catches an upstream that keeps sending an unchanged price.
What should happen to a price the app has marked stale Stop treating it as current. Grey it out in the UI, block any automated decision that depends on it, and log the age. Whether to show the last known value or hide it is a product choice.
Does a reconnect need a REST snapshot if the server sends the last cached value on subscribe For a single symbol the cached value covers it. A REST snapshot with an explicit as_of gives you one place to resync many symbols, log the gap, and discard late frames consistently.
Checklist#
- Store venue timestamp and receive time separately; use a monotonic clock for the latter.
- Set freshness thresholds per asset class and session, derived from observed gaps, never a global constant.
- Gate staleness alerts on market open status.
- Send an application ping at least every 60 seconds; treat a missing pong as a dead socket.
- Track sequence numbers where the protocol has them; monitor timestamp monotonicity where it doesn't.
- Reconnect with capped exponential backoff and full jitter; reset the attempt counter only after a stable run.
- Stop retrying on auth_failed and max_connections.
- Seed the cache from a REST snapshot before subscribing; drop ticks older than the snapshot's as_of.
- Alert on freshness, reconnect rate, heartbeat round trip, resync counts, degraded counts, and clock skew.
The WebSocket frames, error codes, and REST snapshot endpoints referenced above are documented in full at Read the docs.
