How do you evaluate a market data API before it carries production traffic? A quick demo proves very little, since almost any provider can return a plausible price for AAPL on a Tuesday afternoon. The failures that cost real money show up later. A quote goes stale and keeps rendering as live. A WebSocket drops at 3 a.m. and nothing reconnects. A licensing clause turns out to forbid the exact chart you shipped. Most of these are catchable in a week of structured testing. The checklist below is vendor neutral; run it against every provider you're considering, including the one publishing this post.
Price accuracy, bad ticks, and staleness#
Start with the hardest question: is the price right? For a fragmented market like crypto or FX there's no single official print, venues disagree by design, so "right" means defensible. Pull the same instrument from the candidate API and at least two independent references, log all of them for a few days, and chart the deviation. Small, symmetric noise is normal. Persistent one-sided offsets, or occasional spikes to a nonsense value, are the finding.
Then ask how the provider handles bad ticks. Every raw feed occasionally prints garbage: a fat-finger trade, a crossed book, a decimal shifted three places. The evaluation question is whether that garbage reaches you. Look for documentation of the filtering method, and prefer providers that expose quality metadata (source, ingest, and publish timestamps, or an explicit quality flag) over ones that simply claim their data is clean.
Staleness is the quieter version of the same problem. A feed that keeps serving the last known price during an upstream outage looks healthy on every dashboard while being wrong. Test it directly: request a live quote for an instrument whose market is closed and inspect what comes back. Is it labeled? Does the timestamp make the age obvious? Some APIs return an explicit stale-data error with the last good timestamp once data exceeds a freshness threshold. Silence is the answer you don't want.
Live vs historical consistency, gaps, and timestamps#
Backtests run on historical bars; production runs on the live feed. If those two disagree, research results won't survive deployment. The test is mechanical: record the live stream for a full session, then download the historical 1m bars for the same session and diff them. Small differences at bar boundaries are expected, because late prints and aggregation windows guarantee some. Systematic drift between the two means they come from different pipelines.
While you have a month of minute bars, count them. A liquid US large-cap trades roughly 390 regular-session minutes a day. Thin symbols legitimately have empty minutes, so run the check on something liquid; if whole stretches are missing there, backfills are unreliable. Check a holiday and a half-day too, and verify the provider's market calendar matches what actually happened.
A historical pull looks like this against one concrete API shape (adjust for whichever provider you test):
curl -s -H "X-API-Key: $SIFTING_KEY" -H "Accept-Encoding: gzip" \
"https://api.sifting.io/v1/hist/stocks/AAPL/bars?interval=1m"
Finally, pin down timestamp semantics in writing: UTC or venue-local, bars labeled by open time or close time, and how daylight saving transitions are handled for session-based markets. Getting one of these wrong shifts every candle you store, and nothing errors.
Latency with p50, p95, and p99#
One fast response is luck. Latency only means something as a distribution over hundreds of requests, summarized as p50, p95, and p99, measured from the region where your code will actually run. Benchmarking from a laptop on Wi-Fi mostly measures your ISP.
import os, time, requests
url = "https://api.sifting.io/v1/last/quote/forex/EURUSD"
session = requests.Session()
session.headers["X-API-Key"] = os.environ["SIFTING_KEY"]
samples = []
for _ in range(500):
t0 = time.perf_counter()
session.get(url, timeout=5).raise_for_status()
samples.append((time.perf_counter() - t0) * 1000)
time.sleep(0.2)
samples.sort()
for p in (50, 95, 99):
print(f"p{p}: {samples[int(len(samples) * p / 100) - 1]:.1f} ms")
The p99 is what your users experience during a busy market. If p50 is 40 ms and p99 is 2 seconds, there's a queuing problem to design around, and it's better to know before launch.
Give the WebSocket the same scrutiny over a longer window. Leave a connection subscribed for at least 48 hours and log every disconnect with a timestamp. Then test the parts your client has to get right: how fast a reconnect plus resubscribe completes, whether ticks are missed during the gap, and what keepalive the server expects (many close idle connections after 60 to 90 seconds without a client frame, so a client that never pings dies quietly). Kill the connection yourself at least once and watch your own recovery path run. Schedule part of the soak across a high-impact economic release, because disconnects and bad ticks cluster exactly when you can least afford them.
Coverage, rate limits, and the operational fine print#
Marketing pages list asset classes; production needs specific symbols. Write down every instrument your product requires and query each one before signing anything. The exotic FX pair or small-cap ticker you need is exactly what a coverage page glosses over. Check history depth per plan tier as well, since "full history" often belongs to a higher tier than the one you priced.
Rate limits deserve a sustained test, because the number that matters is rarely the burst figure. Run your projected production request rate for an hour and watch the response headers (X-RateLimit-Remaining and Retry-After are the common ones). Learn whether limits are per second, per minute, or a monthly quota, and how the API behaves at the boundary. A clean 429 with Retry-After is workable; silent throttling is worse.
Then read the operational record. A status page with real incident history beats a spotless one, since a provider that has published nothing for a year is more likely under-reporting than perfect. Check whether the SLA has teeth (service credits, a defined uptime target per tier) or is best-effort wording, and how incidents get communicated while they're happening.
Licensing comes last on the checklist and first in cost of getting wrong. Display rights, redistribution rights, and internal use are three different permissions, and data sourced from major US exchanges in particular carries display and redistribution restrictions. Get written confirmation that your specific use, whether that's showing prices to end users, storing history, or feeding a model, is covered by the plan you're buying.
Common pitfalls while benchmarking#
- Cold connections poison latency numbers. The first request on a fresh connection pays for the TCP and TLS handshake, often 100 ms or more. Reuse a session (as the script above does) or your p50 measures handshakes instead of the API.
- Heavy endpoints may require gzip. Some providers return 406 on large historical or fundamentals endpoints unless the request sends Accept-Encoding: gzip. A client that treats that 406 as an empty result will log a successful backfill containing nothing.
- Weekend tests flatter every provider. Quiet markets produce clean feeds, so an evaluation that runs entirely on a Saturday only shows how the API behaves when nothing is happening. Overlap at least one CPI print, jobs report, or central bank decision.
Score each candidate on evidence you collected yourself rather than claims on a pricing page. If you want a published spec to point these tests at, read the docs.
