Market data API rate limits: handle HTTP 429 without a retry storm

Market data API rate limits produce HTTP 429 in two different situations, and the correct reaction differs. For a temporary rate limit, wait at least as long as the server requests, then retry within a fixed attempt budget. Continued traffic can keep the limit exhausted. An exhausted monthly allowance does not clear until the counter resets, and a retry loop that treats it like a burst only generates rejected requests and fills logs. This guide shows how to tell the two apart from the error code in the response body, gives a small bounded retry for idempotent GET requests, and lists the changes that stop the 429s from happening in the first place.

The examples use the SiftingIO REST API, whose errors and rate limits page documents the headers and codes involved, and they query the EURUSD quote from the forex data API; the example targets authenticated market-data REST requests governed by this usage limiter. A free API key from the register page lets you try a normal quote request. Test both limit cases with the synthetic fixtures below instead of deliberately exhausting an allowance or flooding the API.

What the 429 is telling you#

SiftingIO rate limits REST calls with a token bucket. The bucket, and the monthly allowance behind it, belong to the billing account rather than to an individual key: every API key issued under the account draws from the same bucket and counts against the same monthly cap, and team members on the account share both. Responses processed by this usage limiter include two rate headers: X-RateLimit-Limit, the burst capacity of your tier, and X-RateLimit-Remaining, the tokens left in the current window. When the bucket is empty the API answers 429 with the body { "error": "rate_limit_exceeded", "retry_after": ... }, and the docs state that a Retry-After header, in seconds, is sent on 429 responses. On the day of writing the documented burst limits are 60 requests per minute on Free, then 100, 150 and 250 requests per second on Builder, Pro and Ultra. Check the pricing page before hard-coding any of these; they change.

You can see the headers on any successful call:

curl -sD - -o /dev/null -H "X-API-Key: $SIFTING_KEY" \
  "https://api.sifting.io/v1/last/quote/forex/EURUSD" | grep -iE 'ratelimit|quota'

The second situation is the monthly quota. On plans with a monthly call cap, requests beyond the allowance return 429 with a different error code: { "error": "monthly_quota_exceeded", "quota": ..., "used": ..., "retry_after": ... }, where quota is the monthly cap, used is the count so far and retry_after is the number of seconds until the counter resets. Capped plans also get X-Quota-Limit, X-Quota-Used and X-Quota-Remaining in the response headers, so a client can watch the month drain before the cap is reached. The reset is computed against the next calendar month in UTC, so don't derive it from your invoice date; read retry_after. That's a hard cap by design. Which plans are capped is on the pricing page.

The error code is the signal. Read the body first: rate_limit_exceeded is a burst and monthly_quota_exceeded is the month, whatever Retry-After says. A 429 whose body is neither, or isn't JSON at all, deserves a look before another request goes out, because the code could come from another gateway or an API error this client does not yet handle. The size of Retry-After on its own proves nothing about which case you're in. A client can choose the longest wait it will honour inline and defer anything longer to a scheduler, but that threshold is the client's policy, not a diagnosis. Two supporting signals help when logs are all you have: a burst 429 sits between successful calls, while an exhausted quota answers 429 to every call on every endpoint for as long as you keep trying, and X-Quota-Remaining: 0 supports the exhausted-allowance diagnosis. Two things a 429 never means are a bad key, which is 401 unauthorized, and a market your plan doesn't include, which is 403. Neither is fixed by waiting.

Diagnostic table#

A bounded retry for idempotent GETs#

The function below wraps a requests.Session and handles one GET at a time. It is deliberately small: a hard cap on attempts, connect and read-inactivity timeouts on every call, exponential backoff with full jitter, Retry-After honoured as a minimum when present and parseable, and three distinct exceptions so the caller knows whether to fix configuration, stop for the month, or hand the job to something that can wait longer. It only retries GETs, which are safe to repeat; don't reuse it for anything that writes.

import email.utils
import random
import time
from datetime import datetime, timezone

import requests

BASE = "https://api.sifting.io/v1"
RETRYABLE_5XX = {502, 503}


class NotRetryable(Exception):
    """4xx a retry cannot fix: bad key, no entitlement, bad request."""


class MonthlyQuotaExceeded(Exception):
    """The account's monthly call quota is used up. No retry will help."""


class Deferred(Exception):
    """Attempts ran out, the wait exceeded the budget, or the 429 body was not recognised."""


def error_code(resp):
    """The error field of a JSON error body, or None when the body is not JSON."""
    try:
        body = resp.json()
    except ValueError:
        return None
    return body.get("error") if isinstance(body, dict) else None


def retry_after_seconds(resp, now=None):
    """Parse Retry-After as delta-seconds or an HTTP date. None if absent or malformed."""
    raw = resp.headers.get("Retry-After")
    if raw is None:
        return None
    raw = raw.strip()
    if raw.isdigit():
        return float(raw)
    try:
        when = email.utils.parsedate_to_datetime(raw)
    except (TypeError, ValueError):
        return None
    if when is None:
        return None
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    now = now or datetime.now(timezone.utc)
    return max(0.0, (when - now).total_seconds())


def backoff(attempt, max_wait):
    """Full-jitter backoff: uniform between 0 and min(max_wait, 0.5 * 2 ** attempt)."""
    return random.uniform(0, min(max_wait, 0.5 * 2 ** attempt))


def get_with_retry(session, path, params=None, *, max_attempts=4, max_wait=20.0,
                   jitter=0.5, timeout=(3.05, 10), sleep=time.sleep):
    """Bounded retry for an idempotent GET. Returns the response or raises."""
    for attempt in range(1, max_attempts + 1):
        try:
            resp = session.get(BASE + path, params=params, timeout=timeout)
        except (requests.ConnectionError, requests.Timeout) as exc:
            if attempt == max_attempts:
                raise Deferred(f"network error after {attempt} attempts: {exc}")
            sleep(backoff(attempt, max_wait))
            continue

        if resp.status_code < 400:
            return resp

        if resp.status_code == 429:
            code = error_code(resp)
            if code == "monthly_quota_exceeded":
                # The month is spent. Nothing in this loop can change that.
                raise MonthlyQuotaExceeded(resp.text[:200])
            if code != "rate_limit_exceeded":
                # Unknown does not prove a burst; leave it for investigation.
                raise Deferred(f"unrecognised 429 body: {resp.text[:200]}")
        elif resp.status_code not in RETRYABLE_5XX:
            raise NotRetryable(f"{resp.status_code} {resp.text[:200]}")

        if attempt == max_attempts:
            raise Deferred(f"{resp.status_code} after {attempt} attempts")

        hint = retry_after_seconds(resp)
        if hint is None:
            wait = backoff(attempt, max_wait)
        elif hint > max_wait:
            # Never shorten a server delay and retry early. Hand the decision up.
            raise Deferred(f"server asked for {hint:.0f}s, budget is {max_wait:.0f}s")
        else:
            wait = hint + random.uniform(0, jitter)
        sleep(wait)
    raise Deferred("no attempts made")


if __name__ == "__main__":
    import os
    s = requests.Session()
    s.headers["X-API-Key"] = os.environ["SIFTING_KEY"]
    s.headers["Accept-Encoding"] = "gzip"
    print(get_with_retry(s, "/last/quote/forex/EURUSD").json())

The body's error field is read before anything else. monthly_quota_exceeded raises MonthlyQuotaExceeded on the first response without sleeping, whether Retry-After is missing, malformed or a few seconds, because no wait inside a request loop can bring the month back. A 429 whose body is not JSON or carries an unknown code raises Deferred straight away; the origin and meaning need investigation rather than an assumed burst retry. Only rate_limit_exceeded continues into the wait logic.

The max_wait budget is a client policy, not a diagnosis. It is the largest server-requested delay accepted inline before adding up to 0.5 seconds of jitter, not a total request deadline. With the default settings, a requested wait above twenty seconds is deferred to a scheduler that can honour the full period, and the function raises Deferred with the server's figure rather than shortening the delay and retrying early, which would only earn another 429. When Retry-After is present and within budget it is treated as a minimum: a uniform 0 to 0.5 s of jitter is added so that many workers released by the same header don't all retry in the same instant. When the header is missing or malformed the function falls back to full-jitter backoff. With max_attempts=4 the caps are 1, 2 and 4 seconds; the fourth attempt is the last, so the 8-second step never runs and the longest single fallback wait is 4 seconds. A 401, 403, 404, 400 or 406 raises NotRetryable on the first response, because a second identical request will get the same answer.

The timeout=(3.05, 10) tuple is a connect timeout and a read timeout. The read value is an inactivity limit between bytes, not a deadline for the whole response, so a slow but steady body can take longer than 10 seconds to arrive. If you need a hard ceiling per request, enforce it in the caller. Scope: one request, one process, no shared state. It doesn't throttle proactively, and it doesn't coordinate between processes; those are covered below.

Test it with fixtures, not with your quota#

The session and sleep parameters exist so the boundaries can be checked without a network or a real key. Save the function above as retry429.py, put the following in test_retry429.py and run pytest. A scripted session returns canned responses in order and a list captures the waits:

import json

from retry429 import Deferred, MonthlyQuotaExceeded, NotRetryable, get_with_retry

PATH = "/last/quote/forex/EURUSD"
BURST = json.dumps({"error": "rate_limit_exceeded", "retry_after": 2})
MONTHLY = json.dumps({"error": "monthly_quota_exceeded", "quota": 10000,
                      "used": 10000, "retry_after": 259200})


class FakeResp:
    def __init__(self, status, headers=None, text=""):
        self.status_code, self.headers, self.text = status, headers or {}, text

    def json(self):
        return json.loads(self.text)  # ValueError on an empty or non-JSON body, like requests


class FakeSession:
    def __init__(self, script):
        self.script = list(script)

    def get(self, url, params=None, timeout=None):
        return self.script.pop(0)


def run(script):
    """Return (waits, responses never requested, exception or None)."""
    waits, sess = [], FakeSession(script)
    try:
        get_with_retry(sess, PATH, sleep=waits.append)
    except (Deferred, MonthlyQuotaExceeded, NotRetryable) as exc:
        return waits, len(sess.script), exc
    return waits, len(sess.script), None


def test_burst_honours_retry_after_as_a_minimum_plus_jitter():
    waits, left, exc = run([FakeResp(429, {"Retry-After": "2"}, BURST), FakeResp(200)])
    assert exc is None and left == 0
    assert len(waits) == 1 and 2.0 <= waits[0] <= 2.5


def test_monthly_quota_stops_at_once_without_retry_after():
    waits, left, exc = run([FakeResp(429, {}, MONTHLY), FakeResp(200)])
    assert isinstance(exc, MonthlyQuotaExceeded)
    assert waits == [] and left == 1


def test_monthly_quota_ignores_a_short_or_malformed_retry_after():
    for header in ("3", "soon"):
        waits, left, exc = run([FakeResp(429, {"Retry-After": header}, MONTHLY), FakeResp(200)])
        assert isinstance(exc, MonthlyQuotaExceeded), header
        assert waits == [] and left == 1


def test_unknown_429_body_stops_for_investigation():
    for text in ("", "<html>throttled</html>", json.dumps({"error": "something_else"})):
        waits, left, exc = run([FakeResp(429, {"Retry-After": "1"}, text), FakeResp(200)])
        assert isinstance(exc, Deferred), text
        assert waits == [] and left == 1


def test_attempts_are_bounded_and_backoff_caps_at_four_seconds():
    waits, left, exc = run([FakeResp(429, {}, BURST)] * 4)
    assert isinstance(exc, Deferred) and left == 0
    assert len(waits) == 3
    for wait, cap in zip(waits, (1.0, 2.0, 4.0)):
        assert 0 <= wait <= cap


def test_malformed_retry_after_falls_back_to_backoff():
    waits, left, exc = run([FakeResp(429, {"Retry-After": "soon"}, BURST), FakeResp(200)])
    assert exc is None and len(waits) == 1 and 0 <= waits[0] <= 1.0


def test_past_http_date_waits_only_the_jitter():
    past = "Wed, 21 Oct 2015 07:28:00 GMT"
    waits, left, exc = run([FakeResp(429, {"Retry-After": past}, BURST), FakeResp(200)])
    assert exc is None and len(waits) == 1 and 0 <= waits[0] <= 0.5


def test_long_retry_after_is_deferred_not_shortened():
    waits, left, exc = run([FakeResp(429, {"Retry-After": "86400"}, BURST), FakeResp(200)])
    assert isinstance(exc, Deferred)
    assert waits == [] and left == 1


def test_auth_and_entitlement_errors_never_retry():
    for status in (400, 401, 403, 404, 406):
        waits, left, exc = run([FakeResp(status, {}, '{"error":"unauthorized"}'), FakeResp(200)])
        assert isinstance(exc, NotRetryable), status
        assert waits == [] and left == 1

All of these fixtures are synthetic; none is a recorded API response. The monthly body uses the field names the API sends, with numbers chosen for the test.

Cut the polling that caused the 429#

A retry loop is damage control. The cheaper fix is fewer calls.

Move live prices off REST. Polling a snapshot endpoint many times a second for a handful of symbols can quickly hit a request-rate limit. The WebSocket at wss://stream.sifting.io/ws/v1 sends the last cached value on subscribe and then pushes updates, which removes the loop entirely. The trade-offs, and when a snapshot is still the right call, are in Real-time FX and crypto quotes: REST snapshots and WebSocket streams. The stream has its own caps, returned as max_connections and max_subscriptions error frames, so reconnect with the same backoff discipline as above. A reconnect storm is a retry storm on another port.

Cache what you already have. A dashboard that repaints every 250 ms doesn't need a new quote every 250 ms. Keep the last response with its timestamp and refresh on a timer you choose, then decide from the age what to show; Delayed vs live stock prices: how to tell which one your app is showing covers how to classify that age honestly.

Poll less when the market is closed. GET /v1/fnd/markets/us_equities/status reports whether US equities are open, and a cron that pulls a US stock snapshot every minute all weekend mostly spends its allowance re-fetching the same value. Slow the timer down outside session hours instead of switching it off.

Store history and top it up. Historical bars are paginated with a cursor; fetch a window, store it, and on the regular schedule request only the bars after the last one you hold. Stored bars are not final forever, since later corrections can change values you already have, so plan an occasional re-pull of a trailing window rather than treating the archive as write-once. Send Accept-Encoding: gzip on every /fnd/* and /hist/* call so heavy endpoints don't answer 406 and cost a second request.

Coordinate the account, not the key. Every key on the account draws from the same bucket and the same monthly cap, so three workers with three keys are still three workers on one bucket, and issuing more keys adds no capacity. Put a single fetcher, or a shared token bucket in something like Redis, in front of the account so requests are spaced before they leave your network. Then the retry logic above becomes the rare path, which is where it belongs. If you're building a quote consumer from scratch, Live bid and ask price API: build a correct EURUSD and BTCUSD quote client shows the client structure this retry function slots into.

For the current headers, codes and per-tier limits, read the docs.