Cursor pagination is the mechanism that lets you pull a multi-year run of 1-minute OHLCV bars through a REST API that returns at most 200 rows per response. The question comes up the first time a backtest needs more than a screenful of history: how do you page through roughly 98,000 minute bars per year for a single US stock, keep the sequence intact, and stay inside a monthly call budget This post walks through the mechanics using the SiftingIO historical bars endpoint, but the pattern applies to any cursor-based API.
Why market data APIs use cursors instead of page numbers#
Offset pagination (?page=37 or ?offset=7200) assumes the underlying list doesn't move while you're reading it. Market data moves constantly. New bars land at the end of the series every minute during the session, and occasionally a late print corrects a bar that was already served. With offsets, a row inserted ahead of your position shifts everything by one, so page 38 repeats the last row of page 37, or skips one. You won't see an error. You'll see a duplicate timestamp, or a hole, three weeks later when a strategy behaves oddly.
A cursor fixes this by encoding the position itself. The SiftingIO API returns a meta object on every paginated response with three fields: next_cursor, total, and as_of. The cursor is an opaque string. You pass it back as ?cursor=... on the next request and the server resumes exactly after the last row it sent you. When next_cursor is null, you've reached the end. The as_of timestamp tells you the snapshot moment the page reflects, which matters when the final page includes a bar that is still forming.
Page size is controlled with ?limit. The default is 50 and the ceiling is 200. For bulk history you want 200 every time; there is no reason to make four times as many calls.
The arithmetic before you write code#
A US equity trades about 390 regular-session minutes a day across roughly 252 trading days, so one year of 1-minute bars is close to 98,000 rows for one ticker (more if the response includes extended hours). At 200 rows per page that's around 490 requests per ticker-year. A three-year pull for ten tickers is about 14,700 requests.
That number decides which plan you need. The free tier allows 10,000 REST calls a month at 60 requests per minute, and history depth is capped at one month, so the free tier is for testing the loop rather than filling a research database. Builder raises the ceiling to 250,000 calls a month with a full year of history. Pro and above have full history. Check /pricing for current figures before budgeting; they change.
Rate limits are exposed on every response as X-RateLimit-Limit (burst capacity) and X-RateLimit-Remaining (tokens left). On a 429 the body carries rate_limit_exceeded and the Retry-After header tells you how many seconds to wait. A pagination loop should read those headers rather than sleeping a fixed interval.
A Python loop that follows the cursor#
The historical bars endpoints require gzip. Send Accept-Encoding: gzip or you'll get a 406 with the error code gzip_required. The requests library adds that header by default and decompresses transparently, so the code below only needs to be explicit about it for readers using a lower-level client.
import os, time
import requests
import pandas as pd
BASE = "https://api.sifting.io/v1"
HEADERS = {
"X-API-Key": os.environ["SIFTING_KEY"],
"Accept-Encoding": "gzip",
}
def fetch_bars(ticker, interval="1m", start=None, end=None):
params = {"interval": interval, "limit": 200}
if start: params["start"] = start # ISO 8601 date, see /docs for exact names
if end: params["end"] = end
rows, cursor = [], None
while True:
if cursor:
params["cursor"] = cursor
r = requests.get(f"{BASE}/hist/stocks/{ticker}/bars",
headers=HEADERS, params=params, timeout=30)
if r.status_code == 429:
time.sleep(int(r.headers.get("Retry-After", "1")))
continue
r.raise_for_status()
body = r.json()
rows.extend(body["data"]) # bar rows; check /docs for the exact key
cursor = body["meta"]["next_cursor"]
if int(r.headers.get("X-RateLimit-Remaining", "1")) < 5:
time.sleep(1)
if cursor is None:
return pd.DataFrame(rows), body["meta"]["as_of"]
bars, as_of = fetch_bars("AAPL", start="2025-01-01", end="2025-12-31")
print(len(bars), "bars as of", as_of)
Two design choices in that loop are worth naming. First, the cursor is the only state. Nothing is computed from row counts or timestamps to decide where the next request begins. Second, the 429 branch retries the same request with the same cursor. That's safe because the cursor is idempotent: the server resumes after the same row regardless of how many times you ask.
The same function works for the other bar endpoints by swapping the path segment. /hist/forex/EURUSD/bars returns OHLC in UTC with volume always zero. /hist/crypto/BTCUSD/bars returns base-asset volume. Intervals run from 1m to 1mo for stocks and 1m to 1h for forex and crypto.
Checking the result for gaps and duplicates#
Once the frame is built, verify it. Cursors protect the transport, but a session boundary, a half-day, or a resumed pull from a saved position can still leave the series in a state you didn't expect.
bars["ts"] = pd.to_datetime(bars["t"], unit="ms", utc=True)
bars = bars.drop_duplicates("ts").set_index("ts").sort_index()
# gaps larger than one interval inside a single trading day
for day, frame in bars.groupby(bars.index.date):
diffs = frame.index.to_series().diff().dropna()
gaps = diffs[diffs > pd.Timedelta(minutes=1)]
if len(gaps):
print(day, len(gaps), "intra-day gaps, largest", gaps.max())
Some gaps are real. A 1-minute bar exists only when there was activity in that minute, and a thinly traded ticker will legitimately skip minutes. Gaps that line up exactly with the boundary between two pages are the ones that mean the loop was wrong.
If you persist results and resume later, store the last t you have rather than the cursor. Cursors are opaque and shouldn't be assumed to stay valid indefinitely. Restarting with the start date set to your last timestamp and dropping the duplicate first row is the durable approach.
Common pitfalls#
A 406 on a request that worked yesterday from a different tool. curl does not send Accept-Encoding unless you pass --compressed, so a snippet copied from a Python session into a shell script fails with gzip_required. The fix is curl --compressed -H "X-API-Key: $SIFTING_KEY" "https://api.sifting.io/v1/hist/stocks/MSFT/bars?interval=1d". The same requirement applies to the XBRL financials and screener endpoints.
Treating total as a target count. meta.total describes the result set at as_of. If you page through a live session, the last page may reflect a bar that was still open when served, and a rerun an hour later may return one more row. Use next_cursor being null to know you've finished, and treat total as a sanity check only.
Building a cursor from the last timestamp. The cursor string is opaque and its format is not part of the API contract. Code that inspects, decodes, or constructs cursors will break without warning. Pass it back unchanged, and let your HTTP library handle URL encoding instead of concatenating it into the URL by hand.
Once the loop is reliable it's boring, which is the point. The same forty lines pull a decade of daily bars or a week of minute bars, and the only tuning is the sleep between pages when X-RateLimit-Remaining runs low. Read the docs for the exact request parameters, the bar schema, and the per-endpoint interval limits.
