Cursor pagination is how you pull a multi-year run of 1-minute OHLCV bars through a REST API that caps each response at a fixed number of rows. The question comes up the first time a backtest needs more than a screenful of history: how do you page through roughly 98,000 minute bars per year for a single US stock, keep the sequence intact, and stay inside a monthly call budget? This post walks through the mechanics on the SiftingIO historical bars endpoint, using a page size of 1,000 bars in the examples below, but the pattern applies to any cursor-based API.
Why market data APIs use cursors instead of page numbers#
Offset pagination (?page=37 or ?offset=7200) assumes the underlying list doesn't move while you're reading it. Market data moves constantly. New bars land at the end of the series every minute during the session, and occasionally a late print corrects a bar that was already served. With offsets, a row inserted ahead of your position shifts everything by one, so page 38 repeats the last row of page 37, or skips one. You won't see an error. You'll see a duplicate timestamp, or a hole, three weeks later when a strategy behaves oddly.
A cursor fixes this by encoding the position itself. Every bars response from the SiftingIO API carries a meta object with four fields: next_cursor, as_of, symbol, and interval. The cursor is an opaque string. You pass it back as ?cursor=... on the next request and the server resumes exactly after the last row it sent. When next_cursor is absent or null, you've reached the end. The as_of timestamp is the snapshot moment the page reflects, which matters when the final page includes a bar that is still forming.
Page size is controlled with ?limit. The examples below request 1,000 rows per page; that is the example page size, not a universal API maximum. Check the per-class historical bars reference for the current default and maximum on the endpoint you use. Stock bars come back ascending, oldest first, and cover regular-session minutes back to 2000. Prices are as-traded, with no split or dividend adjustment applied.
The request math at 1,000 bars per page#
A US equity trades about 390 regular-session minutes a day across roughly 252 trading days, so one year of 1-minute bars is close to 98,000 rows for one ticker. At 1,000 rows per page that is 99 requests per ticker-year (98,280 divided by 1,000, rounded up). A three-year pull for ten tickers is under 3,000 requests. That is a small number. It fits inside the free tier's 10,000 calls a month three times over.
Free, Builder, and Pro all include the full available historical archive. Stock bars can reach back to 2000 where listing history supports it. At the example page size, a complete 1-minute backfill can take around 2,600 requests per ticker, so request allowance and loading time matter even when history is included. Builder raises the monthly allowance to 250,000 calls. Check pricing for current quotas and rate limits before budgeting.
Rate limits matter more than the quota for this workload. The free tier allows 60 requests a minute, so a full year of one ticker takes just under two minutes of wall clock. Builder allows 100 requests a second, which moves the bottleneck to gzip decoding on your side. Every response includes X-RateLimit-Limit (burst capacity) and X-RateLimit-Remaining (tokens left). On a 429 the body carries rate_limit_exceeded and the Retry-After header says how many seconds to wait. A pagination loop should read those headers rather than sleeping a fixed interval.
A Python loop that follows the cursor#
The historical bars endpoints require gzip. Send Accept-Encoding: gzip or you'll get a 406 with the error code gzip_required. The requests library adds that header by default and decompresses transparently, so the code below is explicit about it only for readers using a lower-level client.
import os, time
import requests
import pandas as pd
BASE = "https://api.sifting.io/v1"
HEADERS = {
"X-API-Key": os.environ["SIFTING_KEY"],
"Accept-Encoding": "gzip",
}
def fetch_bars(ticker, interval="1m", start=None, end=None):
params = {"interval": interval, "limit": 1000}
if start: params["start"] = start # inclusive, YYYY-MM-DD or RFC 3339
if end: params["end"] = end # inclusive, defaults to now
rows, cursor = [], None
while True:
if cursor:
params["cursor"] = cursor
r = requests.get(f"{BASE}/hist/stocks/{ticker}/bars",
headers=HEADERS, params=params, timeout=30)
if r.status_code == 429:
time.sleep(int(r.headers.get("Retry-After", "1")))
continue
r.raise_for_status()
body = r.json()
rows.extend(body["data"])
cursor = body["meta"].get("next_cursor")
if int(r.headers.get("X-RateLimit-Remaining", "1")) < 5:
time.sleep(1)
if not cursor:
return pd.DataFrame(rows), body["meta"]["as_of"]
bars, as_of = fetch_bars("AAPL", start="2025-01-01", end="2025-12-31")
print(len(bars), "bars as of", as_of)
Each row has six fields: t (bar open time in Unix epoch milliseconds), o, h, l, c, and v. Two design choices in the loop are worth naming. First, the cursor is the only state. Nothing is computed from row counts or timestamps to decide where the next request begins. Second, the 429 branch retries the same request with the same cursor. That's safe because the cursor is idempotent: the server resumes after the same row regardless of how many times you ask.
The same function works for the other bar endpoints by swapping the path segment and always passing start, which is required there. /hist/forex/EURUSD/bars returns OHLCV in UTC; v is aggregated traded volume for the bar, summed across contributing sources. /hist/crypto/BTCUSD/bars returns base-asset volume. All bar routes accept the same interval set, 1m through 1h intraday plus 1d, 1w, and 1mo. If start predates coverage on crypto or DEX bars, the API returns 422 with code data_unavailable and an earliest field carrying the first date it can serve. Clamp start to that value and retry.
Checking the result for gaps and duplicates#
Once the frame is built, verify it. Cursors protect the transport, but a session boundary, a half-day, or a resumed pull from a saved position can still leave the series in a state you didn't expect.
bars["ts"] = pd.to_datetime(bars["t"], unit="ms", utc=True)
bars = bars.drop_duplicates("ts").set_index("ts").sort_index()
# gaps larger than one interval inside a single trading day
for day, frame in bars.groupby(bars.index.date):
diffs = frame.index.to_series().diff().dropna()
gaps = diffs[diffs > pd.Timedelta(minutes=1)]
if len(gaps):
print(day, len(gaps), "intra-day gaps, largest", gaps.max())
Some gaps are real. A 1-minute bar exists only when there was activity in that minute, and a thinly traded ticker will legitimately skip minutes. Gaps at page boundaries can indicate a pagination problem, but are not proof by themselves. Compare the timestamps on both sides of the boundary and re-request the affected window before attributing the gap to the loop.
If you persist results and resume later, store the last t you have rather than the cursor. Cursors are opaque and shouldn't be assumed to stay valid indefinitely. Restarting with start set to your last timestamp and dropping the duplicate first row is the durable approach, and it works because start accepts a full RFC 3339 timestamp, not only a date.
Common pitfalls#
A 406 on a request that worked yesterday from a different tool. curl does not send Accept-Encoding unless you pass --compressed, so a snippet copied from a Python session into a shell script fails with gzip_required. The fix is curl --compressed -H "X-API-Key: $SIFTING_KEY" "https://api.sifting.io/v1/hist/stocks/MSFT/bars?interval=1d". The same requirement applies to the XBRL financials and screener endpoints.
Assuming bars use the same page size as the rest of the API. Most list endpoints (filings, insider transactions, ownership) default to 50 rows and cap at 200. The bars examples here use 1,000 rows per page. For a large pull, a generic helper with limit=200 needs roughly five times as many requests as the same pull at limit=1000, with rounding at the final page. Set the limit per endpoint using its documented supported range, or leave it off to use its documented default.
Treating meta as a row count. The bars meta object has no total field. Don't pre-size buffers or progress bars from a count that isn't there. Estimate from the calendar (sessions times 390) and treat the estimate as approximate, because minutes with no activity produce no bar.
Requesting reverse order on stocks. order=desc is supported on forex and commodities bars only, where it anchors the first page at the newest bar so the latest N rows cost one request. Stock and crypto bars are ascending-only. To get the most recent session, set start to that date instead of paging forward from 2000.
The loop above, plus the gap check, is enough to fill a research database with a year of minute bars per ticker in about 99 requests each. Read the docs for the full parameter list on every bars route.
