Missing candles in OHLCV data have three different causes, and each one needs a different fix. Did the market close early or not open at all that day? Did the instrument simply not trade during that minute? Or did your download drop a page somewhere between the API and your database? A gap detector that can't tell these apart will either flag hundreds of false positives on every holiday or paper over real faults with invented prices. This post walks through a detection method that separates the three cases, using historical US stock bars, and then covers what a safe backfill looks like.
Build the expected bar grid in UTC#
Every bar carries a numeric t field: the bucket open time as Unix epoch milliseconds, in UTC. The detector's job is to generate the set of t values that should exist for a session and diff it against the set that came back. Getting the expected set right is most of the work.
For US stocks, the regular session runs 09:30 to 16:00 in New York time, which is 390 one-minute bars (6.5 hours times 60). The open times run from 09:30 through 15:59 inclusive. For a 24/7 instrument such as BTCUSD, a full day is 1440 one-minute bars and there is no session boundary to think about. Weekends and holidays don't apply either.
The trap is daylight saving time. Build the grid in the exchange's local zone and convert to UTC afterwards. If you hard-code a UTC offset instead, the grid breaks twice a year. US DST starts on Sunday 2026-03-08. Friday 2026-03-06 is at UTC-5, so its session runs 14:30 to 20:59 UTC. Monday 2026-03-09 is at UTC-4, so the session runs 13:30 to 19:59 UTC. A grid that assumes Friday's offset for Monday will report the first 60 bars of Monday as missing and the 60 real bars after 19:59 as unexpected extras: 120 wrong answers on a day with perfect data. Crypto bars never have this problem because the UTC day doesn't move.
Remove holidays and half-days with the trading calendar#
The grid also has to know which days are sessions at all. The market calendar endpoint returns closures and early closes for a market over a date range:
curl -H "X-API-Key: $SIFTING_KEY" \
"https://api.sifting.io/v1/fnd/markets/us_equities/calendar?from=2026-11-01&to=2026-12-31"
The response is a top-level data array plus a meta object. Each entry has date (YYYY-MM-DD), name, kind, state, and on half-days an early_close timestamp. kind is either holiday for a full closure or half_day for an early close, and state is closed or early_close. Thanksgiving comes back like this:
{"date":"2026-11-26","name":"Thanksgiving Day","kind":"holiday","state":"closed"}
Skip grid generation for any date whose kind is holiday. For a half_day entry, use its early_close value (an RFC3339 timestamp, present only on half-days) as the end of that day's grid instead of 16:00 local. A half-day that ends at 13:00 has 210 bars rather than 390, and a detector that doesn't know that will report 180 missing candles that were never supposed to exist. The from parameter defaults to today and to defaults to 90 days later, so fetch the calendar once per quarter and cache it. Field details are in the market calendar documentation.
Pull the bars and diff against the grid#
The example below fetches one session of 1-minute AAPL bars and compares it against the expected grid. The start and end parameters are both inclusive bounds on bar time, so requesting 13:30:00Z through 19:59:00Z on the DST Monday returns exactly the 390 regular-session bars if none are missing.
import os
import requests
import pandas as pd
HEADERS = {"X-API-Key": os.environ["SIFTING_KEY"], "Accept-Encoding": "gzip"}
BASE = "https://api.sifting.io/v1"
def fetch_bars(ticker, start, end, interval="1m"):
params = {"interval": interval, "start": start, "end": end, "limit": 1000}
rows = []
while True:
r = requests.get(f"{BASE}/hist/stocks/{ticker}/bars",
headers=HEADERS, params=params, timeout=30)
r.raise_for_status()
body = r.json()
rows.extend(body["data"])
cursor = body.get("meta", {}).get("next_cursor")
if not cursor: # next_cursor is absent on the last page
break
params["cursor"] = cursor
return pd.DataFrame(rows)
def expected_grid(day, session_end="15:59"):
# Build in exchange-local time, then convert to UTC epoch ms
idx = pd.date_range(f"{day} 09:30", f"{day} {session_end}",
freq="1min", tz="America/New_York")
return set((idx.tz_convert("UTC").asi8 // 1_000_000).tolist())
bars = fetch_bars("AAPL", "2026-03-09T13:30:00Z", "2026-03-09T19:59:00Z")
have = set(bars["t"].tolist()) # t is already numeric, compare it as-is
grid = expected_grid("2026-03-09")
missing = sorted(grid - have)
extra = sorted(have - grid)
print(f"expected={len(grid)} got={len(have)} missing={len(missing)} extra={len(extra)}")
Three details in that code matter. Each page is capped at 1000 bars (the default and the maximum), so a single session fits in one page but a multi-day pull walks meta.next_cursor until it's absent. The t, o, h, l, c, and v fields arrive as JSON numbers (t and v as integers, the prices as floats), so the diff compares them as numbers; the moment you serialize t to a string or parse it into a datetime, two representations of the same minute can stop matching. And the grid is a set difference in both directions. A non-empty extra list should first prompt checks for DST, half-day handling, and extended-hours data before you treat it as a fault in the source.
For a full history pull, loop over every date the calendar says is a session, build each day's grid with the right session_end, and record the missing list per day. The count and shape of a gap help narrow the cause, but they do not prove it. A day with all 390 bars missing may indicate a calendar mistake, a failed request, or an upstream outage. A contiguous block near a page boundary is a reason to inspect pagination and retries. Scattered single-minute holes may be genuine no-trade intervals or isolated delivery gaps, so verify them before classifying them.
Backfill without inventing prices#
The tempting fix is a forward fill: copy the previous bar into every hole. Suppose a session came back with 45 of its 390 bars missing, which is 11.5% of the day. Forward-filling those 45 bars copies the prior close into o, h, l, and c and either copies v or sets it to 0. If v is copied, the day's volume is inflated by whatever those 45 phantom bars carry. If h and l are copied, the day contains 45 bars of a range that never traded, which feeds straight into ATR, range breakouts, and anything built on true range. Copying volume distorts VWAP. Setting synthetic volume to zero avoids that particular distortion, but copied OHLC values can still corrupt range-based indicators. And none of it is labeled, so the next person to query the table can't tell a real bar from a copied one.
A safe backfill has three steps in a fixed order. First, re-request the exact missing window from the API with start and end set to the first and last missing t. Dropped pages and interrupted loops are common causes of gaps, and a second request can resolve them. Second, if the API returns the bar now, insert it. Third, if a second check still returns no bar, record the gap explicitly in a companion audit table or a clearly marked nullable row. Do not present it as a bar returned by the provider. Downstream code can then decide whether to drop the interval, carry the previous close only for chart rendering, or skip it in an indicator while preserving the distinction between observed and synthetic data.
Never fill volume. Never fill high and low. If a chart needs a continuous line, carry the close at render time and leave the stored table honest.
Common pitfalls#
HTTP 406 with gzip_required. The historical bars endpoint requires the Accept-Encoding header to include gzip. Most HTTP libraries send it by default, but a hand-built client, a proxy that strips encoding headers, or a curl call that never sets the header gets a 406 and a body of {"error":"gzip_required", ...}. Add the header; retrying without it won't help. See the historical stocks documentation for details.
A grid built with a fixed UTC offset. The 2026-03-09 example above is the whole story: 60 bars flagged missing from 13:30 to 14:29 UTC and 60 flagged extra from 20:00 UTC onwards, on a day with no data problem at all. If a detector suddenly reports exactly 60 or 120 errors on a Monday in March or November, check the offset before checking the data.
Comparing t after a type change. Some JSON parsers turn large integers into floats, some ORMs store epoch values as strings, and some pipelines round to seconds along the way. Two representations of the same minute then fail to match and every bar looks missing. Keep t as the integer millisecond value from the response through the whole comparison, and cast only when writing to a datetime column.
