Sentiment Data Latency, Publication Lags and Vintages

Article

Why do timing errors ruin sentiment research?

A calculation can be arithmetically perfect and still be invalid, because it used information from the future. This is especially common with sentiment and positioning data, since many of the best-known series describe an earlier moment and are released afterwards.

Consider a weekly futures-positioning figure. A chart plots it on Tuesday, because Tuesday is the date the positions were measured. Years later, a reader sees a tidy picture: positioning on Tuesday, price action on Wednesday and Thursday. But on that Tuesday, nobody outside the reporting system knew the aggregate number. It became public later in the week. A backtest that "acts" on the figure on Tuesday is using a value that did not yet exist for the public. This error is called look-ahead bias, and the look-ahead bias guide covers its general mechanics.

The lesson is not that slow data is bad data. It is that every observation has two clocks, and a careful reader keeps both in view. For a wider view of how much weight different kinds of evidence deserve, see the source ladder for market sentiment.

What four dates should every data point carry?

Most dashboards store one column called "date." That single field hides at least four different things. A careful analyst, or a careful reader of someone else's analysis, separates them.

These dates solve different problems. The observation date tells you what the market looked like. The publication date tells you when anyone could have reacted. The retrieval date tells you how fresh your copy is, which says nothing about how fresh the source is. The methodology date tells you whether two points on a long chart are comparable at all.

When any of these is missing, a quiet assumption fills the gap, and that assumption is usually that everything happened at once. It did not.

How do the major sentiment sources differ in timing?

Different datasets sit at very different points on the lag spectrum. Here are the common ones.

CFTC Commitments of Traders

The Commodity Futures Trading Commission (CFTC) describes its Commitments of Traders (COT) reports as built from data for a Tuesday and released later in the same week. The CFTC receives the data from reporting firms on Wednesday morning, then corrects and verifies it before release, which its own page says generally happens on Friday afternoon. So the "Tuesday" position is, in practice, public roughly three days later.

Any historical analysis that treats Tuesday as the moment of knowledge is retrospective, not tradable. That is fine for studying how positioning related to later price moves, as long as the analysis is labelled that way. The Commitments of Traders guide and the COT net position indicator page explain what the data measure and where they fall short.

The CFTC's explanatory notes also flag something that matters for long histories: staff may reclassify a trader if they have additional information about how that trader uses the markets. The categories in the report are therefore judgments that can change, not fixed labels. The notes also say reported positions typically represent 70 to 90 percent of open interest in a given market, so the data are a large sample, not a complete census.

SEC Form 13F

Form 13F is the quarterly holdings report that large institutional managers file with the Securities and Exchange Commission (SEC). The filing describes holdings as of the end of a calendar quarter. The SEC's frequently asked questions state that filings are due within 45 days after the end of the quarter. A position held on June 30 may therefore not be visible until mid-August.

Amendments exist too. The SEC's guidance discusses an amendment submission type (13F-HR/A) used to correct errors or add omitted securities, and says multiple amendments for the same quarter can occur. A holdings snapshot you downloaded in August may differ from the one on file next year. The 13F reporting lag page goes deeper on this particular delay.

SEC Form 4

Form 4 is far timelier. An SEC investor bulletin explains that, in most cases, an insider who executes a transaction must file Form 4 within two business days following the transaction date. Even so, the trade and the public filing are two separate timestamps. The guide to reading Form 4 on EDGAR shows where each date appears on the form.

There is also a second layer of lag. The SEC publishes insider transaction data sets compiled from Forms 3, 4 and 5, and states that those data sets are updated quarterly. If you work from a bulk data set rather than the filing itself, the dataset's own refresh cadence adds to the original filing lag. A transaction filed in week two of a quarter might not appear in the compiled file until later. Always ask which copy of the data you are reading.

FINRA margin balances

FINRA's margin statistics describe month-end debit balances in customer margin accounts. FINRA's page says the data are collected as of the last business day of the month and that updates are generally published in the third week of the month following the reference month. That means the figure labelled with one month's end is typically public well into the next month.

Margin balances are therefore useful for broad leverage context, not for timing. The FINRA margin debt indicator page and the margin debt and leverage guide explain what the series can and cannot say.

Cboe market statistics

Exchange-published statistics such as put/call ratios can arrive far faster, sometimes within the trading day. Cboe's daily market statistics page, for example, presents put/call ratios and volume tables, and states that the summary data are compiled for the convenience of site visitors and furnished without responsibility for accuracy. A reader still needs to know whether a figure is real-time, delayed, end-of-day, or based on a settlement price, because those are different things even when they sit on the same page.

Is publication lag a flaw?

No. Hiding it is the flaw.

Slow data can still be valuable. Form 13F reveals institutional holdings that no daily series can reveal, and COT data show classified futures positioning that price alone cannot. Their value is not destroyed by the lag. Their use case changes: they are better for structural context ("has positioning been crowded for months?") than for fast tactical timing ("what happened this morning?").

A trustworthy chart or article tells the reader what the lag permits. "This is quarter-end holdings, public up to 45 days later" is an honest label. "Institutions are buying" is not, because it implies a present tense the data cannot support.

How should freshness be described?

Plain freshness labels beat vague ones. The useful classes are:

• Intraday: updates during the trading session.
• Daily: updates once per trading day, usually after the close.
• Weekly: updates once a week.
• Monthly: updates once a month, often with a delay after month end.
• Quarterly: updates once per quarter or filing cycle.
• Event-filed: appears when something happens, such as a filing.
• Irregular: no fixed schedule.

Then show the age of the latest observation next to the label. A monthly series should never be called "live" because a website refreshed its page today.

What is the difference between retrieval time and source freshness?

A common dashboard bug looks like this. A page says "Updated 2 minutes ago," but the economic series underneath was last observed three weeks ago. The label only describes when the software fetched the data. It says nothing about when the source last changed.

A reader can protect themselves by looking for two separate statements:

Data retrieved: 2 minutes ago Latest observation: August 31

If a page offers only the first, assume nothing about the second. A small distinction like this one changes how much trust a chart deserves.

What are vintages, and why do they matter?

Some datasets are revised. When the source corrects a value, today's chart shows the corrected history, while the values analysts had at the time may have been different.

A vintage is the version of a series as it stood on a particular date. The Federal Reserve Bank of St. Louis maintains ALFRED (Archival Federal Reserve Economic Data), which describes itself as "economic data time travel": it preserves each release of a series so you can retrieve the data as it was published on a specific past date. The FRED documentation explains the same idea through a "real-time period," meaning when facts were true or when information was known until it changed.

For research, both views are useful, and they answer different questions:

• Latest vintage answers "what does the source report today about that period?"
• As-published vintage answers "what could an analyst have known on that date?"

A backtest that tries to mimic what a person could have done should use as-published values whenever possible. A study of how an economy actually evolved may reasonably prefer the corrected history. The mistake is mixing the two without saying so. The point-in-time macro data and revision risk guide covers this problem for economic series in more depth, and the guide on data lineage and point-in-time research covers it for backtests more generally.

How do filing amendments and classification changes affect history?

Two kinds of change can alter a historical series even when the raw numbers are never "revised" in the usual sense.

Amendments. A corrected filing can change how history should be read. An amended 13F or a corrected Form 4 supersedes an earlier version. A careful researcher keeps a record of which filing a number came from (the SEC assigns each filing an accession number) and whether it was later amended, so the lineage stays inspectable.

Methodology changes. Category definitions, index rules, product specifications and reporting thresholds can all evolve. The CFTC itself notes that it periodically adjusts reporting thresholds and may reclassify traders. A long historical series can therefore cross several methodology regimes. Unless a source says its history is comparable across a break, treat the chart as one that may contain seams, and be cautious about reading a level from 2010 against one from 2024 as if the ruler never changed.

What is the safe alignment rule for backtests?

The default that avoids most look-ahead errors is simple:

A value becomes usable only at or after the moment it became public.

If the exact publication time is not available for older data, use a conservative assumption, such as the next trading session after the known release date, and say so. It is better to shift a value later and lose a few observations than to build an artificially perfect result.

Daily data can still be misaligned. A statistic published after the 4:00 p.m. equity close cannot have influenced that day's closing trade. Different markets close at different times, so a cross-asset study has to attend to time zones as well as dates. Where a source gives only a date (no clock time), keep it as a date. Converting a date into midnight in some time zone can shift it to a different calendar day when displayed elsewhere.

A worked example with illustrative numbers

The numbers below are invented to show the mechanics, not real data.

Suppose a weekly positioning series shows a value of 70 for a Tuesday in an imaginary month. Suppose the source publishes that value the following Friday afternoon. Suppose a researcher tests a rule over ten years using price data that are available every day.

• Naive alignment: The researcher places 70 on Tuesday and lets the rule react at Tuesday's close. The rule gets three days of information that did not exist yet.
• Publication-aligned: The researcher makes the value usable only after Friday afternoon, so the earliest reaction is the next session after publication (Monday).

Between Tuesday and Friday the market moved. If positioning and price are correlated across those days, the naive version captures a relationship that was never accessible. The result can look impressive and still be nothing a person could have done. Note, too, that the first version tells you something true about how positioning related to the market. It simply answers a different, retrospective question.

The same logic applies at a larger scale. For a quarterly filing, the gap is up to 45 days, not three. An analysis that aligns 13F holdings to the quarter end can gain weeks of information the public lacked.

What happens when data go stale?

Dashboards often carry the last value forward so that charts have no gaps. Carrying forward is reasonable. Presenting it as new information is not.

A good display does four things. It keeps the last value, because the history is still useful. It marks the value stale once the expected update window has passed. It visually distinguishes stale values from current ones, in a way that does not depend on color alone. And it does not recalculate a "current" combined reading as though the stale component had just updated.

Staleness must be judged against each source's cadence. An intraday options statistic may be stale after a few hours. A daily close is stale after the next expected session. A weekly COT value is stale once the next scheduled release has passed. A quarterly 13F snapshot is stale once the next filing cycle completes. One universal rule, such as "older than 24 hours," is wrong for almost all of them: it would flag healthy weekly and monthly series as broken, and pass a broken daily one.

How does freshness interact with a combined sentiment reading?

When several indicators are blended into one reading, their ages differ. A weekly positioning value, a daily credit spread, an intraday options figure and a monthly margin balance cannot sensibly share one timestamp. Each component keeps its own observation and publication dates, and the combined view records when the whole picture was assembled. The sentiment composite framework discusses how combined readings are built, including why completeness matters.

If one component is stale, a responsible blend either reduces its influence by a stated rule or leaves it out and reports lower completeness. Keeping full weight while calling the output current is the failure to avoid.

One way to downweight by age uses the expected update interval T and the age A:

Freshness = max(0, 1 - A / (k x T))

Here k is a tolerance multiplier chosen and disclosed in advance. It is one possible method among many. What matters is that the decay follows the source's own cadence (a weekly series should not decay like an intraday ratio) and that the rule is public.

A related caution applies to cross-asset comparisons, such as those in the cross-asset risk appetite guide: markets close at different times and many series update on different calendars, so "same day" often means "different information."

What data-quality states should a reader look for?

A data point is not simply right or wrong. A well-labelled source or display distinguishes among these states:

• Current: within its expected update window.
• Stale: past the window, last value carried forward.
• Missing: no value exists, which is not the same as zero.
• Revised: the value changed after first publication.
• Preliminary: first estimate, expected to be revised.
• Source unavailable: the provider could not be reached.
• Methodology break: the definition changed.

Look in particular for the difference between missing and zero. A missing observation treated as zero looks like a calm, neutral market and is simply false.

What does backfilled history actually prove?

When an indicator is built today, its history is usually computed after the fact from archived data. That backfilled history is useful for context. It does not prove the indicator was available, or that anyone could have used it, in real time.

A backfilled series was typically computed with today's definitions, today's data cleaning and today's knowledge of which categories mattered. A series that was actually published at the time carries a different kind of evidence: it shows what the world could see. If a chart does not say which one it is, treat it as backfilled.

Which misconceptions come up most often?

"If the date on the chart is Tuesday, the market knew it on Tuesday." No. The chart date is the observation date. The knowledge date is the publication date.

"Updated today means current." No. Updated today usually describes the software, not the data.

"Revised data are always better data." They are better for describing what happened. They can be misleading for judging what people could have done.

"A stale value is a useless value." No. Slow data are still informative when labelled as slow.

"One timestamp is enough." For daily prices the observation and publication moments nearly coincide, so one field seems fine. For filings, positioning reports, margin statistics and macroeconomic releases, they do not coincide, and a single field hides the difference.

"A longer history is always a stronger test." Only if the definitions, classifications and publication practices stayed comparable across the whole period.

What does latency not tell you?

Knowing a series' lag does not tell you whether the signal in it is any good. A well-timed series can still be uninformative, and a slow one can be useful for the right question. Latency analysis answers only whether a claim about timing is honest. It is a check on method, not a source of conclusions.

It also does not imply that faster is better. Intraday statistics have their own problems: they can be noisy, revised, or dependent on how products are counted. And no timing discipline turns descriptive data into a forecast. Sentiment and positioning data describe behaviour, exposures or prices that already exist.

A reading routine for any sentiment chart

Before trusting a sentiment chart or a claim built on one, a careful reader asks a short series of questions, roughly in this order:

1. What was observed, and when? Find the observation date or period.
2. When did the public first see it? Find the publication date, or at least the source's usual cadence.
3. How old is the latest point? Compare the latest observation date with today, not the retrieval time.
4. Could this value have changed since? Check for revisions, amendments or reclassification.
5. Do the dates in a historical chart match when people could have known? If not, is the chart labelled as retrospective?
6. Are stale or missing values flagged? Look for a visible, non-color-only marker.
7. Did the definitions stay constant? Look for notes on methodology breaks.

This is also a practical exercise. Take one Form 13F filing, one Form 4 filing, one COT release and one monthly margin observation. Write down the period each describes and the day it became public, then place all four on one timeline. The exercise shows quickly that "market data" are not one synchronous stream.

How should time-sensitive claims be written?

The same discipline applies to prose. A sentence like "institutional investors own X" invites readers, and automated summarizers, to treat it as current. A sentence such as "as of the quarter ended June 30, a filer reported holding X, in a filing due within 45 days of quarter end" can be quoted without losing its meaning. Putting the as-of date inside the sentence is cheap and prevents old numbers from passing as contemporary.

For a broader view of how this fits into evaluating a claim, the research source ladder shows how to judge the quality of evidence behind a number, and the market sentiment hub gathers the related guides. Readers who want a more technical treatment of how historical data can be stored so that past states can be replayed can look at the guide on point-in-time storage and historical replay.

Educational use and limitations

This material is educational. It is not a recommendation to buy, sell, short, hedge or hold any security, option, futures contract, fund or digital asset. Sentiment and positioning data describe behaviour, exposures or prices that already exist. They do not reveal a complete causal explanation, and they do not guarantee future results. The practical value of timing discipline is that it makes assumptions explicit and helps a reader test a thesis against more than one independent source.

Frequently Asked Questions

References

• CFTC: Commitments of Traders
• CFTC: Commitments of Traders Explanatory Notes
• SEC: Frequently Asked Questions About Form 13F
• SEC: Investor Bulletin, Insider Transactions and Forms 3, 4, and 5
• SEC: Insider Transactions Data Sets
• FINRA: Margin Statistics
• Cboe: U.S. Options Daily Market Statistics
• Federal Reserve Bank of St. Louis: ALFRED, Archival Federal Reserve Economic Data
• Federal Reserve Bank of St. Louis: FRED API Real-Time Periods