Key Takeaways

  • A number without provenance is incomplete. Source, period, unit, retrieval time, definition, transformation, and revision status should travel with important data.
  • "Official" does not mean "simple." Treasury curves are derived, economic releases can be revised, SEC XBRL facts require period and taxonomy interpretation, and retirement limits change by rule and year.
  • Current data and historical-vintage data solve different problems. Use the value known at the time when reconstructing a historical decision.
  • Missing data should stay missing. Replacing an unavailable observation with zero, a stale value, or an inferred figure changes the meaning of the analysis.
  • Transformations must be reversible or documented. If Swoopr calculates growth, a spread, a ratio, a real return, or an indexed series, the inputs and formula should be visible.
  • Reusable data should have a stable citation surface. A dataset should expose a canonical URL, data-as-of date, version, methodology, limitations, and downloadable format when licensing permits.
  • The library should connect evidence to education. Every dataset should link to the Swoopr guide or tool that explains how to use it.

What the Investment Data Library Is

The Investment Data Library is the evidence layer underneath Swoopr's guides, calculators, research notes, and reference pages.

It should answer five questions for any published data asset:

  1. What is this?
  2. Where did it come from?
  3. What does each field mean?
  4. What changed between the source and the displayed value?
  5. What would make this dataset stale or misleading?

That is a different job from the existing Stock Research Workbench. The Workbench organizes a research process around a company or fund. The Data Library organizes reusable evidence across many research questions.

Related Swoopr resources

The Seven Parts of a Defensible Dataset

1. Source

The source is the organization or system that produced the underlying record.

Examples:

  • SEC EDGAR for public-company filings;
  • U.S. Treasury for official daily Treasury rate series;
  • IRS for retirement-plan limits;
  • BLS for labor and inflation statistics;
  • Federal Reserve Bank of St. Louis for FRED and ALFRED distribution of economic series;
  • FINRA BrokerCheck for brokerage registration and disclosure lookup.

The name alone is not enough. Store the canonical source URL and, when possible, the specific endpoint or document used.

Example. Instead of recording just "Source: SEC," record the full reference:

SEC EDGAR XBRL Company Facts API
https://data.sec.gov/api/xbrl/companyfacts/CIK##########.json
retrieved 2026-08-30

The SEC documents that data.sec.gov provides JSON-formatted submissions and XBRL data and states the update schedule for those APIs.

2. Definition

A field name is not a definition.

"Revenue," "free cash flow," "yield," "inflation," "expense ratio," "spread," and "return" each have variants. A dataset should state the concept precisely enough that two people can calculate or retrieve the same thing.

Examples:

  • Is revenue GAAP revenue from the issuer filing?
  • Is free cash flow operating cash flow minus capital expenditures?
  • Is yield the Treasury par curve, bill investment rate, auction high rate, or a fund's SEC yield?
  • Is inflation month-over-month seasonally adjusted CPI, year-over-year CPI-U, core PCE, or something else?
  • Is an ETF expense ratio gross or net?

If a definition comes from a source methodology, link it. If Swoopr creates the definition, publish the formula in Swoopr Research Methodology.

3. Period and Frequency

Dates have multiple meanings. For investment data, the minimum useful metadata often includes:

  • observation date;
  • start and end of measurement period;
  • fiscal quarter or fiscal year;
  • publication and release date;
  • effective date;
  • retrieval date.

A quarterly financial fact should not be compared to an annual fact simply because both are labeled "revenue." A daily yield should not be treated as an average for the month unless that average is explicitly calculated.

The SEC warns developers to account for different reporting start and end dates when using XBRL frame data because companies do not all use the same fiscal calendar.

4. Vintage and Revision Status

Some data changes after first publication. That creates two legitimate questions:

  • What is the best current estimate?
  • What was known at the time?

FRED is widely used for current and historical economic series. ALFRED preserves historical vintages so researchers can reconstruct data as it existed on a past date.

The difference matters in backtests and historical analysis. If a strategy is tested on revised data that was unavailable when the hypothetical trade occurred, the backtest can accidentally use future information.

Swoopr already covers this issue in its point-in-time macro-data material. The Data Library should make vintage status a reusable metadata field rather than leaving that idea inside one macro article.

5. Transformation

A transformation is anything Swoopr does to the source value before publishing it. Examples:

  • currency conversion;
  • inflation adjustment;
  • index rebasing;
  • annualization;
  • rolling average;
  • percentage change;
  • spread between two series;
  • ratio of two filing facts;
  • winsorization;
  • missing-value handling;
  • date alignment.

Every transformation should answer:

  • formula;
  • input fields;
  • unit;
  • rounding rule;
  • missing-data rule;
  • date alignment rule.

Swoopr standard: If a reader cannot reproduce the transformation from the published method and source data, the transformation is not sufficiently documented.

6. Limitations

A limitations field protects users from giving the number more authority than it deserves. Examples:

  • preliminary data subject to revision;
  • issuer-defined non-GAAP metric;
  • stale filing period;
  • interpolated yield rather than observed transaction;
  • incomplete entity coverage;
  • proprietary labels that cannot be independently verified;
  • data affected by survivorship bias;
  • source unavailable for specific historical dates.

Limitations should be written in plain language, not buried in generic legal language.

7. Version and Change Log

A reusable research dataset should be versioned. At minimum publish:

  • version;
  • data-as-of date;
  • generated-at date;
  • last methodology change;
  • change summary;
  • superseded version link if retained.

A version number is especially useful when a dataset supports a Swoopr chart, research note, or downloadable file. It lets another writer cite the exact state they used.

The Swoopr Provenance Record

Every first-class dataset should expose a machine-readable record. An example of the required fields:

{
  "dataset_id": "treasury-par-yield-curve",
  "title": "U.S. Treasury Par Yield Curve",
  "publisher": "Swoopr Investment",
  "source_authority": "U.S. Department of the Treasury",
  "source_url": "https://home.treasury.gov/resource-center/data-chart-center/interest-rates/TextView?type=daily_treasury_yield_curve",
  "definition_url": "https://home.treasury.gov/policy-issues/financing-the-government/interest-rate-statistics",
  "data_as_of": "2026-08-28",
  "retrieved_at": "2026-08-30T00:00:00-04:00",
  "frequency": "business-day",
  "unit": "percent",
  "vintage_policy": "Treasury-published official series; methodology changes preserved",
  "transformations": [],
  "missing_value_policy": "preserve missing values as null",
  "version": "2026.08.30",
  "license_or_reuse_note": "Check source terms; Swoopr transformation metadata is published separately",
  "limitations": [
    "Par yields are derived from the Treasury par yield curve rather than a transaction in a single security."
  ]
}

The exact implementation can differ. The principle should not.

Core Data Collections

Collection A: SEC Filing and Company Fundamentals References

What belongs here:

  • filing index;
  • filing metadata;
  • XBRL company facts;
  • definition mapping;
  • derived metrics only when formula is public.

Primary sources from SEC EDGAR: EDGAR Search, EDGAR APIs, data.sec.gov.

Swoopr integration: SEC EDGAR Filing Search, Research Workbench, Fundamental Analysis hub.

What Swoopr should not do: Do not silently treat every XBRL tag with a similar name as comparable. Preserve the source taxonomy concept and period context.

Collection B: Treasury and Interest-Rate References

Treasury publishes daily official rate series and methodology. Primary sources: Interest Rate Statistics, Daily Treasury Rates.

Treasury explains that par yield curves are based on indicative bid-side market quotations and documents methodology changes and series gaps. Those notes should accompany Swoopr charts instead of disappearing during ingestion.

Candidate assets:

  • Treasury yield history explorer;
  • nominal versus real yield comparison;
  • curve-shape history;
  • maturity spread calculator;
  • methodology timeline.

Collection C: Economic Data and Historical Vintages

FRED and ALFRED support two different use cases: current and latest economic research; and point-in-time reconstruction.

Primary sources: FRED, FRED API, ALFRED.

Candidate assets:

  • recession and recovery data workbook;
  • inflation component history;
  • policy-rate and Treasury relationship explorer;
  • real-time versus revised series comparison.

Non-negotiable field: vintage_date. If a dataset can be revised, the vintage should be visible in any historical research that depends on what was known at the time.

Collection D: Retirement and Tax Reference Tables

The IRS publishes annually adjusted limits and current retirement-plan guidance. Primary sources: COLA Increases for Dollar Limitations, Retirement Topics: Contributions, 401(k) Contribution Limits, IRA Contribution Limits.

Swoopr already publishes a U.S. Investment Rule and Limit Tracker and a guide on how contribution limits are set and indexed. The Data Library should add structured historical data with a year field and source link, so articles do not need to hard-code independent copies.

Candidate assets:

  • retirement limit history dataset;
  • contribution-limit change timeline;
  • account-rule source index;
  • annual "what changed" diff.

Collection E: Registration, Enforcement, and Investor-Protection Data

Useful public systems include:

Swoopr already has investor-scam verification and enforcement education. The Data Library can add source indexes and structured metadata without turning allegations into rankings.

Editorial note: Registration, disclosure, allegation, settlement, judgment, and criminal conviction are different states. Preserve the legal status exactly.

Collection F: Swoopr-Owned Educational Data

Swoopr can also publish datasets it creates, as long as it clearly separates source data from Swoopr transformation or classification. Examples:

  • structured glossary export;
  • content knowledge graph;
  • formula registry;
  • tool registry;
  • research note index;
  • rule-source registry;
  • citation coverage status.

Swoopr's glossary methodology already documents a structured export generated from the same source records used to render the glossary. See Glossary Methodology.

Original, well-documented datasets give other publishers something concrete to cite. A large library of articles can demonstrate breadth; a versioned public dataset can demonstrate reusable evidence.

Missing Data Is a Result

The Data Library should adopt a strict rule: never convert "unknown" into zero.

This applies to:

  • missing filing facts;
  • unavailable historical rates;
  • undefined valuation ratios;
  • incomplete provider coverage;
  • absent API responses.

Zero is a value. Missing is a state. If a data point is not available, expose null, not_available, not_meaningful, or an explicit status field. The correct state depends on the reason.

Example. A P/E ratio for a company with negative earnings is not zero. It is generally not meaningful under the usual definition. A Treasury maturity that did not exist for a historical period should not be forward-filled merely to make a chart look complete. Treasury itself documents gaps and series breaks in its historical rate pages.

Dates: The Most Important Columns Nobody Sees

Every first-class Swoopr dataset should expose several dates instead of one vague "updated" timestamp.

observation_date
When the measured event or value belongs.
period_end
For financial statements or cumulative periods.
published_at
When the source released the information.
effective_date
When a rule begins to apply.
retrieved_at
When Swoopr fetched or verified the source.
data_as_of
Latest observation contained in the published Swoopr asset.
methodology_updated_at
When the transformation or definition changed.

These dates make stale data visible.

How Swoopr Should Handle Revisions

A revision should not silently erase history when the historical state matters. Use one of three patterns.

Pattern 1: Replace

Appropriate when the only relevant question is the current authoritative value. Example: a typo in a Swoopr field label.

Pattern 2: Version

Appropriate when researchers may need the prior state. Example: a Swoopr dataset generated monthly.

Pattern 3: Preserve Vintage

Appropriate when the old value was genuinely what users knew at the time. Example: macroeconomic releases. ALFRED exists specifically because historical vintages are analytically valuable.

Licensing and Reuse

Publicly accessible does not automatically mean "free to republish in any form." Every dataset record should include a reuse field such as:

  • public-domain government source;
  • source terms apply;
  • redistribution restricted;
  • Swoopr transformation licensed separately;
  • attribution requested;
  • download unavailable due to provider restrictions.

When licensing is unclear, Swoopr should link to the primary source rather than republish bulk data.

Citation-Ready Dataset Pages

Every Swoopr dataset page should contain:

  1. Title.
  2. Direct answer.
  3. Data-as-of date.
  4. Source authority.
  5. Source URL.
  6. Methodology.
  7. Field definitions.
  8. Revision policy.
  9. Missing-data policy.
  10. Limitations.
  11. Version and change log.
  12. Download links where permitted.
  13. Human-readable table or chart.
  14. Accessible alternative representation.
  15. "How to cite this dataset."
  16. Related guides and tools.

This format is deliberately useful to journalists, educators, researchers, and AI systems without being written specifically for any one retrieval engine.

DataCite emphasizes that data citation improves access, transparency, reproducibility, reuse, and credit. Swoopr does not need a DOI on day one to borrow the underlying discipline: make the dataset identifiable, attributable, dated, and citable.

Proposed Swoopr Data Catalog

Dataset Status Canonical source Update cadence
Investment Rule Source Register Existing / extend IRS, SEC, FINRA, Fed, DOL quarterly/event-driven
Retirement Contribution Limit History Build IRS annual
SEC Filing Index Existing tool / extend metadata SEC EDGAR continuous
Research Note Index Existing Swoopr primary-source research on publication
Glossary Structured Export Existing Swoopr on glossary release
Formula Registry Build from test/module catalog Swoopr methodology on formula change
Treasury Yield History Build U.S. Treasury business-day
Macro Vintage Examples Build FRED/ALFRED monthly
Investor Protection Source Index Build SEC, FINRA, CFTC, states monthly
Tool Registry & Methodology Index Build Swoopr on tool release

How This Helps Investors

The benefit is not "more data." It is less ambiguity.

A reader should be able to move from:

"I saw a number."

to:

"I know what this number means, where it came from, when it was true, what changed it, and what it does not prove."

That is the foundation of better research.

How This Helps Journalists and Educators

A citation-ready data page saves work. Instead of independently finding:

  • the original source;
  • the time period;
  • the definition;
  • the change history;
  • the downloadable file;

a publisher can see those elements in one place and still follow the link to the authority. Swoopr should not ask for a link merely because it assembled a page. It should earn the link by reducing verification work.

How This Helps Search and AI Visibility

Google's official guidance for generative AI search emphasizes valuable non-commodity content, technical crawlability, and normal SEO fundamentals. It specifically says there is no special AI schema or required content chunking.

A well-built data library fits that guidance naturally because it is:

  • original in organization;
  • useful beyond search traffic;
  • rich in verifiable entities;
  • internally connected;
  • updated on real evidence;
  • difficult to replace with generic generated prose.

The point is not to "hack citation extraction." The point is to publish something worth retrieving.

Frequently Asked Questions

What is data provenance in investing?

Data provenance is the record of where an investment data point came from, when it was obtained, how it was transformed, and which assumptions or limitations apply. Good provenance lets someone reproduce or audit the number instead of trusting a chart at face value.

What is the difference between FRED and ALFRED?

FRED is the Federal Reserve Bank of St. Louis platform for economic data. ALFRED preserves historical vintages, allowing researchers to see what a series looked like on an earlier date. That distinction matters when testing historical decisions because revised data may contain information that was not available at the time.

Are SEC API values automatically comparable across companies?

No. SEC structured data makes filings easier to retrieve, but analysts still need to consider taxonomy concepts, reporting periods, units, fiscal calendars, and company-specific presentation. A machine-readable fact is not automatically an economically comparable metric.

Why not just use a market-data provider?

Providers are useful, especially for standardized price and reference data. But material research should document the provider, field definition, latency, coverage, adjustment policy, and licensing. When a primary source is available for a consequential company or regulatory fact, Swoopr should preserve the link to that source.

Should Swoopr publish every dataset it uses?

No. Licensing, privacy, contractual restrictions, and usefulness matter. If Swoopr cannot legally redistribute a source dataset, it should publish methodology and source references rather than copying the data.

What is the difference between an update date and a data-as-of date?

The update date tells you when the Swoopr page changed. The data-as-of date tells you the latest observation included. They can be different and should not be conflated.

Related Swoopr Resources

References