Evaluating an alternative data vendor reduces to a checklist of measurable properties — point-in-time delivery integrity, provenance and collection method, history depth, entity mapping quality, and out-of-sample incremental value over public data — with the industry standard on coverage assembled in practitioner literature and the Fintech/Open Data research programs through the 2010s: vendor claims are vendor claims, and the buyer's job is to test them against the vendor's own delivered data. UZU NEWS publishes information, not investment advice, and evaluates data as measurements, not as magic.
Alternative data — card transactions, satellite imagery, web-scraped prices, mobility, job postings — grew into a multi-billion-dollar industry on the promise of earlier, finer signals. Some of it is that. Much of it is conventional data with a story attached. The evaluation procedure is mechanical, and the mechanical version is the difference between an edge and an expense.
What is the checklist?
Seven questions, each with a test attached.
- Point-in-time integrity: when was each row actually available? Test: compare delivery timestamps against publication lags; reconstruct what a historical user could have known, and evaluate on that alone — backfill is the alternative-data version of lookahead, and backfilled history is the single most common killer.
- Provenance and collection method: where does the data physically come from, and did the sources consent? Test: demand methodology documentation specific enough to be checkable; vague provenance is disqualifying on legal grounds alone.
- History depth: how many observations does the coverage actually span? Test: count by entity and by date — panel density plots usually reveal sparse early years that the headline coverage figure conceals.
- Entity mapping: how are rows matched to tradable identifiers? Test: sample mappings by hand; mapping errors are silent performance killers in any panel analysis.
- Revision behavior: do delivered values change after the fact? Test: compare successive vintages of the same period; revised alternative data needs the same vintage discipline as government statistics.
- Overlap with public data: does it add anything measurable beyond public releases? Test: regress the vendor signal on public predictors; the residual is what you are buying.
- Incremental value out of sample: does the signal improve predictions in a properly temporal evaluation? Test: walk-forward, chronological, costs included — the same standards this site applies to any model.
Why does point-in-time integrity lead the list?
Because alternative datasets are marketed on history, and history is where backfill hides. A panel assembled last year, delivered with five years of reconstructed coverage, was not observable five years ago: the entity set is today's survivors and today's client list, the coverage density reflects current collection, and the reconstructed values inherit corrections. Evaluation on delivered history then measures an availability nobody had — survivorship and lookahead compounded. The vendor-level question is simple to ask: what fraction of the history was delivered in real time, and can the vendor evidence vintage snapshots? The honest answers are documented percentages; the evasive answers are the finding.
How do decay and crowding show up?
Two documented industry patterns complete the evaluation. Signal decay: alternative signals age — sources change collection, panels shift composition, and the predictive content measured on early history attenuates; the evaluation should measure performance by sub-period, and a signal whose edge lives entirely in its first years is an expiring asset, priced accordingly. Crowding: widely sold datasets are widely held; the documented industry concern is that identical signals concentrated in similar portfolios change the microstructure the signal once measured — the edge is competed toward the cost of exploiting it. Neither pattern invalidates the data; both belong in the value estimate, which is a forecast with stated conditions, not a fact.
What does the documentation package look like?
For a credible vendor, a methodological paper precise enough to be checkable: collection mechanics, entity resolution method, delivery lags, revision policy, vintage availability, coverage by period — plus sample data sufficient to run the checklist before contract. Vendors who decline sample access for testing, or whose methodology reads as marketing, have answered the evaluation anyway. The legal dimension is part of measurement: datasets assembled from sources that did not consent to resale carry regulatory and reputational risk that no backtest prices, and the sector's compliance documentation — privacy basis, licensing chain — belongs in the technical file, not as an afterthought.
| Checklist item | Test | Red flag |
|---|---|---|
| Point-in-time integrity | Vintage snapshots vs delivery dates | Backfilled history, no vintages |
| Provenance | Checkable methodology | Vague sourcing story |
| History depth | Panel density by date | Sparse early years hidden in averages |
| Entity mapping | Hand-sampled matches | Unverifiable identifier joins |
| Overlap with public data | Residual after public regressors | No residual analysis offered |
| Incremental value | Chronological out-of-sample | Full-sample correlations only |
What is the standing verdict?
Alternative data is a measurement purchase: the buyer is paying for observations with properties, and the properties — timeliness, coverage, integrity, exclusivity — are each testable with the vendor's own sample. The checklist's order is deliberate: integrity before value, because value measured on corrupted history is not value. The academic and practitioner literature on dataset evaluation formalizes these tests, and the model-risk parallel is exact — banks adopting vendor models for production face independent-validation requirements documented in supervisory guidance like the Federal Reserve's SR 11-7 letter at federalreserve.gov, and vendor data deserves no less before it touches a portfolio.
For more context, read How do you detect lookahead bias in someone else's backtest?.
For more context, read What does out-of-sample mean in a forecasting paper?.
For more context, read fed stress test 2026.
