AI Analytics

Technical writing

CFTC Commitments of Traders: The Federal Database Behind Futures Market Positioning

· 11 min read· AI Analytics
CFTCFuturesCommitments of TradersMarketsFederal Data

The Commitments of Traders report is the Commodity Futures Trading Commission's weekly X-ray of who holds the open positions in America's futures markets. Every Friday afternoon the agency publishes, market by market, how the prior Tuesday's open interest in crude oil, gold, corn, Treasuries, the E-mini S&P 500, the euro, and dozens of other contracts was split among the major categories of trader — commercial hedgers, swap dealers, managed-money funds, and small speculators. The dataset described here gathers roughly 98,728 market-week rows of those breakdowns: a long, short, and spread census of positioning that traders, economists, and regulators have leaned on for more than half a century to answer one deceptively simple question — not what the price is, but who is on each side of it.

What It Is and How Often It Comes Out

The Commitments of Traders report — universally abbreviated COT — is a weekly publication of the CFTC, the federal agency that regulates the US derivatives markets. Its purpose is disclosure of aggregate positioning: rather than reporting any individual firm's book, it sums the holdings of broad categories of market participant so that the public can see how the total open interest in a contract is distributed. The report has a long lineage. Position reporting in agricultural futures dates to the 1920s under the Grain Futures Act, the modern annual and then monthly COT figures appeared in the 1960s and 1970s, and since 2000 the report has been published on a fixed weekly schedule. Over those decades it has grown from a handful of farm commodities into a comprehensive cross-section of energy, metals, grains, softs, livestock, interest-rate, equity-index, currency, and now even cryptocurrency futures.

The single most important operational fact about the COT — the one that every new user must internalize — is its two-date structure. The positions in any given report are measured as of the close of business on Tuesday. The report itself is normally released the following Friday at 3:30 p.m. Eastern. When a federal holiday falls in the week, the release slips by a day. This means the data the market reads on Friday is already three days old, and by the time a trader acts on it the following week the snapshot can be the better part of a week stale. The lag is not a flaw to be fixed; it is baked into the design, the byproduct of collecting position data from every reporting firm, validating it, and assembling it into category totals. Any use of the COT has to account for the fact that the world has kept moving since the Tuesday the picture was taken.

The data arrives in two parallel forms for most markets: a futures-only report, which counts positions in the futures contract alone, and a futures-and-options-combined report, which folds in options positions converted to their futures-equivalent (delta-adjusted) size. The two can paint noticeably different pictures in markets where options activity is heavy, and mixing them across a time series is a classic source of error. The fut_only flag in the schema below marks which version a given row belongs to.

The Four Report Types

The CFTC does not publish a single COT report but a family of them, each carving the same open interest into a different set of trader categories. The categories matter enormously, because the entire analytical value of the data rests on the assumption that the buckets correspond to meaningfully different kinds of behavior. There are four report types.

Legacy. The original and simplest format, preserved for its decades-long continuity. It splits reportable positions into just two groups — commercial and non-commercial — and adds a residual non-reportable bucket for traders below the reporting threshold. The commercial category is meant to capture entities using futures to hedge a business exposure in the underlying physical or financial commodity: an oil producer, a grain elevator, a bank hedging a rate book. The non-commercial category is the catch-all for speculators — the large funds and traders with no offsetting physical business. The non-reportable bucket is inferred: it is whatever open interest is left after subtracting the reportable longs and shorts, and is treated as a proxy for the positions of small traders. The Legacy report's appeal is its unbroken history; its weakness is that the two-bucket split is crude, lumping genuine hedgers together with swap dealers and lumping every kind of speculator into one pile.

Disaggregated. Introduced in 2009 in the wake of the 2008 commodity price spike and the political debate over the role of financial speculation, the Disaggregated COT breaks the physical-commodity markets (energy, metals, agriculture) into four reportable categories that are far more informative:

Traders in Financial Futures (TFF). The financial analog to the Disaggregated report, also introduced in 2010, applied to interest-rate, equity-index, and currency futures — markets where “producer” and “processor” have no meaning. The TFF report splits traders into four financial categories:

Supplemental. A separate report covering a set of agricultural markets that breaks out a thirteenth category not visible elsewhere: index traders — positions associated with long-only commodity index investment products. The Supplemental report exists precisely to answer the “is index money distorting the market?” question, by isolating the passive, roll-driven flow that would otherwise be hidden inside the commercial or swap-dealer buckets.

How the CFTC Classifies Large Traders

The COT is built on the CFTC's large-trader reporting system, and understanding that plumbing explains both the power and the limits of the data. The agency sets, for each contract, a reporting threshold — a position size, expressed in number of contracts, above which a trader becomes “reportable.” The thresholds vary by market (they are larger for deep, liquid contracts and smaller for thin ones) and are set out in CFTC Regulation 15.03. When a firm's position in a contract crosses the threshold, clearing members and the trader file daily position reports, and the trader also files a Form 40 identifying the nature of its business. It is that Form 40 self-identification, reviewed by CFTC staff, that determines which category a trader is placed in.

Two consequences follow. First, the categories are assigned at the level of the trader, not the individual trade: a firm classified as a swap dealer has all of its positions in that market counted as swap-dealer positions, even if some of them are, in economic substance, directional speculation. The CFTC explicitly cautions that a trader's activity may not always fit neatly within a single category. Second, the reporting threshold creates the “non-reportable” residual: every trader below the threshold is invisible individually, and their aggregate position can only be inferred by subtracting all reportable longs and shorts from the total open interest. The reportable traders typically account for the large majority of open interest in the major contracts — often 70 to 90 percent — so the picture is far from complete only in the smallest, most retail-heavy markets.

Longs, Shorts, Spreads, and Open Interest

Every futures contract has a buyer and a seller, so total long positions and total short positions across all participants are, by construction, equal — that common figure is the market's open interest, the number of contracts outstanding (counted one side only). The COT decomposes that open interest by category and by direction. For each trader group the report states a long figure and a short figure, and for most categories a spreading figure as well.

Spreading deserves a word, because it is frequently misread. A spread position is one in which a trader holds offsetting long and short futures positions in the same commodity (for example, long the near-month contract and short a deferred month). The CFTC reports the matched portion of such positions in a separate spreading column rather than counting it as either a net long or a net short, because a calendar spread expresses a view on the shape of the forward curve rather than on the outright direction of price. For directional analysis, the quantity that matters is the net position of a category — its long contracts minus its short contracts — which strips the spreading out and leaves the directional bet. The single most-watched number in the entire dataset is the net position of the speculative category: managed money in physical commodities, leveraged funds in financials.

-- cftc_cot_weekly: ~98,728 market-week rows
-- One row per market (contract), per report type, per Tuesday report date.

report_date            DATE     -- the "as of" Tuesday the positions were measured
publish_date           DATE     -- the Friday the report was released (normally +3 days)
report_type            TEXT     -- "legacy", "disaggregated", "tff", "supplemental"
fut_only               BOOLEAN  -- TRUE = futures-only; FALSE = futures-and-options combined
market_name            TEXT     -- human contract name, e.g. "WTI CRUDE OIL - NYMEX"
cftc_contract_code     TEXT     -- 6-digit CFTC market code (stable key across weeks)
exchange               TEXT     -- listing exchange, e.g. "NYMEX", "CBT", "CME", "CFE"
open_interest          NUMERIC  -- total open interest (one side) for the market that week

-- Category positions vary by report_type. Disaggregated long/short columns:
prod_merc_long         NUMERIC  -- Producer/Merchant/Processor/User long
prod_merc_short        NUMERIC  -- Producer/Merchant/Processor/User short
swap_long              NUMERIC  -- Swap Dealer long
swap_short             NUMERIC  -- Swap Dealer short
swap_spread            NUMERIC  -- Swap Dealer spreading
managed_money_long     NUMERIC  -- Managed Money long
managed_money_short    NUMERIC  -- Managed Money short
managed_money_spread   NUMERIC  -- Managed Money spreading
other_rept_long        NUMERIC  -- Other Reportables long
other_rept_short       NUMERIC  -- Other Reportables short
other_rept_spread      NUMERIC  -- Other Reportables spreading
nonrept_long           NUMERIC  -- Non-Reportable (small trader) long
nonrept_short          NUMERIC  -- Non-Reportable (small trader) short

n_traders_total        INTEGER  -- count of reportable traders in the market that week

The Canonical Use: A Positioning and Sentiment Indicator

The reason the COT has a following far beyond regulators is that it is the only public, regular, market-wide measure of who is positioned how. Out of that has grown a whole tradition of using the report as a sentiment and contrarian indicator, organized around a few durable ideas.

Commercials as the “smart money.” The folk theory holds that commercial hedgers — the producers and processors who live in the physical market — have the best information about supply and demand, and that they hedge most aggressively precisely when prices are at unsustainable extremes. Because they are structurally net short (a producer hedges by selling futures against inventory it will deliver), the interesting signal is taken to be deviations from their normal posture: when commercials become unusuallyless short or even net long, the contrarian reading is that the people who know the commodity best think it is cheap.

Managed money as trend-followers. The mirror image is the speculative category. Managed-money funds and CTAs are, as a class, momentum-driven: they build large net long positions after a rally and large net short positions after a decline. The contrarian logic is that when this trend-following crowd has piled almost entirely onto one side — an extreme net long after a sustained run-up, say — there is little fresh buying power left, the marginal speculator has already bought, and the market is vulnerable to a reversal. Crowded positioning, in this view, is fuel for a snap-back when the trend stalls.

The COT index. Because the raw net-position numbers are not comparable across markets of different size or across eras of different open interest, practitioners normalize them into a COT index: the current net position rescaled to a 0–100 range against its own trailing window (commonly three years). A reading of 100 means the category is as net long as it has been in three years; a reading of 0 means as net short as it has been. The index turns “net managed-money longs are 250,000 contracts” — a figure with no intuitive scale — into “managed money is at a 97 on its three-year COT index,” which immediately reads as an extreme. The worked example later in this article builds exactly this index.

Extremes preceding reversals — and the heavy caveats. The seductive empirical claim is that positioning extremes tend to precede price reversals: when speculators are maximally long and commercials maximally short, a top is near, and vice versa. There is genuine signal in this — the academic literature on hedging pressure and on speculative positioning finds real, if modest, predictive content — but the naive version of COT trading is one of the most reliably disappointing strategies retail traders attempt. The problems are structural. Positioning can stay at an “extreme” for months while a trend grinds on, so an extreme is not a timing signal. Extremes are defined relative to a lookback window, so they drift as the window rolls. The category definitions shift composition over time. And a net position that looks extreme in contract count may be ordinary once open interest growth is accounted for. The honest summary is that the COT is a context and risk indicator — a way to know when the boat is crowded — not a mechanical buy-sell system, and every credible practitioner treats it as one input among many.

The Markets Covered

The COT spans essentially every actively traded US futures market, and the same schema applies across asset classes. A representative cross-section of the contracts that draw the most analytical attention:

What You Can Do With It

The combination of weekly cadence, multi-decade history, and consistent category structure makes the COT unusually amenable to systematic analysis. The most common workflows:

Python: Managed-Money Net and a 3-Year COT Index

The script below pulls the weekly Disaggregated (futures-only) COT records for a single market from the CFTC's public Socrata API, computes the managed-money net position for each week, and then builds the rolling three-year COT index that rescales that net to a 0–100 range against its own trailing window. The same approach works against the legacy CSV archives if a longer continuous history is needed, and against the TFF endpoint by swapping the resource ID and the category column names.

import io
import csv
import datetime
import requests
from collections import OrderedDict

# ---------------------------------------------------------------------------
# CFTC Commitments of Traders (COT) - Managed-Money Net & 3-Year COT Index
# Sources:
#   Socrata API (Disaggregated, futures-only):
#     https://publicreporting.cftc.gov/resource/72hh-3qpy.json
#   Legacy CSV archives (annual zips):
#     https://www.cftc.gov/files/dea/history/deacot{year}.zip
#
# Strategy:
#   1. Pull the weekly Disaggregated rows for one market by contract code.
#   2. Compute Managed Money net = managed_money_long - managed_money_short.
#   3. Build a rolling 3-year COT index that rescales the net position to
#      0..100 within its own trailing min/max window (the canonical extreme
#      detector). 0 = most short in 3 years, 100 = most long in 3 years.
# ---------------------------------------------------------------------------

SOCRATA = "https://publicreporting.cftc.gov/resource/72hh-3qpy.json"
HEADERS = {"User-Agent": "research@example.com (cot research project)"}

# CFTC 6-digit market codes (stable across weeks):
#   067651 = WTI Crude Oil (NYMEX)      088691 = Gold (COMEX)
#   023651 = Corn (CBOT)                005602 = Soybeans (CBOT)
#   13874A = E-mini S&P 500 (CME)       099741 = EUR/USD (CME)
CONTRACT = "067651"   # WTI crude oil; change to any market code


def fetch_weekly(contract_code, years=5):
    """Return weekly Disaggregated futures-only rows for one market, oldest first."""
    cutoff = (datetime.date.today() - datetime.timedelta(days=365 * years)).isoformat()
    params = {
        "cftc_contract_market_code": contract_code,
        "$where": "report_date_as_yyyy_mm_dd >= '" + cutoff + "T00:00:00'",
        "$order": "report_date_as_yyyy_mm_dd ASC",
        "$limit": "5000",
    }
    resp = requests.get(SOCRATA, params=params, headers=HEADERS, timeout=60)
    resp.raise_for_status()
    return resp.json()


def to_int(row, key):
    try:
        return int(float(row.get(key, 0) or 0))
    except (TypeError, ValueError):
        return 0


def managed_money_net(rows):
    """Build a list of (date, net, long, short) for the Managed Money category."""
    out = []
    for r in rows:
        date_s = (r.get("report_date_as_yyyy_mm_dd") or "")[:10]
        long_  = to_int(r, "m_money_positions_long_all")
        short_ = to_int(r, "m_money_positions_short_all")
        out.append((date_s, long_ - short_, long_, short_))
    return out


def cot_index(series, window=156):
    """Rescale each net value to 0..100 vs its trailing window (~3yr = 156 wk)."""
    nets = [pt[1] for pt in series]
    indexed = []
    for i in range(len(nets)):
        lo_i = max(0, i - window + 1)
        win  = nets[lo_i:i + 1]
        lo, hi = min(win), max(win)
        span = (hi - lo) or 1
        idx = 100.0 * (nets[i] - lo) / span
        indexed.append((series[i][0], nets[i], idx))
    return indexed


# -- Run ---------------------------------------------------------------------

print("Fetching COT (Disaggregated, futures-only) for code " + CONTRACT + " ...")
rows = fetch_weekly(CONTRACT, years=5)
print("  Retrieved " + str(len(rows)) + " weekly reports")

series  = managed_money_net(rows)
indexed = cot_index(series, window=156)

latest_date, latest_net, latest_idx = indexed[-1]
print("\nMarket: " + (rows[-1].get("market_and_exchange_names") or "?"))
print("Latest report date (Tuesday): " + latest_date)
print("Managed Money net position:    " + format(latest_net, ",d") + " contracts")
print("3-year COT index:              " + format(latest_idx, ".1f") + " / 100")

if latest_idx >= 90:
    print("  -> crowded LONG: positioning near a 3-year extreme")
elif latest_idx <= 10:
    print("  -> crowded SHORT: positioning near a 3-year extreme")
else:
    print("  -> mid-range positioning, no extreme")

print("\nLast 12 weeks (date  net  index):")
for d, net, idx in indexed[-12:]:
    bar = "#" * int(idx / 5)
    print("  " + d + "  " + format(net, "10,d") + "  " + format(idx, "5.1f") + "  " + bar)

A few implementation notes. The CFTC publishes the COT through the Socrata open-data platform at publicreporting.cftc.gov; each report type and futures/options variant has its own resource ID, and the column names differ by report type (the Disaggregated managed-money long field is m_money_positions_long_all, the legacy non-commercial field is named differently again), so a parser must be told which dataset it is reading. The 6-digit cftc_contract_market_code is the stable join key across weeks — contract names occasionally change wording, but the code does not — and it is the right field to filter on. The three-year window is conventionally 156 weeks; using a trailing window means the index for the earliest rows is computed against a shorter history, so a clean implementation either warms up the window or discards the first three years. As with any public-data API, requests should carry a descriptive User-Agent and respect the platform's rate limits; for full historical pulls, downloading the annual archive zips is gentler on the API than paging the whole history week by week.

Caveats

The Tuesday lag is structural. Positions are measured Tuesday and published Friday, so the freshest COT a trader can read already trails the market by three days, and by the next week the snapshot is nearly stale. The report is a description of where positioningwas, not where it is. It cannot be used as a real-time feed, and any backtest must be careful to align the report only with the prices available after its Friday release, never with the Tuesday close it describes.

Category reclassification muddies time series. Trader categories are assigned from Form 40 self-identification and CFTC staff review, and a given firm's classification can change — a trader may be reclassified, the population of swap dealers or managed-money funds in a market shifts over time, and the CFTC itself revised the whole category framework when it introduced the Disaggregated and TFF reports. A long managed-money net-position series is therefore not measuring a fixed set of traders; its composition drifts. Comparisons across many years should be made with that in mind, and the legacy two-category report, for all its crudeness, is the only series with truly unbroken definitions back to the 1980s.

Futures-only versus futures-and-options. The two report variants can diverge materially in options-heavy markets, and silently mixing them — pulling futures-only for some weeks and combined for others, or comparing one analyst's futures-only chart to another's combined chart — produces nonsense. A series must be built consistently from one variant, and the choice between them should be deliberate: combined captures the full delta-adjusted exposure, futures-only is cleaner but ignores the options book.

It is not a timing system. The deepest caveat is conceptual. Positioning extremes raise the probability of a reversal and flag elevated squeeze risk, but they do not date it — crowded can get more crowded, and an extreme can persist through an entire leg of a trend. The COT is a measure of crowding, hedging pressure, and risk concentration; it is genuine context that sharpens a thesis, not a standalone signal that generates entries and exits. Treated as the former it is one of the most useful public datasets in markets; treated as the latter it is a reliable way to be early and wrong.

Related writing

For the position-level counterpart in the fund world — the monthly all-asset holdings every mutual fund and ETF must disclose — see SEC N-PORT Mutual Fund Holdings: The Federal Database Behind Every Fund Portfolio Position.

For the equities analog to watching large speculative stakes — the disclosure any investor crossing 5% beneficial ownership must file — see SEC Schedule 13D Filings: The Federal Database Behind Activist Investor Stakes.

For the insider-level signal that, like COT positioning, is mined for predictive content — the real-time record of corporate insider buying and selling — see SEC Form 4 Insider Trading: The Federal Database Behind Corporate Insider Stock Transactions.