Get API key

Market data

View as Markdown

Read the market data feeds added to a workspace from Python: find a feed, see its files and query them with DuckDB. The SDK handles the keys.

Note

PrivateMind turns market data on one organisation at a time. If you have compute access and Compute has no Market data page, it probably isn't on for yours yet. Your organisation admin can ask your account team.

The privatemind SDK reads market data feeds from inside a workspace. You add feeds your organisation holds to a workspace, and pm.feed() gets each one by name. You get its files as a table and its contents as a DuckDB relation, which you can filter, join and turn into pandas.

Nothing is copied into the workspace. A query reads each file's footer, the metadata at the end of a Parquet file. It then fetches only the columns and row groups (blocks of rows) it needs, so one column of a multi-gigabyte tick file is a small read.

Python
import privatemind as pm

pm.feeds()                                      # the feeds added to this workspace

cta = pm.feed("cta")                            # by name, dataset id, or part of either
cta.files("2026-06-26", slot="close")           # that day's close files

df = cta.read("2026-06-26", slot="close", columns=["Symbol", "Close"]).df()

The examples use a feed called Cta. Use a name from pm.feeds(), and schema() to see a feed's columns. Symbols keep the form the feed gives them, which for Cta is IBM..

Market data needs SDK 0.7.0 or newer, which the newest workspace images include along with DuckDB and its S3 extension. If pm.feed is missing, the workspace runs an older image: press Restart to update on its row.

Add feeds to a workspace

The Market data page under Compute lists the feeds your organisation holds. A workspace's owner or a compute admin adds the ones it needs under the workspace's Settings, Market data. A feed with a future start can be added once it starts.

A feed you add appears in a running workspace within about a minute, with nothing to re-run. A workspace started before market data was set up for it needs one restart first, which closes its running sessions.

Find a feed

pm.feeds() lists the feeds added to this workspace. In a notebook it shows as a table, and print() gives the same as text.

pm.feed() takes a feed's name as PrivateMind shows it, its dataset id, or part of either, as long as only one feed has that part. When nothing matches, the error lists the added feeds and names the closest one if one is close:

Python
pm.feed("cta")     # Cta (CTA TAPE A), while no other feed has "cta" in its name or id
pm.feed("ctaa")    # MarketDataError: no feed named 'ctaa' is added to this workspace. Did you mean 'Cta (CTA TAPE A)'? ...

A Feed has a name, a dataset id and a path: the s3:// prefix its files sit under.

List the files

files() lists a feed's files, walking its folders for you:

Python
feed.files(start=None, end=None, *, slot=None, tier=None, site=None)

Give start and end as days, both included, or as times, with end excluded:

Python
cta.files("2026-06-22", "2026-06-26", slot="close")             # five days of close files
cta.files("2026-06-26 13:00", "2026-06-26 15:00", slot="tick")  # the 13:00 and 14:00 tick files
cta.files(slot="tick")                                          # the most recent day that has tick files

end defaults to the end of start's day. With no dates, files() lists the most recent day of each tier with matching files, searching the 31 most recent days that have data.

Three filters narrow the list:

Filter What it picks
slot The kind of file: for example tick, close or static
tier staging or gold
site One site, on a feed delivered from several

Without tier or site, files() and read() include all of them, so a day held in both tiers comes back twice. Pass the filter to keep one.

A time range picks the files whose times overlap it. File names use UTC, so a time with no zone is read as UTC and a datetime with a zone is converted to UTC:

Python
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo

start = datetime(2026, 6, 26, 9, 0, tzinfo=ZoneInfo("America/New_York"))
cta.files(start, start + timedelta(hours=1), slot="tick")

A file whose name has no time is listed only for whole days.

Like pm.feeds(), the list shows as a table, ending with the file count and total size. Past 60 files it lists only the first and last 20. It is still a plain Python list of FeedFile objects. Each has a path, date, tier, site, slot, size and name, plus start and end: the times in its name, in UTC. Each day is one listing, so a year of a feed takes some seconds.

Read a feed

Python
feed.read(start=None, end=None, *, slot=None, tier=None, site=None, columns=None, files=None)

read() takes the same days, times and filters as files(), or the files themselves with files=, and returns a DuckDB relation. Every row also carries the date, tier and site of the file it came from.

read() lists the files and reads a footer for the column names. Rows are fetched only when you use the relation. .df() gives pandas, and .pl() gives polars once you've installed polars and pyarrow. .filter(), .aggregate() and .order() work before any rows are fetched:

Python
hour = cta.read("2026-06-26 13:00", "2026-06-26 14:00", slot="tick",
                columns=["SystemTimestamp", "Symbol", "Bid"])
ibm = hour.filter("Symbol = 'IBM.'").order("SystemTimestamp").df()

month = cta.read("2026-05-26", "2026-06-26", slot="close",
                 columns=["SystemTimestamp", "Symbol", "CloseDate", "Close"])
closes = (month.filter("Symbol IN ('IBM.', 'JPM.') AND CloseDate IS NOT NULL")
               .aggregate("Symbol, CloseDate, arg_max(Close, SystemTimestamp) AS Close", "Symbol, CloseDate")
               .df())

A few rules follow from how the files are stored:

  • Times pick whole files. For anything finer, filter on the feed's own timestamp column.
  • Name your columns on large reads. Tick files can have hundreds of columns, so once the files hold more than 1 GiB, read() asks for columns. columns="*" reads every one, and cta.schema("tick").show() lists them without fetching any data.
  • One slot at a time. Files of more than one slot are refused; pass slot.
  • Text feeds come back one line per row. Line endings are removed and a blank line becomes NULL. columns and schema() don't apply to them.
  • Rows come in no set order. Add .order() when order matters.

Run your own SQL

pm.market_data.sql() runs DuckDB on a connection that already holds the workspace's key. Paths from files() work directly. feed.path needs a glob, and a glob over a whole feed reads every file's footer.

Python
newest = cta.files(slot="close")[-1]

pm.market_data.sql("""
    SELECT Symbol, Close, CloseCumulativeVolume AS Volume
    FROM read_parquet(?)
    WHERE Mic IS NULL AND Close IS NOT NULL
    ORDER BY Volume DESC NULLS LAST
    LIMIT 15
""", params=[newest.path]).df()

For longer work, pm.market_data.connect() returns such a connection. Each call returns a new one; use one per thread. pm.market_data.configure(con) gives the key to a DuckDB connection you opened yourself, and leaves any other S3 secrets on it working for other buckets.

Stay within memory

Connections from sql() and connect() share a quarter of the container's memory, and write any overflow to /tmp. A large scan slows down or stops with an error, and your Jupyter and terminals keep running.

Footers need memory of their own, outside that limit. Large files have large footers: the footer of a 7 GB file takes about 800 MB to open. read() and schema() first check that the footers fit in the workspace's free memory, and refuse with the numbers if they don't. sql() makes no such check. Read large files an hour or a file at a time:

Python
import pandas as pd

cta.read("2026-06-26 14:00", "2026-06-26 15:00", slot="tick", columns=["Symbol", "Bid"]).df()

parts = [cta.read(files=[f], columns=["Symbol", "Bid"]).filter("Symbol LIKE 'IBM%'").df()
         for f in cta.files("2026-06-26", slot="tick")]
ibm = pd.concat(parts)

Check a workspace

pm.market_data.status() says whether market data can be read here, and if not, why. It checks the mount, the feeds, DuckDB and its S3 extension, and makes one listing on the first feed. It also reports the key's expiry and DuckDB's memory limit. .ready is True when those checks found nothing, and .problem gives the first thing that would stop a query. A read can still be refused for memory.

Python
pm.market_data.status()

Access and keys

Each workspace has its own S3 key and its own list of feeds. PrivateMind mounts both into the workspace as read-only files and replaces the key before it expires. The SDK re-reads them on every call, which covers a Feed you kept in a notebook. A connection from connect() picks up the new key the next time you call connect(), sql() or read(). A connection you set up with configure() doesn't: call configure() on it again, because the old key stops working soon after it is replaced.

The SDK gives the key to DuckDB as a temporary secret. It appears in no SQL text or query log, and is not written to disk.

A key reaches only feeds your organisation holds, and only while it holds them. Within your organisation it isn't limited to one workspace. The SDK shows this workspace's feeds, and the key can also reach feeds added to your other workspaces. Anyone with a shell in the workspace can read the key. Every read made with it is recorded.

Where it works

Market data reaches interactive workspaces, and workspace jobs that run on the home volume of a workspace with feeds added. Training runs don't get it.

Market data can't be read:

  • outside a workspace
  • in a workspace with no feeds added
  • in a workspace started before market data was set up for it

In each case pm.feeds() returns an empty list, pm.market_data.status() reports the problem, and the other calls raise MarketDataError saying which.

In a virtual environment in a workspace, install the SDK with pip install 'privatemind[market-data]' and keep the image's DuckDB version. A DuckDB of another version needs an S3 extension built for it, and workspaces can't download one.

Where next

  • SDK overview: authentication, configuration and the exception hierarchy.
  • Workspaces: where the SDK comes installed and set up.
  • AtlasFeed MCP: market data and feed metadata in chat.