# Market data Read the market data feeds added to a workspace from Python: find a feed, see its files and query them with DuckDB. The SDK handles the keys. > [!NOTE] > PrivateMind turns market data on one organisation at a time. If you have compute access and **Compute** has no **Market data** page, it probably isn't on for yours yet. Your organisation admin can ask your account team. The `privatemind` SDK reads market data feeds from inside a [workspace](https://docs.privatemind.com/workspaces.html). You add feeds your organisation holds to a workspace, and `pm.feed()` gets each one by name. You get its files as a table and its contents as a DuckDB relation, which you can filter, join and turn into pandas. Nothing is copied into the workspace. A query reads each file's footer, the metadata at the end of a Parquet file. It then fetches only the columns and row groups (blocks of rows) it needs, so one column of a multi-gigabyte tick file is a small read. ```python import privatemind as pm pm.feeds() # the feeds added to this workspace cta = pm.feed("cta") # by name, dataset id, or part of either cta.files("2026-06-26", slot="close") # that day's close files df = cta.read("2026-06-26", slot="close", columns=["Symbol", "Close"]).df() ``` The examples use a feed called Cta. Use a name from `pm.feeds()`, and `schema()` to see a feed's columns. Symbols keep the form the feed gives them, which for Cta is `IBM.`. Market data needs SDK 0.7.0 or newer, which the newest workspace images include along with DuckDB and its S3 extension. If `pm.feed` is missing, the workspace runs an older image: press **Restart to update** on its row. ## Add feeds to a workspace The **Market data** page under **Compute** lists the feeds your organisation holds. A workspace's owner or a compute admin adds the ones it needs under the workspace's **Settings**, **Market data**. A feed with a future start can be added once it starts. A feed you add appears in a running workspace within about a minute, with nothing to re-run. A workspace started before market data was set up for it needs one restart first, which closes its running sessions. ## Find a feed `pm.feeds()` lists the feeds added to this workspace. In a notebook it shows as a table, and `print()` gives the same as text. `pm.feed()` takes a feed's name as PrivateMind shows it, its dataset id, or part of either, as long as only one feed has that part. When nothing matches, the error lists the added feeds and names the closest one if one is close: ```python pm.feed("cta") # Cta (CTA TAPE A), while no other feed has "cta" in its name or id pm.feed("ctaa") # MarketDataError: no feed named 'ctaa' is added to this workspace. Did you mean 'Cta (CTA TAPE A)'? ... ``` A `Feed` has a `name`, a `dataset` id and a `path`: the `s3://` prefix its files sit under. ## List the files `files()` lists a feed's files, walking its folders for you: ```python feed.files(start=None, end=None, *, slot=None, tier=None, site=None) ``` Give `start` and `end` as days, both included, or as times, with `end` excluded: ```python cta.files("2026-06-22", "2026-06-26", slot="close") # five days of close files cta.files("2026-06-26 13:00", "2026-06-26 15:00", slot="tick") # the 13:00 and 14:00 tick files cta.files(slot="tick") # the most recent day that has tick files ``` `end` defaults to the end of `start`'s day. With no dates, `files()` lists the most recent day of each tier with matching files, searching the 31 most recent days that have data. Three filters narrow the list: | Filter | What it picks | |---|---| | `slot` | The kind of file: for example `tick`, `close` or `static` | | `tier` | `staging` or `gold` | | `site` | One site, on a feed delivered from several | Without `tier` or `site`, `files()` and `read()` include all of them, so a day held in both tiers comes back twice. Pass the filter to keep one. A time range picks the files whose times overlap it. File names use UTC, so a time with no zone is read as UTC and a `datetime` with a zone is converted to UTC: ```python from datetime import datetime, timedelta from zoneinfo import ZoneInfo start = datetime(2026, 6, 26, 9, 0, tzinfo=ZoneInfo("America/New_York")) cta.files(start, start + timedelta(hours=1), slot="tick") ``` A file whose name has no time is listed only for whole days. Like `pm.feeds()`, the list shows as a table, ending with the file count and total size. Past 60 files it lists only the first and last 20. It is still a plain Python list of `FeedFile` objects. Each has a `path`, `date`, `tier`, `site`, `slot`, `size` and `name`, plus `start` and `end`: the times in its name, in UTC. Each day is one listing, so a year of a feed takes some seconds. ## Read a feed ```python feed.read(start=None, end=None, *, slot=None, tier=None, site=None, columns=None, files=None) ``` `read()` takes the same days, times and filters as `files()`, or the files themselves with `files=`, and returns a DuckDB relation. Every row also carries the `date`, `tier` and `site` of the file it came from. `read()` lists the files and reads a footer for the column names. Rows are fetched only when you use the relation. `.df()` gives pandas, and `.pl()` gives polars once you've installed `polars` and `pyarrow`. `.filter()`, `.aggregate()` and `.order()` work before any rows are fetched: ```python hour = cta.read("2026-06-26 13:00", "2026-06-26 14:00", slot="tick", columns=["SystemTimestamp", "Symbol", "Bid"]) ibm = hour.filter("Symbol = 'IBM.'").order("SystemTimestamp").df() month = cta.read("2026-05-26", "2026-06-26", slot="close", columns=["SystemTimestamp", "Symbol", "CloseDate", "Close"]) closes = (month.filter("Symbol IN ('IBM.', 'JPM.') AND CloseDate IS NOT NULL") .aggregate("Symbol, CloseDate, arg_max(Close, SystemTimestamp) AS Close", "Symbol, CloseDate") .df()) ``` A few rules follow from how the files are stored: - **Times pick whole files.** For anything finer, filter on the feed's own timestamp column. - **Name your columns on large reads.** Tick files can have hundreds of columns, so once the files hold more than 1 GiB, `read()` asks for `columns`. `columns="*"` reads every one, and `cta.schema("tick").show()` lists them without fetching any data. - **One slot at a time.** Files of more than one slot are refused; pass `slot`. - **Text feeds come back one `line` per row.** Line endings are removed and a blank line becomes NULL. `columns` and `schema()` don't apply to them. - **Rows come in no set order.** Add `.order()` when order matters. ## Run your own SQL `pm.market_data.sql()` runs DuckDB on a connection that already holds the workspace's key. Paths from `files()` work directly. `feed.path` needs a glob, and a glob over a whole feed reads every file's footer. ```python newest = cta.files(slot="close")[-1] pm.market_data.sql(""" SELECT Symbol, Close, CloseCumulativeVolume AS Volume FROM read_parquet(?) WHERE Mic IS NULL AND Close IS NOT NULL ORDER BY Volume DESC NULLS LAST LIMIT 15 """, params=[newest.path]).df() ``` For longer work, `pm.market_data.connect()` returns such a connection. Each call returns a new one; use one per thread. `pm.market_data.configure(con)` gives the key to a DuckDB connection you opened yourself, and leaves any other S3 secrets on it working for other buckets. ## Stay within memory Connections from `sql()` and `connect()` share a quarter of the container's memory, and write any overflow to `/tmp`. A large scan slows down or stops with an error, and your Jupyter and terminals keep running. Footers need memory of their own, outside that limit. Large files have large footers: the footer of a 7 GB file takes about 800 MB to open. `read()` and `schema()` first check that the footers fit in the workspace's free memory, and refuse with the numbers if they don't. `sql()` makes no such check. Read large files an hour or a file at a time: ```python import pandas as pd cta.read("2026-06-26 14:00", "2026-06-26 15:00", slot="tick", columns=["Symbol", "Bid"]).df() parts = [cta.read(files=[f], columns=["Symbol", "Bid"]).filter("Symbol LIKE 'IBM%'").df() for f in cta.files("2026-06-26", slot="tick")] ibm = pd.concat(parts) ``` ## Check a workspace `pm.market_data.status()` says whether market data can be read here, and if not, why. It checks the mount, the feeds, DuckDB and its S3 extension, and makes one listing on the first feed. It also reports the key's expiry and DuckDB's memory limit. `.ready` is `True` when those checks found nothing, and `.problem` gives the first thing that would stop a query. A read can still be refused for memory. ```python pm.market_data.status() ``` ## Access and keys Each workspace has its own S3 key and its own list of feeds. PrivateMind mounts both into the workspace as read-only files and replaces the key before it expires. The SDK re-reads them on every call, which covers a `Feed` you kept in a notebook. A connection from `connect()` picks up the new key the next time you call `connect()`, `sql()` or `read()`. A connection you set up with `configure()` doesn't: call `configure()` on it again, because the old key stops working soon after it is replaced. The SDK gives the key to DuckDB as a temporary secret. It appears in no SQL text or query log, and is not written to disk. A key reaches only feeds your organisation holds, and only while it holds them. Within your organisation it isn't limited to one workspace. The SDK shows this workspace's feeds, and the key can also reach feeds added to your other workspaces. Anyone with a shell in the workspace can read the key. Every read made with it is recorded. ## Where it works Market data reaches interactive workspaces, and workspace jobs that run on the home volume of a workspace with feeds added. Training runs don't get it. Market data can't be read: - outside a workspace - in a workspace with no feeds added - in a workspace started before market data was set up for it In each case `pm.feeds()` returns an empty list, `pm.market_data.status()` reports the problem, and the other calls raise `MarketDataError` saying which. In a virtual environment in a workspace, install the SDK with `pip install 'privatemind[market-data]'` and keep the image's DuckDB version. A DuckDB of another version needs an S3 extension built for it, and workspaces can't download one. ## Where next - [SDK overview](https://docs.privatemind.com/sdk.html): authentication, configuration and the exception hierarchy. - [Workspaces](https://docs.privatemind.com/workspaces.html): where the SDK comes installed and set up. - [AtlasFeed MCP](https://docs.privatemind.com/atlasfeed-mcp.html): market data and feed metadata in chat.