When several people use one of your organization's dedicated models at once, the model can serve only so many requests at a time and the rest wait. Fair share decides who goes first. It does not refuse requests or cut answers short: it changes only the order of the requests that are waiting. When the model has room to spare, it makes no difference.
Fair share is available on models deployed for your organization alone; a model's settings say if it cannot use it. Org admins set it up on the Inference tab of the organization settings.
How it decides
Fair share adds up each person's recent use of the model as the work it took to serve them: by default, the prompt tokens the model had to process plus the tokens it generated. Prompt tokens served from the model's cache are free. Older use counts for less: after one usage window it counts half, after two a quarter, and so on.
While the model is busy, the request from the person with the least recent use, relative to their weight, goes first. Someone with a weight of 200% is entitled to twice the share of someone at 100%. Over any busy stretch, everyone who keeps requests waiting gets a share in line with their weight.
Someone coming back after a quiet spell has little recent use, so without a limit they could stay ahead of everyone for a long time. The burst cap limits that: a returning person leads the busy users for about the cap, then takes turns like everyone else.
The method follows the virtual token counter in Fairness in Serving Large Language Models (Sheng et al., OSDI 2024), with older use fading over time.
Shadow and enforce
- Shadow (measure only) works out the order without applying it. Requests are served as before, and you can see what fair share would have done. Use it to check your settings against real traffic.
- Enforce applies the order.
Switch to Enforce at a quiet moment. Requests already waiting when you switch keep the place they had, so on a busy model they can wait behind newer requests for several minutes.
Settings
Leave a setting empty to use the default, which the form shows in the field.
| Setting | What it does |
|---|---|
| Mode | Weighted fair share orders requests as described above. Off serves requests in the order they arrive. |
| Enforcement | Shadow (measure only) or Enforce, as above. Inherit takes the platform default, shown under the field. |
| Usage window (minutes) | How quickly past use fades: after this long it counts half. A longer window remembers heavy use for longer. |
| Burst cap (minutes) | How long someone returning after a quiet spell can lead the busy users. 0 gives a returning person no head start at all. Weighted fair share only. |
| Input (uncached), Cached input, Output | How each kind of token counts towards a person's use. By default uncached prompt tokens and output tokens count the same and cached prompt tokens count nothing. |
User weights
Weights are set per person, as a percentage: 100% is normal, 200% gets twice the share and 50% half. A weight applies to every model on weighted fair share in your organization.
A change takes effect from that moment on. Past use isn't recounted, so a new weight never suddenly holds anyone back or lets them jump ahead.
An application key counts as one person. If many people work through one app, its key shares a single person's portion unless you give it a higher weight.
Watching it
If your organization has a PrivateMind monitoring portal, its Fair share dashboard shows the mode in use, each person's share of the work, how long their requests took, and how busy the model was.
What it does not do
- It does not limit how much anyone can use. For that, see Budgets & limits and Rate limits & budgets.
- It does not make a busy model faster. It changes the order of the requests that are waiting, not how many the model can serve.
- It does not apply to models shared with other organizations, where the order is managed for you.