# Kubernetes Reach your managed Kubernetes clusters with your PrivateMind account and decide who can use them. **PrivateMind Kubernetes Service (PMKS)** gives your organization managed Kubernetes clusters that you reach with your PrivateMind account. Your cluster's operations team builds and runs them under your contract. You decide who can use each cluster, and those people connect with `kubectl` and run your workloads on it. PMKS is separate from [workspaces](https://docs.privatemind.com/workspaces.html) and doesn't replace them. You get either a whole cluster or your own namespaces on a [shared cluster](https://docs.privatemind.com/kubernetes.html#shared-clusters) that other customers also use. You connect, give access and read the page the same way for both. > [!NOTE] > Clusters are set up for an organization by contract. **Compute → Kubernetes** appears once you can see a cluster. If it's missing, ask your org admin or your account team. Where to start: | To | Go to | | --- | --- | | Connect to a cluster | [Connect](https://docs.privatemind.com/kubernetes.html#connect) | | Fix a connection that fails | [When a cluster will not connect](https://docs.privatemind.com/kubernetes.html#when-a-cluster-will-not-connect) | | Give someone access | [Give people access](https://docs.privatemind.com/kubernetes.html#give-people-access) | | See how a cluster is doing | [What the page shows](https://docs.privatemind.com/kubernetes.html#what-the-page-shows) | | Work on a shared cluster | [Shared clusters](https://docs.privatemind.com/kubernetes.html#shared-clusters) | | Change access from a script | [Kubernetes API](https://docs.privatemind.com/kubernetes-api.html#common-tasks) | | Run `kubectl` from CI or another unattended job | [What PMKS does not do](https://docs.privatemind.com/kubernetes.html#what-pmks-does-not-do) | ## Connect Install the CLI once, then sign in and connect. You need: - An access entry on the cluster, from an org admin. - A way to reach the cluster. A cluster that [runs through PrivateMind](https://docs.privatemind.com/kubernetes.html#two-routes-to-a-cluster) needs only a connection to PrivateMind. Otherwise you need a network route to the cluster's API address, and your cluster's operations team can tell you how. The `address` step of [`pm kube doctor`](https://docs.privatemind.com/kubernetes.html#when-a-cluster-will-not-connect) checks that you can reach it. ### Install pm ```bash curl -fsSL https://privatemind.com/pm/install.sh | sh # macOS / Linux # Windows: irm https://privatemind.com/pm/install.ps1 | iex ``` Open a new terminal afterwards. You also need `kubectl`, which `pm` doesn't install. To update `pm`, run the installer again: it has no update command. `kubectl` runs the first `pm` on your `PATH`, so an older copy earlier on the path wins over a new one. [Installer details](https://docs.privatemind.com/kubernetes.html#installer-details) has the rest. ### Sign in and connect ```bash pm login # device code, approved in the browser where you're signed in pm kube ls # list the clusters you can use pm kube login # add one to your kubeconfig and switch to it kubectl get pods ``` `pm login` prints a link with the code in it, and the code alone for typing by hand, then tries to open your browser. Over SSH, on Linux with no display, or with `--no-browser`, you open the link yourself. The code lasts 10 minutes. Approve it in a browser where you're signed in; the mobile app can't. An access entry, your own or a team's, is live until its end date. `pm kube ls` lists the clusters where you hold a live entry, including any in maintenance, suspended or being set up: ```text NAME ROLE STATUS REGION NAMESPACES NOTE analytics edit active example-1 k7q2xm-main,k7q2xm-batch - research edit active example-1 all - ``` `NAMESPACES` reads `all` for a whole cluster and lists your namespaces on a shared one. A long list ends in `…`, and the page and the API show all of it. `pm kube login` takes a cluster name or a unique prefix, and an exact name wins over a prefix. It prints the context it set and the file it changed. It doesn't contact the cluster or fetch a [pass](https://docs.privatemind.com/kubernetes.html#how-a-connection-works), the short-lived token `kubectl` connects with, so your first `kubectl` command is the first test of the connection. On a whole cluster the context sets no namespace, so `kubectl` starts in `default`. Use `-n ` for another, and ask your cluster's operations team which namespaces you can use. On a shared cluster the context starts in your first namespace, and `pm kube login` prints the others. If `kubectl` answers `Forbidden`, see [Answers from the cluster](https://docs.privatemind.com/kubernetes.html#answers-from-the-cluster). ### Stay signed in To keep getting passes, run `pm login` every 12 hours, even though your sign-in lasts 7 days. Three time limits apply: | Limit | Length | When it runs out | | --- | --- | --- | | `pm` sign-in | 7 days | `pm` commands that need a sign-in stop working until you run `pm login`. Nothing refreshes it. | | Sign-in age for a new pass | 12 hours since `pm login` | PrivateMind refuses new passes until you run `pm login`. `pm kube ls` keeps working. | | Pass | 15 minutes | `pm` fetches a new one by itself. | `pm` keeps using the pass it holds, so the 12 hour refusal can arrive up to about 14 minutes late. `pm status` shows when your 7 day sign-in ends, but not the 12 hour limit. In the last 24 hours of the 7 days, `pm kube ls`, `pm kube login` and `pm kube doctor` print a warning; `kubectl` never does. `PM_EXPIRY_WARN_HOURS` sets that window, and `0` turns the warning off. ### Sign out `pm logout` signs you out on every device and deletes held passes, but leaves your kubeconfig alone. A pass already issued keeps working until it runs out, because the cluster checks only the pass. To sign out one device, find it with `pm sessions` and run `pm sessions revoke `. ### Your kubeconfig `pm kube login` writes a cluster, a user and a context, all named `pmks-` plus the cluster's name. They go in the file `KUBECONFIG` points at, or in `~/.kube/config` if that isn't set. It also sets `current-context`, so `kubectl` moves to the new cluster at once. Move back with `kubectl config use-context`. A cluster that runs through PrivateMind also gets entries for its [direct route](https://docs.privatemind.com/kubernetes.html#two-routes-to-a-cluster), named with `.direct` on the end, such as `pmks-research.direct`. The file holds no secret. The cluster entry has the API server address and, if one was registered, its CA certificate. The user entry tells `kubectl` to run `pm kube token ` for a pass, so `pm` has to be on your `PATH`. On a shared cluster you can switch to another of your namespaces with `kubectl config set-context --current --namespace `, and a later `pm kube login` keeps it. A namespace that isn't yours, or none, goes back to your first. When `KUBECONFIG` lists several files, `pm` uses the first that exists, or the last if none does. It follows a symlink, so the real file changes and the link stays. An existing file keeps its permissions; a new one is readable only by you. Other clusters, users, contexts and settings stay as they were. Run `pm kube login` again to pick up a new address or certificate. There is no `pm kube logout`: remove a cluster with `kubectl config delete-context`, `delete-cluster` and `delete-user`. If `pm kube login` can't add the cluster without changing what the rest of your file means, as can happen with YAML anchors or merge keys, it leaves the file unchanged and says so. Use **Download kubeconfig** on the cluster's **Connect** tab and point `KUBECONFIG` at that file. It's a standalone file with the same entries, such as `pmks-research.yaml`, and still needs `pm` installed and signed in. ### Installer details The installers need no sign-in. `pm` comes in five builds: Linux and macOS on amd64 and arm64, and Windows on amd64, which Windows on ARM runs through emulation. On macOS and Linux, `pm` installs to `~/.local/bin`, or to `PM_BIN_DIR` if set, and the installer adds that folder to `PATH` in your shell's startup file. On Windows it installs to `%LOCALAPPDATA%\pm` and adds the folder to your user `PATH`. Both installers also try to add the JetBrainsMono Nerd Font; if that fails, the install still succeeds. On macOS and Linux you can skip the `PATH` edit, the font, or both: ```bash curl -fsSL https://privatemind.com/pm/install.sh | PM_SKIP_PATH=1 PM_SKIP_FONT=1 sh ``` With `PM_SKIP_PATH` set, the installer prints the line for you to add. The Windows installer has neither option. ### Other kubeconfig tools Other tools that read a kubeconfig work the same way, as long as they run the exec credential plugin in `client.authentication.k8s.io/v1` and find `pm` on their own `PATH`, which can differ from your shell's. The helper never prompts, because the kubeconfig sets `interactiveMode` to `Never`. When a sign-in runs out, the tool shows a refusal and you run `pm login` yourself. ## How a connection works When `kubectl` needs to talk to a cluster, `pm` asks PrivateMind for a **pass**: a signed token that says who you are, which cluster it's for and your role. A pass lasts 15 minutes. `pm` holds it and fetches a new one when under a minute is left, so most `kubectl` commands don't ask PrivateMind for a pass. After a new `pm login`, `pm` drops the held pass and your next command gets a new one. A pass names one cluster. Another cluster refuses it, and on a shared cluster it reaches nothing in another customer's namespaces. The cluster checks the signature against PrivateMind's [public keys](https://docs.privatemind.com/kubernetes-api.html#passes-and-the-issuer) and sends PrivateMind nothing about you. PrivateMind stores no credential for your cluster. `pm login` also creates a key on your machine and ties your sign-in to it. A pass needs that tie, so a copied `pm` token file can't get a pass on its own. The token, the key and the held pass live in `~/.config/pm`, or in `PM_CONFIG_DIR` if set. Three parties share the work on a cluster: | Area | Your cluster's operations team | PrivateMind | Your organization | | --- | --- | --- | --- | | **Clusters** | Builds, upgrades and monitors them | Records which organization each one is for | Runs workloads on them | | **Access** | Sets the cluster to trust PrivateMind and maps roles to permissions | Signs you in and signs passes | Decides who gets access | The [Kubernetes API](https://docs.privatemind.com/kubernetes-api.html) lists your organization's clusters, downloads their kubeconfigs and changes access entries from a script. It gives no pass, so you always connect through `pm`. Everything inside a cluster goes through the cluster's own API server, which `kubectl` talks to. ### Two routes to a cluster Every cluster has a direct route to its own API address. A cluster can also run through PrivateMind, which lets its API server stay off the public internet. | | Direct | Through PrivateMind | | --- | --- | --- | | **Address** | The cluster's own | `https://privatemind.com/k8s/` | | **You need** | A network route to the cluster | A connection to PrivateMind | | **Context** | `pmks-`, or `pmks-.direct` when both routes exist | `pmks-` | | **Who sees your traffic** | You and the cluster | You, PrivateMind in transit, and the cluster | On a shared cluster the address carries the shared cluster's id. The `proxy_endpoint` field of the [Kubernetes API](https://docs.privatemind.com/kubernetes-api.html#cluster-fields) gives the full address. Through PrivateMind, PrivateMind checks the pass on each request and forwards the request with the same pass, and the cluster checks it again. So your role and the cluster's permissions work as on the direct route, and everything `kubectl` does works, including `exec`, `cp`, `port-forward`, `logs -f` and watches. PrivateMind decrypts this traffic in memory to forward it, including any Secrets you read and your `exec` sessions. It stores none of it, and logs one line per request: the cluster, you, the method, the status, the time taken and your address. It never logs a path, a header value or a body. To keep PrivateMind out of the path, use the direct route. Connections through PrivateMind close after 12 hours. Each person can hold 256 open requests at a time, and each organization 512. ### If PrivateMind is unavailable Nobody can get a new pass, so new `kubectl` commands fail once the current pass has under a minute left, within about 14 minutes. A cluster that runs through PrivateMind stops answering on that route at once. If your network reaches the cluster, switch with `kubectl config use-context pmks-.direct`, which works for as long as your pass lasts. The page, the **Access** tab and the Kubernetes API are unavailable too. Workloads already on the cluster keep running. A `could not sign a pass, try again` refusal stops new passes the same way. ## Shared clusters On a shared cluster your organization gets one or more namespaces on a cluster that other customers also use. It suits teams that don't need control of the whole cluster. ### What you get - **Your own namespaces.** PrivateMind names them from a random prefix and a short label chosen at setup, such as `k7q2xm-main`. They're listed under **Your namespaces** on the page, in `pm kube ls` and in the [Kubernetes API](https://docs.privatemind.com/kubernetes-api.html#cluster-fields). - **A quota in each namespace.** Your contract sets the CPU, memory, storage and pods, and the page shows how much you use. - **Defaults for every pod.** A pod that sets no CPU or memory gets a default request and limit, so the quota admits it. - **A closed network by default.** Pods in a namespace reach each other and DNS, and nothing else, until you change that namespace's network policy. - **Kubernetes' `restricted` Pod Security standard.** The cluster refuses a pod unless it runs as a non-root user, with no privilege escalation, all capabilities dropped and the default seccomp profile. ### How customers stay apart Every pass for a shared cluster carries a group and a username that name your cluster on it, such as `pm:pmks::edit` and `pm::`. The cluster's permissions are bound to those names in your namespaces only, so one customer's pass matches nothing in another's. Your cluster's operations team also keeps customers apart on the cluster itself, with network walls a customer can't remove, quotas, pod security and storage one customer can't read from another. ### What you can't do - Create, change or delete anything outside a namespace: nodes, custom resource definitions, operators, admission webhooks and storage classes. A Helm chart with such parts needs your cluster's operations team. - Create a namespace, or change your quota or limits. Ask your account team. - List anything across all namespaces. `kubectl get pods -A` answers `Forbidden`. - Create NodePort or LoadBalancer Services. - Use GPUs. They are on whole clusters only. If you need your own custom resources or operators, a whole cluster is the right fit. ### What other customers can see Another customer could learn the names and ports of your services. Every signed-in user of a Kubernetes cluster can see which APIs and custom resources it has installed, and any pod can look up service names through the cluster's DNS. ## Roles An access entry gives one of three roles. Each role is a group on the cluster, and your pass carries exactly one. The cluster's own permissions for that group decide what the role can do and which namespaces it reaches. PrivateMind only names the group: it holds no list of Kubernetes permissions. | Role | Group on a whole cluster | Group on a shared cluster | For | | --- | --- | --- | --- | | `admin` | `pmks:admin` | `pm:pmks::admin` | The strongest of the three. Your cluster's operations team can say what it allows. | | `edit` | `pmks:edit` | `pm:pmks::edit` | Creating and changing workloads | | `view` | `pmks:view` | `pm:pmks::view` | Reading workloads | Your cluster's operations team can also put `pm:` in front of the groups on a whole cluster. The `groups` field in the [Kubernetes API](https://docs.privatemind.com/kubernetes-api.html#cluster-fields) gives each cluster's exact strings. On a shared cluster you appear as `pm::`, the name an `admin` uses to bind a role to one person in their own namespace. To see what your role allows in a namespace, run `kubectl auth can-i --list -n `. The `permissions` line of `pm kube doctor` samples up to three permissions. ## Give people access Org admins give access on the cluster's **Access** tab or with the [Kubernetes API](https://docs.privatemind.com/kubernetes-api.html). An **access entry** names a person or a team in your organization, with a role and an optional end date. The person must already be in your organization, so a new hire [signs in to PrivateMind first](https://docs.privatemind.com/users-and-teams.html#how-users-get-into-the-org). Nobody has access by default, and a role in PrivateMind gives no role on a cluster. An org admin sees every cluster and manages who can use it, but must add an entry for themselves to connect. Until then the page shows them the cluster and `pm kube ls` doesn't. ### Teams A team's entry follows its members: someone added to the team can get a pass, and someone removed can't. If someone has more than one live entry, their strongest role applies, and when two give the same role their own entry counts for **Last connected**. A [service identity](https://docs.privatemind.com/remote-hosts.html#unattended-access-for-ci-service-identities) in a team still can't get a pass. Because a team's entry hands out access, only an org admin can add a member to a team that holds a cluster entry, even an expired one. If the team's own owner or admins try, they see `This team can use a Kubernetes cluster, so only an org admin can add members`. Teams appear in the **Access** tab's search only while **Member Sharing & Visibility** is on in your [org settings](https://docs.privatemind.com/org-settings.html). The API takes a team's id either way. ### End dates The page offers end dates from 1 hour to 30 days ahead. The API takes any time up to a year ahead. An expired entry stops working but stays on the list. An entry can't be added twice, so renew it by changing the existing row. ### When a change takes effect | Change | Takes effect | | --- | --- | | A role changes | At the person's next pass, within about 14 minutes, or at once after a new `pm login`. | | A team gains or loses a member | At that person's next pass, because PrivateMind reads membership when it issues a pass. | | An entry is removed or reaches its end date | New passes stop at once. A pass already issued can't be recalled and runs out within 15 minutes. | The remove button asks for no confirmation. Removing an entry changes nothing inside the cluster: workloads the person created stay. ### When someone leaves Someone who leaves your organization gets no pass, whatever entries remain. Those entries leave the list and you can no longer read or remove them, but they apply again if the person returns. So remove their entries before the person leaves, which also records the removal in your [audit log](https://docs.privatemind.com/kubernetes.html#audit). To do that across clusters from a script, see [Offboard someone](https://docs.privatemind.com/kubernetes-api.html#offboard-someone). ### Last connected **Last connected** is when that entry last produced a pass. `kubectl` reuses a pass until it's close to expiry, so this isn't a record of every command. A team's entry shows the latest pass by any member who got their role from it. The live view doesn't update it. ## What the page shows **Compute → Kubernetes** (`/compute/kubernetes`) lists your clusters with their status, region and your role. A shared cluster has a **Shared** badge with its number of namespaces. A cluster is **Active**, in **Maintenance** or **Suspended**, and a shared one can also be **Being set up** or **Being removed**: | Status | New passes | [Live view](https://docs.privatemind.com/kubernetes.html#the-live-view) | Download kubeconfig | | --- | --- | --- | --- | | **Active** | Issued | Shown | Available | | **Maintenance** | Issued | Shown | Available | | **Suspended** | Refused. A pass already issued runs out on its own. | Not shown | Available | | **Being set up** | Refused until your cluster's operations team has made your namespaces | Not shown | Not on the page. The API still serves it. | | **Being removed** | Refused. Its access entries are removed when it starts. | Not shown | Not on the page. The API still serves it. | PrivateMind staff set the status. On a shared cluster, maintenance or suspension of the shared cluster itself shows on yours too. A maintenance note shows on the page and in `pm kube ls`. You can change access entries in every status, except that a cluster being removed takes no new entry. A cluster's page shows its API address, region and your role, your namespaces on a shared cluster, and these tabs: | Tab | What it shows | | --- | --- | | **Overview** | **Needs attention**: nodes and pods with a problem, such as a crash loop, a failed image pull or a pod stuck pending. Also total CPU, memory and GPUs on a whole cluster, or your quota on a shared one. | | **Nodes** | Each node's status, roles, CPU, memory, GPUs, pod count and version. Whole clusters only. | | **Workloads** | Deployments, StatefulSets and DaemonSets, with how many replicas are ready. | | **Events** | Warning events from the last hour. | | **Connect** | The commands to connect, and **Download kubeconfig**. | | **Access** | Who can use the cluster. Org admins only. | ### The live view The **live view** is the first four tabs and the health figures above them. It's read-only and refreshes every 30 seconds. It appears when PrivateMind can read the cluster, and only for someone with a role on it. PrivateMind reuses a result briefly, so a change can take about 45 seconds to appear, and a cluster coming back about 90 seconds. PrivateMind reads the cluster with a short view-only pass in your name, so the cluster's own permissions apply. Everyone sees what the `view` role can see, whatever their own role, and a tab is hidden when the `view` role can't read what it needs. ### The live view on a shared cluster The live view reads only your namespaces: their pods, workloads, warning events and quotas. It never reads nodes or anything cluster-wide, so there's no **Nodes** tab, and **Quota** replaces **Capacity**. It reads up to 20 namespaces, and tells you when one didn't answer in full: `Some of your namespaces did not answer in full, so this is part of the picture.` **Quota** adds up your namespaces: CPU in cores, memory and storage in GiB, and pods, each shown as used out of its limit. For CPU, memory and storage, used means requested. When a namespace has several quotas for the same thing, the tightest counts. ### Health and Needs attention **Health** shows one of three things: - The number of issues, when anything is under **Needs attention**. - **Healthy**, when pods and nodes were both read and nothing needs attention. On a shared cluster it needs pods from every one of your namespaces, and nodes aren't read. - **Unknown** otherwise, so a cluster that doesn't let the `view` role list nodes never shows **Healthy**. With **Unknown**, **Needs attention** reads `This cluster did not answer everything, so its health is unknown.` The cluster may be fine: **Unknown** only means PrivateMind couldn't read everything. Your cluster's operations team can say why. **Needs attention** follows these rules, errors first: | Item | Level | Listed when | | --- | --- | --- | | Node | Error | It isn't Ready (`NotReady`). | | Node | Warning | It reports `MemoryPressure`, `DiskPressure` or `PIDPressure`. | | Pod | Error | A container is waiting on `CrashLoopBackOff`, `ImagePullBackOff`, `ErrImagePull`, `CreateContainerConfigError`, `CreateContainerError`, `InvalidImageName` or `RunContainerError`. | | Pod | Warning | A container exited non-zero in the last 10 minutes and the pod hasn't failed. | | Pod | Error | The pod failed, and it started in the last 24 hours. | | Pod | Warning | Pending for over 5 minutes: `Unschedulable` if it couldn't be scheduled, `Pending` otherwise. | A pod is judged by the first rule that fits. Init containers aren't judged, so a pod stuck on one shows as `Pending` after 5 minutes. A cordoned node isn't listed, though the **Nodes** tab shows it. **Explain** on a Needs attention item opens a new chat with a question about it already typed: the cluster, the item, its reason and its restarts. Nothing is sent until you send it. The chat gets no cluster access and no logs, but sending the question puts the cluster and object names in the chat. ### Pods and Capacity The **Pods** total includes finished pods, so fewer running than total can be normal. The view judges at most the first 1,000 pods on a whole cluster, and the first 1,000 in each namespace on a shared one. Past that, **Pods** shows the total with no running count, and the extra pods aren't judged. If the pod list is too large to read at all, the view drops it without a message: the **Pods** figure goes, no pod is judged and health can't show **Healthy**. On a shared cluster only that namespace's pods are dropped, and the view says it is partial. **Capacity** on the **Overview** tab adds up what the nodes have: CPU cores, memory and, where present, GPUs counted as `nvidia.com/gpu`. It shows neither free capacity nor usage. Allocated GPUs are the sum of container requests. ### When the live view is missing The page gives no message when the live view goes: the health figures and the first four tabs disappear, and **Connect** stays, with **Access** for org admins. Several causes look the same: your browser sign-in is over 12 hours old, PrivateMind can't sign the view-only pass, or it can't read the cluster. Sign out of PrivateMind in your browser and sign in again. If the view stays missing, check with your cluster's operations team. The view is also missing for an org admin with no entry, and for a cluster that is suspended, being set up or being removed. `kubectl` can work while the view is missing, because the two use different network paths. ### What stays private Only names, status, counts and the versions the cluster reports reach the page. It never shows environment variables, container images, ConfigMap contents, logs, commands, node and pod addresses, volumes, Secrets, labels or annotations. The two exceptions are labels: node role labels fill the **Roles** column, and the GPU model label fills the **GPU** column. A node with no role label reads `worker`. A pod has no row on the **Nodes** or **Workloads** tabs. It appears in a count, as a **Needs attention** item, or by name in **Events** when a warning event is about it. Events show their reason alone, except `FailedScheduling`, which keeps its text. ## Audit Your [audit log](https://docs.privatemind.com/audit-logs.html) records who got a pass and who changed access. The event names and fields are in [Audit events](https://docs.privatemind.com/kubernetes-api.html#audit-events), and the rows also reach the [Compliance API](https://docs.privatemind.com/audit-logs.html#compliance-api). | What happens | In your audit log | | --- | --- | | `pm` gets a pass | One row per pass, so a steady `kubectl` user leaves one about every 14 minutes. On a shared cluster the row lists your namespaces. | | An access entry changes on the **Access** tab or through the API | A row. | | A cluster's record is changed on your behalf | A row. On a shared cluster, a change to the shared cluster itself writes one row for each of your clusters on it, naming only that cluster. | | A cluster starts being removed | A row, with how many access entries were removed, if any. | | A team's members change | A team event such as `team_member_added`. | | A pass is refused | No pass row. A refusal by approved-network rules shows as `network_access_blocked`, and repeats from the same address are thinned out. | | Reads, kubeconfig downloads and the live view | No row. | | An entry reaches its end date | No row. | | A command runs inside a cluster | No row. PrivateMind has no record of it; that sits with the cluster's own logging. | ## What PMKS does not do - You can't create, resize, upgrade or delete a cluster yourself, in the page or the API. - There is no node pool management. - Access is per cluster. An entry can't be limited to a namespace, so on a shared cluster it covers all your namespaces. - A pass is for a person signed in with `pm`. API keys and service identities can't get one, and PMKS issues no other credential, so unattended jobs can't reach a cluster through PMKS. Ask your cluster's operations team. - The live view is in the web app only, not the API. - PMKS puts no PrivateMind identity or key in your pods. - The page has no shell, logs or usage figures, and can't act on anything in the cluster. - PrivateMind shows no cost figures for a cluster. Billing follows your contract. - PMKS doesn't cover what happens inside a cluster, such as storage, image pulls or exposing a service. Ask your cluster's operations team what yours offers. ## When a cluster will not connect `pm kube doctor` tries each step of a connection in order and marks it `ok` or `FAIL`: ```bash pm kube doctor ``` ```text ok sign-in https://privatemind.com ok cluster research: role edit, active ok kubeconfig /home/you/.kube/config ok address https://research.example.com:6443, Kubernetes ok pass valid until 14:32 ok identity the cluster sees you as , pmks:edit ok permissions list pods: yes, create deployments: yes, list nodes: yes ``` On a shared cluster three lines differ, and one is added: ```text ok kubeconfig /home/you/.kube/config, namespace k7q2xm-main ok identity the cluster sees you as pm::, pm:pmks::edit ok permissions in k7q2xm-main: list pods: yes, create deployments: yes ok isolation other customers' namespaces are closed to you ``` Later steps need earlier ones, so it stops when the next step can't run. It exits with status `1` if any step failed. | Step | What it checks | If it fails | | --- | --- | --- | | `sign-in` | `pm` can reach PrivateMind as you, which means only that your `pm` sign-in is valid. | Follow the message, usually `pm login`. | | `cluster` | You have a role on the cluster and it is active or in maintenance. | Check the name with `pm kube ls`, or ask an org admin for access. If it says `suspended` or `provisioning`, contact your account team. | | `path` | `pm` is on your `PATH`. Shown only when it isn't. It checks this shell's `PATH`, and a tool started another way can have a different one, so the tool can fail where your shell works. | Add it: `kubectl` can't get a pass without it. | | `kubeconfig` | Your kubeconfig has the cluster at the right address and, on a shared cluster, a context that starts in one of your namespaces. It ignores the current context. | Run `pm kube login `. If `kubectl` reaches the wrong cluster, check `kubectl config current-context`. | | `address` | Your machine reaches the cluster's API server and it presents the registered certificate. Any answer counts as `ok`, whatever its status. Through PrivateMind, this step runs after `pass`, uses the pass and needs a successful answer. | Check your VPN or network. If the message is about the certificate, ask your cluster's operations team. | | `pass` | PrivateMind gives you a pass. It reuses the held pass when it can, so `ok` doesn't prove PrivateMind would issue a new one, and the 12 hour limit or a role change can take up to about 14 minutes to show. | Follow the message; the tables below have the common ones. With `sign-in` at `ok`, it's usually the 12 hour rule, so run `pm login`. | | `identity` | The cluster accepts the pass, and shows who it sees you as. | Ask your cluster's operations team. | | `permissions` | Whether you can list pods, create deployments and list nodes. On a whole cluster it asks across all namespaces, so a role limited to some namespaces reads `no` here and still works inside them. On a shared cluster it asks in your context's namespace and skips nodes. `unknown` means the cluster gave no usable answer. | Never fails. | | `isolation` | Shared clusters only: the cluster refuses to list pods across all namespaces. | If it says the cluster lets you list pods in every namespace, it's set up wrongly. Report it to your account team. | The doctor runs from your machine, through any proxy your environment sets, and gives each call to the cluster 10 seconds. On a cluster with both routes it tests the route through PrivateMind, not the `.direct` one. It never tests the path PrivateMind uses to read the cluster for the live view. The tables below list the messages in the order of a connection. `kubectl` shows the same messages when it can't get a pass, `pm` commands print them after `pm: `, and `pm kube doctor` prints them as the detail on a step. ### Installing | Message | Meaning | What to do | | --- | --- | --- | | pm: unsupported arch <arch> | No build exists for this processor. | Use a machine with one of the five builds. | | pm: unsupported OS <os> - on Windows download https://privatemind.com/pm/pm-windows-amd64.exe | The macOS and Linux installer ran on another system. | On Windows, use the Windows installer. | | pm: unknown command "kube" | Your `pm` is out of date. | Install it again. If the message stays, an older `pm` is earlier on your `PATH`. | | Install the PrivateMind CLI, then run pm login. | `kubectl` can't find `pm`. | Install `pm`, or add it to your `PATH`. | | pm is not on your PATH, so kubectl cannot call it | The same, from the doctor. | Add it to your `PATH`. | | pm: usage: pm kube [ls \| login <cluster> \| doctor <cluster> \| token <cluster-id>] | Wrong arguments. `login`, `doctor` and `token` each take exactly one. | Fix the command. | ### Sign-in | Message | Meaning | What to do | | --- | --- | --- | | not logged in - run `pm login`, not authorized - run `pm login` again, not signed in - run `pm login`, pm: token expired <n> ago. Run `pm login`. | This machine has no sign-in, or PrivateMind refused it, usually because it ended or was revoked. | Run `pm login`. | | cluster access needs a recent sign-in - run `pm login` again | Your last `pm login` was over 12 hours ago. | Run `pm login`. | | cluster access needs a sign-in bound to this machine's key - check its clock, then run `pm login` again | Your sign-in isn't tied to this machine's key. | Check your machine's clock, then run `pm login`. | | pm: token expires in <n>. Run `pm login` to renew. | A warning: your 7 day sign-in ends soon. | Run `pm login` when it suits you. | | login was denied | Someone denied the sign-in in the browser. | Run `pm login` again and approve it. | | login expired - run `pm login` again, login timed out - run `pm login` again | The code ran out or can't be used again. | Run `pm login` again and approve within 10 minutes. | | could not start login: <err> | `pm` couldn't start a sign-in with PrivateMind. | Check your network, then try again. | | approved but no token returned | The approval went through but no sign-in came back. | Run `pm login` again. | If `not authorized` or `not signed in` comes back straight after a `pm login`, look for `PM_SERVICE_TOKEN` in your environment. It takes priority over your sign-in and can't get a pass, so unset it. If your organization limits PrivateMind to approved networks and this machine is outside them, `pm login` keeps waiting after you approve it. It ends in `login expired` when the 10 minutes run out. Connect to your organization's network or VPN first. ### Access | Message | Meaning | What to do | | --- | --- | --- | | no clusters have been shared with you | You hold no live entry on any cluster. An org admin with no entry sees this too. | Ask an org admin for an access entry. | | no cluster named "<name>" - run `pm kube ls` to list them | No cluster you can use has that name. | Check the name with `pm kube ls`. | | "<name>" matches <a>, <b> | That prefix fits more than one cluster. | Type more of the name. | | no access to that cluster | You hold no live entry on the cluster, or it is being removed, which removes every entry. `pm kube token` takes the cluster id, so a name there gets this answer too. | Ask an org admin for an access entry. | | that cluster is suspended | The cluster's status is **Suspended**. | Contact your account team. | | that cluster is still being set up | The cluster's status is **Being set up**. | Wait until it is active. | | This cluster does not exist, or you do not have access to it. | On the page: the link is wrong or you hold no entry, and it doesn't say which. | Ask an org admin for an access entry. | ### Kubeconfig file The messages in the first two rows follow ` was left unchanged: `, and `pm` writes nothing. | Message | Meaning | What to do | | --- | --- | --- | | it is not valid YAML: <err>, it holds more than one YAML document, it is not a kubeconfig, its <section> section is not a list, its <field> field is not a mapping | The file can't be parsed or isn't shaped like a kubeconfig. | Fix the file, or use **Download kubeconfig**. | | its YAML anchors or merge keys stop pm editing it safely, so add the cluster by hand | `pm` can't change the file without changing what it means. | Add the cluster by hand, or use **Download kubeconfig**. | | could not read <path>: <err>, could not write <path>: <err> | `pm` can't read or replace the file. | Check the file and its folder. | | <path> has no entry for it - run `pm kube login ` | From the doctor: the file has no entry for this cluster. | Run `pm kube login `. | | <path> points it at <server> - run `pm kube login ` | From the doctor: the entry has an old address. | Run `pm kube login `. | | pmks-<cluster> starts in <namespace>, which is not one of your namespaces - run `pm kube login ` | From the doctor, on a shared cluster: the context starts in `default` or a namespace that isn't yours, often because an older `pm` wrote it. | Run `pm kube login `. | ### Reaching the cluster | Message | Meaning | What to do | | --- | --- | --- | | the name does not resolve | Your machine can't look up the cluster's address. | Check your VPN or DNS. | | no answer - check your VPN or network | The cluster didn't answer in time. | Check your VPN or network. | | cannot connect - check your VPN or network | The connection to the cluster failed. | Check your VPN or network. | | it does not present the certificate registered for this cluster | The certificate doesn't verify against the registered one, or against your machine's trusted certificates when none is registered. | Ask your cluster's operations team. | | the certificate registered for this cluster cannot be read | The registered certificate is unusable. | Ask your cluster's operations team. | | PrivateMind refused the pass - run `pm login` | PrivateMind didn't accept the pass for this cluster. | Run `pm login`. | | this cluster is suspended | From `kubectl` through PrivateMind: the cluster is suspended. | Contact your account team. | | this cluster is not available | From `kubectl` through PrivateMind: the cluster is being set up or being removed. | Contact your account team. | | PrivateMind refused it: the cluster is suspended, being set up or being removed | From the doctor, through PrivateMind. | Contact your account team. | | this cluster is not available through PrivateMind | The cluster no longer runs through PrivateMind. | Run `pm kube login ` to return to the direct route. | | too many open requests, try again | You've reached the limit of open requests. | Close some watches or port-forwards, then try again. | | PrivateMind could not reach the cluster | PrivateMind accepted the pass but the cluster didn't answer. | Try again, then ask your cluster's operations team. | ### PrivateMind service | Message | Meaning | What to do | | --- | --- | --- | | could not sign a pass, try again | PrivateMind couldn't sign a pass just now. | Try again in about 30 seconds. | | cluster access is not available | PrivateMind can't issue passes here. | Contact your account team. | | Your organization only allows PrivateMind from its approved networks, and this request came from <ip>. Connect to your organization's network or VPN and try again. | This address is outside your organization's approved networks. A second wording says the address `could not be determined`. | Connect to your organization's network or VPN. | | Network access rules are loading. Please retry shortly. | PrivateMind hasn't loaded the approved-network rules yet. | Try again shortly. | | server error <status> | PrivateMind answered with an error and no message. | Try again, then contact your account team. | | A raw network error | `pm` can't reach PrivateMind. | Check your network and proxy settings. | | pm: could not cache the cluster pass: <err> | A warning: `pm` got a pass but couldn't save it, and the command carries on. | Check that you can write to `pm`'s config folder. | | Could not load your clusters. Try again in a moment. | On the page: PrivateMind didn't answer. | Try again in a moment. | | Could not load who has access. Try again in a moment. | On the **Access** tab: PrivateMind didn't answer. | Try again in a moment. | Rules that limit PrivateMind to approved networks cover `pm kube ls` and new passes. They don't apply to `kubectl` traffic to the cluster, on either route. ### Answers from the cluster These come after a pass, so PrivateMind's part has worked. | Message | Meaning | What to do | | --- | --- | --- | | `Forbidden` from `kubectl` | The cluster knows who you are, and your role doesn't allow that action in that namespace. | Check the namespace, then ask an org admin for a stronger role. A new role reaches you within about 14 minutes, or at once after `pm login`. Your cluster's operations team can say what each role allows. | | `Unauthorized` from `kubectl`. From the doctor: the cluster refused the pass, so it does not trust this PrivateMind yet - contact support | The cluster didn't accept the pass. | Ask your cluster's operations team. | | the cluster answered <status> | The cluster gave the doctor an answer it can't use. | Ask your cluster's operations team. | | could not ask the cluster who you are: <reason> | The doctor reached the cluster earlier and lost it at this step. | Check your VPN or network, then run the doctor again. | | `identity` at `FAIL` with no group after your name | The cluster accepted the pass and shows no `pmks:` or `pm:pmks:` group for you. | Ask your cluster's operations team. | | this cluster lets you list pods in every namespace, which it must not - report it to support | From the doctor, on a shared cluster: your role reaches beyond your namespaces. | Report it to your account team. | | could not check that other customers' namespaces are closed to you | From the doctor, on a shared cluster: the cluster gave no usable answer. | Run the doctor again, then ask your cluster's operations team. |