Cohere adds confidential GPU inference to Model Vault

Cohere has added an Encrypted mode to Model Vault, its dedicated, single-tenant service for hosting Cohere models. The new mode protects prompts, responses, model weights, and intermediate activations inside hardware-backed trusted execution environments spanning the CPU and GPU.

The limited beta targets organizations that need managed inference while keeping data hidden from Cohere operators, cloud administrators, and other infrastructure with privileged host access. Those protections depend on successful attestation, approved hardware and software measurements, and the limits of Cohere’s stated threat model.

Protecting data during computation

Managed inference services typically encrypt data while it travels over a network and while it remains in storage. Ordinary execution creates another exposure point because processors need usable data in memory. A privileged operating system, hypervisor, administrator, or compromised host can potentially inspect that memory.

Model Vault Encrypted runs its CPU workload in a confidential virtual machine. Technologies such as Intel TDX encrypt VM memory with processor-controlled keys that are unavailable to the host operating system and hypervisor. Data becomes usable only within the protected processor boundary as instructions execute.

The single-tenant vault isolates each customer’s deployment, while confidential computing addresses a separate risk: access by infrastructure administrators controlling the underlying machine.

The security boundary reaches the GPU

Large language model inference relies heavily on accelerators, so CPU memory protection covers only part of the request. Prompts, weights, and intermediate tensors also move through GPU memory and across the connection between the CPU and GPU.

Cohere uses NVIDIA data-center GPUs running in confidential-computing mode. An SPDM-based session authenticates the GPU and encrypts traffic between it and the confidential VM. SPDM, short for Security Protocol and Data Model, provides a standard way for hardware components to prove their identity and establish a protected connection.

This design keeps request data encrypted against the host while it moves between processors and confines decrypted values to the CPU and GPU security boundaries during computation.

Attestation gates every session

Remote attestation lets a client verify the deployment before releasing a prompt. Model Vault Encrypted combines evidence from the confidential CPU and NVIDIA GPU, with trust rooted in credentials provisioned by the hardware manufacturers.

The Cohere OHTTP proxy enforces the check in the following order:

1. Request signed attestation evidence from the deployment.
2. Validate the CPU and GPU against their manufacturer-backed trust chains.
3. Confirm that debug features are disabled.
4. Compare firmware and software measurements with an approved policy.
5. Verify that the session encryption key belongs to the attested environment.
6. Encrypt and transmit the request only after every check succeeds.

Attestation establishes that a deployment matched an approved configuration at verification time. It does not prove that the application contains no vulnerabilities or that the model will behave safely.

A proxy contains the integration work

Existing Cohere SDK applications can retain their API calls and use the OHTTP proxy as the protected transport layer. Cohere provides a standalone proxy for deployment in the customer’s environment and a Python package that wraps the client transport.

A locally deployed proxy verifies attestation and encrypts each request before sensitive data leaves the customer environment. Cohere also offers a hosted proxy option, whose trust boundary differs from local deployment and requires separate review for workloads that mandate client-side encryption.

Existing Model Vault customers retain the service’s isolated architecture and API surface, but Encrypted mode still requires a compatible vault deployment, confidential-computing hardware, attestation policy, and proxy configuration.

The trust boundary, mapped

Cohere’s design removes several infrastructure layers from the trusted computing base while continuing to rely on customer-controlled endpoints, measured workload code, hardware security mechanisms, firmware, and manufacturer trust roots.

Infrastructure observers may still learn request timing, payload sizes, traffic volume, and the model being called. Confidential computing also provides no availability guarantee; an operator unable to read a request can still delay, block, or terminate it.

Application-layer controls remain necessary for authorization, prompt injection, insecure tool use, model-output handling, audit policy, and data retention. Encrypted Vault addresses infrastructure access to data during inference rather than those broader model-security risks.

Beta access comes with constraints

Cohere is distributing the beta through a limited early-access program managed by its sales team. Encrypted deployments support a subset of the models available in standard Model Vault and require NVIDIA GPUs with confidential-computing capabilities.

Model choice, capacity, and regional availability are confirmed with Cohere. Public beta information does not specify a general-availability date, complete compatibility matrix, pricing, or expected latency and throughput overhead. Prospective users should resolve those details alongside attestation-policy updates, metadata retention, service-level commitments, and failure handling.

The Encrypted Vault documentation covers deployment and client setup.

Where encrypted inference fits

Workloads with sensitive prompts or strict operator-isolation requirements are the clearest candidates:

• Financial applications processing customer records, transaction data, or trading signals.
• Healthcare systems handling protected health information.
• Public-sector deployments requiring verifiable isolation from cloud operators.
• Legal, engineering, and corporate workflows containing contracts, source code, or acquisition material.

Model Vault Encrypted gives developers a managed path to protect inference data while it is actively processed. Its practical value will depend on supported models and regions, measured performance, operational requirements, and whether the documented threat model matches each organization’s security obligations.