Cohere Locks Enterprise AI Prompts Behind Hardware-Encrypted Model Vault

Cohere has opened a public beta for confidential computing on Model Vault, its dedicated managed inference platform. Requests remain encrypted until they reach a hardware-isolated environment spanning the CPU and GPU, reducing the ability of cloud or Cohere administrators to inspect prompts and responses during model execution.

The service combines Model Vault with NVIDIA H100 GPUs running in Microsoft Azure confidential virtual machines. According to Cohere’s launch post, applications can retain their existing Cohere APIs and request formats while adding client-side encryption and verification.

Plaintext stays inside the enclave

Cloud platforms routinely encrypt stored data and network traffic, but inference requires a model to process usable prompt data in memory. Azure’s confidential virtual machines extend a trusted execution environment across the VM and its attached H100 GPU, protecting that processing window with hardware-backed controls.

Attestation becomes the gate

Before an application sends a sensitive payload, customer-side software checks signed evidence from the confidential environment against an approved policy. A mismatch in the expected hardware, firmware, or software measurements causes the client to withhold the request.

Attestation adds a technical control to contractual commitments and provider access policies. Its assurance depends on the measurements a customer approves, the integrity of the verification client, and the process used to review software updates when those measurements change.

API compatibility, plus client work

Cohere says the model endpoints and request schemas remain unchanged, which limits changes to application logic. Developers still need to integrate the confidential-computing flow around those calls:

1. Create an encrypted vault through the beta console.
2. Obtain attestation evidence from the target environment.
3. Validate that evidence against the approved policy.
4. Encrypt and send the standard Cohere API request after successful verification.
5. Receive the protected response and decrypt it inside the customer environment.

This flow is most relevant to retrieval systems, agents, document search, and analysis pipelines that handle sensitive source material. Any component that records plaintext before encryption, including application logs, tracing systems, prompt stores, or retrieval middleware, remains outside the confidential boundary.

Protection stops at the enclave

The confidential boundary protects data during model execution and transport into that environment. Customers remain responsible for controls elsewhere in the system, including:

• Application authentication and authorization
• Prompt, response, and retrieval logging
• Endpoint and client security
• Data retention and residency policies
• Compliance reviews and audit evidence
• Model-output validation and abuse controls
• Protection against authorized users misusing data

Confidential computing also leaves model quality and output safety unchanged. It narrows the infrastructure threat model by restricting access to plaintext during inference.

H100 makes GPU isolation practical

NVIDIA’s H100 introduced hardware support for confidential GPU computing. Earlier trusted execution environments largely covered CPU workloads, leaving large generative models without the accelerator performance required for practical serving. Microsoft later made Azure H100 availability generally available, giving AI vendors a production infrastructure layer on which to build confidential inference services.

Model Vault already provides dedicated managed deployments for enterprises. Adding attested execution addresses a separate concern: infrastructure operators with privileged access should be unable to inspect customer data while the model runs.

The target workloads

Cohere is positioning the beta for data that security policies, contracts, or regulation have kept away from managed model endpoints. Examples include:

• Fraud investigations and financial records
• Privileged legal documents and due-diligence material
• Sensitive public-sector information
• Patent applications and unpublished research
• Proprietary engineering and architecture documents
• Private source-code search and analysis

Cohere’s place in the stack

Microsoft has published a broader Azure design for confidential AI inference and demonstrated workloads such as Whisper on the same class of confidential H100 virtual machines. Cohere packages those infrastructure primitives into its existing enterprise inference service, with standard Cohere endpoints and customer-side attestation.

The product’s dedicated deployment model aligns with Cohere’s focus on banks, telecommunications companies, governments, and other organizations that require stronger isolation and auditability than conventional managed endpoints provide.

Open questions for production

Cohere’s beta announcement leaves several deployment details for customers to establish during evaluation:

• Supported models, Azure regions, and data-residency options
• Latency and throughput overhead from encryption and attestation
• Pricing, quotas, service-level commitments, and support terms
• Attestation policy ownership and measurement-update procedures
• Key-management and recovery responsibilities
• Audit artifacts available for regulatory reviews

The strongest fit is managed inference over data whose policies require controls against provider and infrastructure access. The beta addresses confidentiality during model execution; application security, governance, operational performance, and cost remain separate production-review items.