Microsoft Builds Fake Companies Worth $300K in Savings to Test Copilot

Enterprise AI assistants are hardest to test on the long, domain-specific workflows that carry the greatest privacy risk, such as summarizing a cross-functional safety review or extracting action items from weeks of sprint meetings. Those workflows usually reside in customer tenants that privacy, contractual, and regulatory limits keep out of evaluation datasets. Microsoft Research described its solution in a research report: the Industry Evaluation Data Generation Platform.

The platform generates entire synthetic companies, giving Copilot teams repeatable test environments with fabricated employees, meetings, messages, documents, reporting structures, and project histories. Public business data shapes how each company operates, while automated checks screen the resulting content for quality and safety.

Whole companies as test fixtures

Each simulated workspace is a tenant, an isolated organizational environment with its own users, data, and configuration. The generator combines three public sources to model the company:

• O*NET: The U.S. Department of Labor’s occupational database calibrates roles, responsibilities, work frequency, and industry workflows such as clinical trial reviews and portfolio updates.
• Job postings: Online listings supply realistic job titles, reporting relationships, and departmental structures.
• SEC filings: Corporate disclosures inform company scale, business units, and financial context.

Time is part of the test environment, with meeting histories, project timelines, email threads, and organizational changes developing over weeks. That history supports time-aware evaluations such as asking what a team decided last Thursday or how a project changed after a review.

Microsoft has built tenants for three industries so far:

• Pharmaceuticals: Regulatory submissions, clinical trial discussions, and cross-functional safety reviews.
• Capital markets: Deal pipelines, compliance communications, and client portfolio reviews.
• IT and software: Sprint planning, code review discussions, and product launch coordination.

Queries that depend on history

A synthetic tenant becomes useful when its test questions refer to records and events that exist inside the environment. Microsoft uses a six-stage pipeline to transform generic workplace tasks into industry-specific queries grounded in each tenant:

1. Load: Import generic workplace task templates.
2. Filter: Remove unsuitable or redundant inputs.
3. Adapt: Align each task with the tenant’s industry and roles.
4. Synthesize: Generate queries tied to the tenant’s synthetic records and history.
5. Humanize: Rewrite the prompts to resemble natural workplace requests.
6. Validate and deduplicate: Apply quality and safety checks, then remove repeated queries.

Microsoft scores the environments and queries across four dimensions:

The gains Microsoft reports

Microsoft reports a 39% improvement over its initial generation approach on an internal composite metric that measures how closely the synthetic data reflects enterprise complexity and patterns. The 39% figure applies to synthetic-data quality under Microsoft’s metric; Copilot answer accuracy requires separate measurement.

Microsoft also says the platform is available through an internal shared catalog and serves as Copilot’s first standardized evaluation baseline. A common format and quality threshold allow product teams to compare releases using the same class of tenants and tasks.

How the pattern transfers

Teams building retrieval-augmented generation systems, copilots, or action-taking agents face the same constraint when customer environments contain sensitive or regulated data. Small handwritten test sets cover isolated examples while omitting long conversations, changing reporting structures, specialized terminology, and questions whose answers depend on time.

The reusable approach has four parts:

• Use public occupational and corporate data to define the organization’s structure.
• Generate connected artifacts and events across a coherent timeline.
• Derive evaluation queries from records that exist inside the synthetic tenant.
• Apply automated checks for fidelity, relevance, safety, and duplication.

Microsoft combines those components with large language model generation and safety controls aligned with its Responsible AI framework. The resulting benchmarks can be regenerated and shared without exposing customer information.

Representativeness remains a limitation because a generator can satisfy selected fidelity checks while omitting rare workflows, unusual permission structures, and adversarial behavior. Teams adopting the approach should define coverage requirements, publish metric weights, and review generated artifacts against the failure modes their products encounter.

An operational benchmark also needs reference answers or scoring rubrics, retrieval-permission tests, citation checks, and release-to-release versioning. Microsoft’s reported figures cover synthetic-data quality and generation costs, so each organization would still need to calibrate downstream product metrics for its own systems.

Agents require a live simulation

Copilot’s shift toward multi-step workflows creates a harder evaluation problem because an agent’s actions can change the environment it is being tested against. A realistic tenant may need to evolve over months or years, update records after each action, and expose the consequences of earlier decisions. That requires a stateful simulator capable of reacting to the agent, extending the platform beyond generated documents and fixed organizational histories.