What is the cost of running the CoE?

For a typical mid-large enterprise: 5 to 25 million annual cost (people plus platform plus tooling). The cost is significant; the leverage is real if scope is right.

Your AI Center of Excellence (When You Actually Need One)

Q: What about an AI Steering Committee in addition to the CoE?

Useful for cross-function strategic alignment. A steering committee shapes; an operational committee decides.

Q: How do we measure CoE effectiveness?

Time-to-production for new agents. Standards adoption rate. Platform reliability. Function satisfaction. Avoid measuring CoE output — that incentivizes over-centralization.

Q: Should we have multiple CoEs per business unit?

For very large enterprises (100K plus employees), often yes — federation by BU with a small enterprise team for cross-BU standards. Most enterprises don't need this.

TL;DR

Three things a CoE should do; three it shouldn’t:

Should:

Operate the shared platform (eval, audit, governance, model gateway).
Set the standards every agent in production must meet.
Stand up new agents for functions without AI engineering capacity.

Shouldn’t:

Approve every AI use case (becomes a bottleneck).
Own all AI delivery centrally (kills functional ownership).
Be the AI strategy team (strategy is exec-level, not CoE-level).

Most CoEs over-extend into “should not” territory and become bottlenecks within 12 months.

Three things a CoE should do; three it shouldn’t. Most CoEs become bottlenecks within 12 months because they take on the wrong scope.

The “AI Center of Excellence” is a popular structural answer to the AI question. Most CoEs in 2026 are 12–24 months old; many are at the bottleneck phase, where the original promise of acceleration has become a friction in delivery. This piece is the scope that works and the scope that doesn’t.

What a CoE should do

1. Operate the shared platform

The platform that every agent in production depends on:

Eval and audit infrastructure (the harness, the storage, the monitoring).
Model gateway (foundation model access, routing, cost management).
Governance tooling (policy enforcement, approval workflow, incident tracking).

This is shared infrastructure. The CoE owns it; functions consume it. Centralization is correct for shared infrastructure.

2. Set the standards

The standards every agent in production must meet (covered in The AI Governance Framework to Put in Place Before You Scale). The CoE writes the standards, maintains them, helps functions meet them.

The CoE is not the approval body for individual deployments — function leads approve their deployments against the standards. The CoE is the standards-keeper.

3. Stand up new agents for capacity-constrained functions

Functions that don’t have AI engineering capacity (HR, finance, smaller business units) get help from the CoE for initial agent builds. The CoE delivers; the function operates.

Over time, the function builds its own AI capacity and graduates from CoE delivery. The CoE moves to the next capacity-constrained function.

What a CoE should not do

1. Approve every AI use case

Pattern: every AI use case across the company goes through CoE review. Result: 6-week approval queue. Functions stop bringing use cases to the CoE because the friction is too high. Shadow AI grows.

The fix: function leads approve within their scope; CoE owns the standards but not every approval. Decision rights matter.

2. Own all AI delivery centrally

Pattern: the CoE is the only place AI gets built. Result: the CoE is overloaded; functions are dependent on a queue they can’t influence.

The fix: federated delivery. Functions have AI engineering capacity (or shared resources). The CoE supports and standards-keeps; doesn’t own all delivery.

3. Be the AI strategy team

Pattern: the CoE produces the AI strategy for the company. Result: strategy without exec ownership; recommendations that don’t get implemented.

The fix: AI strategy is exec-level (CTO, CEO, sometimes a CAIO). The CoE supports strategy execution; doesn’t own strategy itself.

When you need a CoE (and when you don’t)

CoE makes sense when:

Multiple business units or functions are deploying AI.
Shared infrastructure (eval, governance) creates leverage.
Standards need a keeper.
Capacity-constrained functions need help.

CoE doesn’t make sense when:

AI is concentrated in one product or function (let that team own it).
The company is small enough for direct coordination (under ~500 employees).
You’re using mostly bought AI products (less platform to share).

The CoE is a structure that creates leverage at scale. Smaller scale: just have an AI program lead.

Sizing the CoE

For a typical mid-large enterprise:

Year 1: 8–15 people. AI program lead, platform team (4–6), standards team (1–2), delivery support team (3–6).
Year 2–3: 15–30 people as platform matures and delivery support expands.
Year 4+: 25–50 people for large enterprises.

Headcount distribution: ~50% platform, ~25% standards/governance, ~25% delivery support. The exact mix shifts based on what the company needs.

Where the CoE reports

Three options:

1. Reports to CTO or CIO. Most common. Works when the CoE’s primary scope is platform and standards. Risk: AI gets framed as a tech function, not a business function.

2. Reports to CEO. Less common but works when AI is genuinely a board-level strategic priority. Tends to attract more funding and attention; risk of disconnection from execution.

3. Reports to a Chief AI Officer. Common when the CAIO role exists. Works when CAIO has real authority; doesn’t work when CAIO is figurehead.

For most enterprises in 2026, reporting to CTO with strong matrixed relationships to CFO, CISO, and General Counsel is the working pattern.

The transition from CoE to federated

Most successful AI programs evolve through stages:

Year 1–2: CoE-centric. CoE delivers most agents; functions consume.
Year 2–3: Hybrid. CoE owns platform and standards; functions own delivery within their domain.
Year 3+: Federated. CoE shrinks to platform team; functions own most delivery; AI capability is distributed across the org.

This is healthy evolution. Don’t fight it; plan for it.

What to do this quarter

Audit your CoE scope. Are you doing the three “should” things or drifting into the three “should not” things?
Adjust if drifting. Decision rights, ownership, scope — push back to functions where appropriate.
Plan the federation transition if you’re 18+ months in. Year 3 is when functions need to own more.
Re-evaluate CoE size. Most CoEs are right-sized for year 1; few are right-sized for year 3.

FAQ

Should the CoE include data science / ML legacy teams? Often yes; the skills overlap. But the operating model is different — modern AI work is closer to engineering and product than to research. Be intentional about the integration.

What about an AI Steering Committee in addition to the CoE? Useful for cross-function strategic alignment. Different from the operational committee (covered in The AI Ethics Board That Actually Works). A steering committee shapes; an operational committee decides.

How do we measure CoE effectiveness? Time-to-production for new agents (lower is better). Standards adoption rate (higher is better). Platform reliability (uptime, eval coverage). Function satisfaction surveys. Avoid measuring “CoE output” — that incentivizes over-centralization.

Should we have multiple CoEs (per business unit)? For very large enterprises (100K+ employees), often yes — federation by BU with a small enterprise team for cross-BU standards. Most enterprises don’t need this.

What’s the cost of running the CoE? For a typical mid-large enterprise: $5M–$25M annual cost (people + platform + tooling). The cost is significant; the leverage is real if scope is right.

Working with JAIN on AI CoE design? We help executive teams design the CoE that creates leverage without becoming a bottleneck. Book a 30-minute call.

Related reading: