AI Literacy: What It Means in an Organization, and How to Measure It

AI literacy is the ability to use AI systems safely within stated rules, judge whether their output is sound in your own domain, and know when not to use them at all.

That definition is deliberately narrow. It leaves out how models are trained, the history of the field, and the ethics seminar. Those are interesting and they are not what stops an organization getting hurt.

Most AI literacy programmes fail because they define the term broadly and then teach the broad version to everybody. The result is a workforce that can explain what a large language model is and cannot tell you whether they are allowed to paste a customer record into one.

The five levels

A definition that names observable behaviour is usable. One that names knowledge is not, because you cannot audit what someone knows, only what they do.

Level	The person can	You can observe it by	Who needs it
Aware	State what the organization permits and prohibits, and who to ask when unsure	A five-question check on the actual policy	Everyone
Safe	Recognise that output arrives confidently wrong; knows what data must never be entered	They correctly flag the unsafe cases in their own work	Everyone
Productive	Uses approved tools for defined tasks and checks the output against the standard their domain already holds	Sampled work meets the same bar it met before	Anyone whose daily work changes
Supervising	Judges whether a colleague’s AI-assisted work is sound, and handles a reported problem	They have run at least one real escalation	Managers
Designing	Specifies where a system should and should not be used in their function	They have written a scoped proposal that survived review	Function leads

Two observations from applying this. Organizations aim everyone at Productive and leave Supervising empty, which is why rollouts stall at the manager layer. And the third column carries the weight: a framework with no observable evidence against each level is a vocabulary exercise that lets everyone declare themselves compliant.

The three things everyone actually needs

Whatever the tiering, the floor is the same, and it is short.

What you are permitted to do. Not principles. A list. These tools are approved, these are not, this data may go in, this may never, and here is who decides when it is not on the list. Most organizations have not written this, which is the single largest cause of both shadow usage and paralysis, often in the same building.

That output arrives confidently wrong. The failure mode with no signal attached. A wrong answer looks exactly like a right one, and the person reading it calibrates to the system usually being correct. People need to have seen this happen in their own domain, with an example they recognise, rather than have been told about it in the abstract.

What to do when something looks off. A route, a name, and permission to raise something you are not sure about. The most damaging cultural outcome of a bad programme is staff who noticed something and decided it was not worth mentioning.

Forty-five minutes to two hours covers all three. Anything longer for the general population is over-serving, and it teaches the organization that AI training is something to sit through.

What the EU AI Act obliges

Article 4 of the EU AI Act requires providers and deployers of AI systems to take measures ensuring a sufficient level of AI literacy among their staff and others operating these systems on their behalf, taking into account those people’s technical knowledge, experience, education and training, and the context the systems will be used in.

Three things follow from the wording.

It is an obligation of outcome rather than curriculum. The Act does not name a syllabus, a duration, or a certificate, and vendors claiming to sell an Article 4 compliant course are selling a course.

It is explicitly proportionate. “Taking into account” the person’s background and the context means a uniform programme for everybody is a poor fit by design. A developer building a high-risk system and a receptionist using a meeting summariser have different obligations attached to them.

It creates an evidential burden. You need to be able to show who was prepared, for what, when, and why that was sufficient for their role and the risk of the system they touch. A completion certificate held by everyone regardless of what they do is weak evidence. A role-based capability definition with dated evidence against it is strong.

The obligation bites where you place a system on the EU market or its output is used in the EU, which catches more organizations than expect it. Organizations with no EU exposure should treat this as forward planning rather than compliance, since other regimes are being drafted against the same reference. The United States Department of Labor has published its own AI literacy framing aimed at workforce development, which is worth reading for the vocabulary even where it carries no obligation for you.

What the evidence record looks like

The levels table and Article 4 both point at the same artefact and neither one describes it. Four fields, one row per person, held wherever your HR system already keeps role data.

The role as the organization defines it, rather than the job title. Two people with the same title and different systems in front of them need different rows.

The level required for that role, and the date somebody decided it. The decision matters more than the level, because the question an auditor asks is why this was sufficient for this person, and the answer lives in who judged it and against what.

The evidence that the person reached it, taken from the third column of the levels table rather than from a completion record. A dated note that they passed the policy check, ran a real escalation, or wrote a proposal that survived review.

The date it was last confirmed. Tooling moves, and a capability recorded eighteen months ago against a system you have since replaced is evidence of nothing.

Most organizations already hold three of these four fields somewhere and have never put them in one place. The work is assembling the row rather than collecting new data.

Why most programmes do not change behaviour

Four causes, and they are usually diagnosable in a fortnight.

The training ran ahead of the policy. People were taught capabilities the organization had not yet decided to permit. The careful ones stopped using the tool entirely, the incautious ones assumed anything demonstrated was allowed, and both outcomes trace back to sequence rather than content.

The content was generic where it had to be specific. Completion is high, satisfaction is fine, and nobody in claims can tell you what changed about handling a claim. Generic content produces generic recall, and the tier where specificity matters most is the tier that generates the value.

The manager layer was skipped. Content went to executives and to end users, and the layer that decides whether a tool is used properly got an email. A manager forms that judgment within about three weeks of rollout and their team follows it.

Nobody set a baseline. The programme may well have worked. Without a before number on the process it was meant to change, year two gets argued on anecdote, and anecdote loses to budget pressure.

Measuring it

Completion rates measure attendance. They are collected because they are easy, and they carry no information about whether the work changed.

What to instrument instead, decided before delivery rather than after: usage of the approved tools by function, which tells you whether people quietly went back to the old way. Quality sampling at a defined rate, checking whether AI-assisted work still meets the standard the function already had. Cycle time on the specific process the training targeted, with the baseline taken beforehand. The volume of questions and reported oddities in the first eight weeks, where silence is a warning rather than a success. And the shadow-tool signal, which tells you whether the approved path is slower than the unapproved one.

One discipline worth imposing: name a single operational metric per function before you start, and report it whether or not it moved. Programmes that report only their favourable numbers stop being believed around month nine.

What AI literacy is not

It is not knowing how the technology works. Useful for a handful of people, irrelevant to whether an underwriter uses a tool safely.

It is not prompt technique. Prompting is a skill with a short shelf life, and interfaces are absorbing most of it. A programme built around prompt craft dates within a year.

It is not a certificate. Certificates are evidence that someone attended something. They become meaningful only when the thing attended was defined against a role.

And it is not a one-off. Tooling changes, the policy changes, and the population turns over. The organizations that hold literacy are the ones that folded the baseline into onboarding and refresh the function-specific layer when the tooling moves, rather than running an annual campaign.

What each tier actually needs to be taught

The levels describe capability. What follows is the content that produces it, which is a different question and the one delivery briefs get wrong.

Everyone. The permitted-use list, walked through with real examples from your own organization rather than generic ones. One demonstration of a confident wrong answer in a domain the audience recognises. The escalation route, with the explicit statement that raising an uncertain concern is welcome. Forty-five minutes to two hours, and it belongs in onboarding rather than in a campaign.

Managers. Everything above, plus how to spot-check their team’s AI-assisted work without becoming a bottleneck, what to do with a reported problem, and what they are expected to say to their team about how the work changes. Two to four hours, delivered before their team’s session.

Function specialists. Taught against their real tooling and real cases. The centre of it is judging output against the standard their function already holds, which means the teaching material has to be built with a domain expert rather than bought. This is the expensive tier and the one that generates the return.

Function leads. How to scope a proposal: where in their process a system should and should not sit, what the human review has to catch, and what would make the case fail. The output of this tier is a written proposal that survives review, which is also the evidence that the capability is real.

Common mistakes

Teaching the technology. How a model is trained is interesting and does not change what anyone does on Tuesday. It is the default filler when nobody has decided what the audience needs to be able to do.

One programme for everyone. Over-serves the general population, under-serves the specialists, and costs the credibility you need for the tier that matters.

Buying before defining. A platform licence bought for the whole workforce commits you to that vendor’s tiering, which is nearly always two tiers where you need five.

Running it as an event. A campaign produces a spike in completion and a population that has turned over by half within eighteen months. Baseline content belongs in onboarding, and function content gets refreshed when the tooling changes rather than annually.

Measuring the wrong thing. Completion, satisfaction and self-assessed confidence all rise reliably and none of them predicts behaviour. Self-assessed confidence in particular tends to rise fastest in the people who are about to cause a problem.

Where to start

If you have nothing, write the permitted-use list first. One page. What is approved, what is prohibited, what data may go where, and who to ask. Every subsequent piece of training references it, and until it exists everything you teach is provisional.

Then train managers, before their teams. Then the function whose work changes most, taught against their real tooling and real cases.

Everyone else can wait, and should be folded into onboarding rather than run as an event.

The AI policy template covers the first step, and communicating AI to your workforce covers the part that determines whether any of it is believed. Where the programme has to span a whole workforce and the tiering argument is the hard part, that is a programme design job rather than a literacy baseline.