Skip to main content

Data and AI Weekend 2026: Measuring AI Adoption

· 5 min read

Reflections from the conversation at Data and AI Weekend 2026, on the subject of measuring AI adoption.

In brief:

  • Measure business value, not minimum use
  • In addition to monetary cost, there may be such a thing as too much use
  • Conversations with the LLMs risk short-circuiting conversations with co-workers
  • Measure the human element — what we bring and share in an artificially intelligent workplace

Minimum Use and Business Value​

In the opening conversation, I was surprised to hear that many organizations have "AI adoption" goals that prioritize token-usage as a metric without giving sufficient thought to effective use of the tools. Leaderboards and spending do not provide any indication of whether the work is delivering real business value. Token usage is the new line count: a measure easily gamed and with near zero practical value.

On the other hand, several participants reported that many of their less tech-enthused coworkers are still reluctant to try these tools. Although a blunt instrument, forced practice could lead to greater comfort and recognition of possibilities.

Ungoverned and forced AI adoption can also lead to duplicated effort (multiple people solving the same problem without sharing) and inefficient solutions (such as probabilistic instructions that should be deterministic code). AI centers of excellence and mentoring can help recognize opportunities for coaching and for encouraging the sharing of prompts, skills, and code.

Output will never be a substitute for outcome when delivering business value. As technologists, we can help our colleagues and organizations adopt AI tools wisely, share and improve on insights and skills, and keep our eyes on the strategic performance measures that drive actual value rather than mere busywork.

Setting a Ceiling​

There is "value" to busywork, albeit in a different sense.

One of the dangers of accelerated effort with AI tools is the loss of downtime between tasks. Brains and muscles alike need recovery time between intense activities. Personally, I have noticed that my AI-assisted work days are leaving me more exhausted and less able to engage in other intellectual pursuits after hours. (Or, maybe that is simply a sign of getting older...)

Enlightened organizations could set a maximum bar on AI usage to help avoid burnout. And managers driving automation could recognize the benefit of encouraging periods of lower intensity work.

A ceiling on AI would combat cognitive apathy (letting the LLM decide everything, failing to question assumptions, throwing slop over the wall) and basic skills loss. Maybe we should force ourselves into temporary "AI free" periods (detox?) — with a manager's support, of course.

Measuring the Human Element​

Measuring widgets produced, sales leads converted, and profit earned is good. Absolutely critical, of course.

At the same time, we know that some parts of the job are intrinsically difficult to measure. Marketing, for example. There, one needs stories — human stories — showing how a customer was influenced, subtly or grossly, by an advertisement at the airport, a booth at a conference, or some other stratagem.

Share and record stories of experiments, successes and failures.

At my office, we have found value in the topical lunch and learn. For example, getting all the technologists together to learn and share about Skills, Hooks, and Agents for coding assistants.

Share prompts and skills. Measure that, not tokens.

At mid-year, my team added short-term performance goals around developing and sharing skills and agentic workflows. Setting explicit and specific performance goals made the priority crystal clear, without adding arbitrary units of measure. A next step will be to treat these as internal products, subject to feedback and continuous improvement. Importantly, these short-term goals were each designed to to support longer-term outcome-oriented goals.

Our conversations with the LLMs risk short-circuiting productive conversations with our co-workers. Active asset sharing and story-telling can help restore human connection and socialize best practices.

Back to Basics​

In our haste to adopt these tools, are we forgetting about simple before/after or A/B testing? When changing a process, take repeated measurement of a baseline, then change the process and take repeated measurements again. Do you see improvement?

That is always good advice, though many elements of creative work are not so repeatable and susceptible to this type of measurement.

Here, anecdote and story come into play again.

  • Has adoption of an AI assisted workflow unblocked work that seemed stuck, or full of friction?
  • Have the provisioned tools offered additional ideas, pushed back on something that needed push back, or forced you to sharpen your argument in favor of some choice?
  • Are tasks being completed in a more timely manner?
  • Is communication smoother?

And most importantly, are the core business goals advancing?

safnet logo