Listen to this post: Audio Overview

What If Your Most Experienced Operator Could Train the AI?

AI-assisted procedure extraction lets you record an experienced operator performing a real production run, send that screen recording to a multimodal AI model, and generate a structured SOP from what actually happened rather than what someone remembers happening. For Life Sciences facilities where tribal knowledge disappears every time a senior operator leaves, this changes the economics of knowledge retention.

AI-assisted procedure extraction is the use of multimodal AI models to analyze screen recordings of operator workflows and generate structured, step-by-step documentation from observed behavior rather than recalled instruction. In GMP manufacturing environments, it matters because it captures tacit knowledge, including real alarm responses, fault handling, and HMI decision points, that never makes it into formally authored SOPs.

Every Life Sciences facility has the same undocumented problem. Operators who have run the same bioreactor process or filling line recipe hundreds of times carry knowledge that exists nowhere in writing. They know exactly what that amber warning on screen means at step 14. They know which alarm you acknowledge and which one you stop everything for. They know the three-second pause before the HMI updates and why you do not touch anything during it.

None of that is in the SOP.

It lives in their hands, their instincts, and the tribal knowledge passed down during shift handoffs. When that operator retires, takes a different role, or is simply unavailable during a deviation, that knowledge walks out the door with them.

How Multimodal AI Extracts Operator Knowledge from Screen Recordings

Mark Kashef published a workflow where you screen-record yourself performing a task, send that recording to Google Gemini for analysis, and use the resulting structured output to build a Claude Code skill that can replicate what you did.

The extraction step is what makes this worth paying attention to.

Gemini watches the video and documents what actually happened, including the hesitations, the corrections, the small decisions that never get written down. It generates a structured procedure from observation rather than from someone trying to articulate what they do from memory. Those two things produce very different outputs.

From there, the structured procedure becomes the foundation for a Claude Code skill that can execute that same workflow, or at minimum, guide someone else through it with the embedded context intact.

Mark applied this to content workflows and client onboarding in his demo. When I watched it, I immediately started thinking about what this looks like on a manufacturing floor.

Applying AI-Generated SOPs to SCADA, HMI, and Batch Record Workflows

Record an operator running a batch record through your SCADA or HMI system from start to finish, including the alarms that fire, the faults that appear, and the real-time decisions the operator makes to handle them. Send that recording to Gemini. Let it watch the whole process and extract a structured procedure.

What comes out the other side is not just a step list. It is a documented account of what actually happens during a real production run, with the complexity included.

You could use that output to:

• Generate a first draft SOP that reflects actual operator behavior rather than idealized procedure
• Build a troubleshooting guide based on real fault conditions observed during the recording
• Create training materials that show new operators what good execution actually looks like
• Identify gaps between what the written SOP says and what experienced operators actually do

That last one matters more than people admit. In a GMP environment, procedure adherence is everything. But if your SOPs do not reflect actual best practice because the tacit knowledge never got captured, you have a compliance exposure and a training gap running simultaneously.

For process validation and qualification work, this approach could also compress the time required to build protocol documentation. If you are qualifying a new piece of equipment or validating a new recipe, recording the development runs and extracting the AI-generated procedure gives your technical writers a detailed starting point that is grounded in what the process actually does.

Why AI-Captured Operator Procedures Strengthen GMP Documentation Defensibility

Here is how I see this applying directly to what we deal with in pharmaceutical and biotech manufacturing:

Record an operator interacting with the equipment or running a recipe from start to finish and create a full SOP of that product manufacturing operation from the HMI or SCADA screen, capturing all the difficulty, faults, and alarms that happen during a complete production run. Then use that to develop both SOPs and troubleshooting guides for the process inside a regulatory facility.

The reason this matters specifically for Life Sciences is that regulators do not just want to see that you have procedures. They want evidence that your procedures reflect how the process actually runs and that your staff can execute them. If you can demonstrate that your SOPs were built from captured real-world operator performance, that is a defensible documentation story.

This does not replace your validation process or your SME review. Nothing automated should ever go into a regulated document without qualified human review and approval. But it fundamentally changes what your starting point looks like and how much of the tacit knowledge problem you can solve before the review cycle begins.

How to Identify Which Processes to Target First for Operator Knowledge Capture

Pick one process that has a knowledge retention risk right now. An operator who is close to retirement, a recipe that only two people know well, a piece of equipment that generates frequent alarms with undocumented workarounds.

Record a full production run. Use Gemini to analyze it. See what comes out.

Compare that output to your current SOP for the same process. The gap between those two documents is your actual documentation risk.

The tools to close that gap are available now. The question is whether you use them before or after the knowledge walks out the door.

Frequently Asked Questions: AI-Assisted SOP Generation in GMP Manufacturing

Can AI-generated SOPs be used directly in a regulated GMP environment?

No. AI-generated procedure drafts must go through qualified SME review, formal approval, and change control before they can function as controlled GMP documents. What this workflow produces is a detailed, observation-based first draft that significantly reduces the time and effort required in the authoring phase. The human review and approval steps remain mandatory and non-negotiable under 21 CFR Part 11, EU Annex 11, and equivalent frameworks.

What types of manufacturing processes are best suited for screen-recording-based knowledge capture?

Any process where the operator interacts primarily through an HMI, SCADA interface, MES, or batch execution system is a strong candidate. Bioreactor control, filling line recipe execution, CIP sequence management, and environmental monitoring data review are practical starting points. Processes with high alarm frequency and undocumented operator workarounds are especially high value because those are exactly the scenarios where the gap between the written SOP and actual practice is widest.

How does this approach handle data integrity and 21 CFR Part 11 considerations for the recordings themselves?

The screen recordings used in this workflow are a knowledge capture tool, not a GMP record in the regulatory sense. They do not need to meet Part 11 requirements any more than a training video does. However, your data governance team should define how recordings are stored, who has access, and how long they are retained. If recordings capture any actual batch data or process parameters from a live production system, loop in your data integrity officer before proceeding.

What is the difference between what Gemini extracts from a recording versus what a traditional technical writer produces from an operator interview?

A technical writer working from an operator interview documents what the operator can articulate and remembers to say. Gemini watching a recording documents what the operator actually did, in sequence, including the micro-decisions, pauses, alarm acknowledgments, and corrections that operators perform automatically and rarely describe verbally. The observation-based output tends to be more complete, more sequentially accurate, and more reflective of real fault conditions than interview-based documentation.

Does this workflow require IT or vendor involvement to implement, or can an engineer run it independently?

In its basic form, this workflow requires only screen recording software, access to Google Gemini, and structured prompting to extract a usable procedure. An engineer can prototype this independently without IT infrastructure changes or vendor involvement. Before scaling it or using output to support any regulated documentation, involve your quality and IT teams to ensure the workflow aligns with your site’s data handling and document control requirements.

---