Human In The Loop and Cognitive Load Fatigue

Why the humans inside our AI systems are the most fragile component, and how to design around it.

This story was originally published on my substack at “anisaknouche.substack.com”. The link to the substack article: https://anisaknouche.substack.com/p/human-in-the-loop-and-cognitive-load

1. Introduction

Considerable engineering effort goes into optimizing AI models with better architectures, more data, and faster inference. Yet most production AI systems still contain a human somewhere in the pipeline, labeling, reviewing, approving, and correcting. That human operates under constraints no GPU shares, such as limited working memory, fatigue, bias, and declining attention over time.

This article proceeds in four movements. It first defines Human in the Loop (HITL) and examines its structural limitations. It then introduces Human on the Loop (HOTL) as the supervisory paradigm that emerges as systems scale and become more autonomous. Third, it turns to the cognitive science underlying both paradigms, cognitive load and task fatigue, and derives design principles for reducing cognitive load fatigue in human-computer interaction (HCI) systems. Finally, it grounds these principles in an experimental case study on how the presentation of AI recommendations shapes employees’ perceptions, behavior, and decision quality.

2. Human in the Loop: Definition and Contributions

Human In The Loop refers to approaches that integrate human knowledge and experience into the machine learning pipeline, with the goal of training accurate models at minimum cost while covering the cases machines struggle with [1]. In operational terms, humans are involved at defined decision points for specified classes of cases: a human reviewer approves, corrects, or rejects an output before it is accepted downstream [1, 2].

Humans contribute to AI systems in three essential ways:

• Labeling. Annotators provide annotations and ground truth for training data [1, 2]. This is the stage where data annotation platforms deliver most of their value, with consistent guidelines, quality controls, and routing of tasks to appropriate expertise.
• Tuning. Humans evaluate and score model outputs to improve performance. The feedback mechanism underpinning modern generative AI, notably reinforcement learning from human feedback [3].
• Validating. Humans verify decisions before they impact downstream processes. An analyst confirming a fraud flag, a radiologist confirming a triage suggestion [2].

3. Challenges and Limitations of HITL

The core tension of HITL is that the human is simultaneously the most trustworthy component of the system and the least scalable one.

Cost and scalability. Supervision is expensive, and the expense compounds along two independent axes, 1) the volume of labels needed and 2) the expertise each label requires. Huyen [3] illustrates both with a single example. On the volume axis, at roughly 5 cents per label, annotating one million images for ImageNet costs about $50,000, doubled if each image is labeled by two annotators for cross-checking, and since the world contains vastly more than ImageNet’s 1,000 object categories, expanding coverage to one million categories would push labeling costs alone to around $50 million, a thousandfold increase driven purely by scale. On the expertise axis, the curve is steeper still. Labeling everyday objects requires no prior training and remains cheap, but producing Latin translations for an English-to-Latin model demands scarce linguistic expertise, and having qualified physicians label whether Computed Tomography (CT) scans show signs of cancer is, in Huyen’s words, astronomical [3]. These two axes interact with a structural constraint. Human throughput is bounded and scales linearly at best, while data volumes and model capacity scale much faster. Annotation and review therefore become the pipeline’s bottleneck, route too much work to humans and latency accumulates, route too little and quality control erodes. The design question is consequently twofold. Economically, whether the cost of errors exceeds the cost of oversight; operationally, how to balance speed against accuracy requirements [1]. Continuous feedback, finally, means neither question is answered once, it recurs for as long as the system runs.

Human factors. Annotator fatigue and the accompanying decline in accuracy over a work session are well documented (see Section 5). Human judgment carries biases. And the training and calibration effort required to obtain consistent labels across a team is underestimated.

Mitigation strategies. The standard playbook combines automation and quality engineering. Automate routine checks so humans concentrate on genuinely ambiguous or high-stakes cases, monitor quality through inter-annotator agreement, rotate annotators to counteract fatigue, and maintain clear guidelines with regular calibration sessions [2]. Platform solutions reduce overhead by standardizing workflows, quality assurance, and expert routing across projects.

4. From Human In the Loop to Human On the Loop

Human In The Loop (HITL) and Human On The Loop (HOTL) describe different placements of the human within an AI workflow [4]:

• Human in the loop. The human occupies a mandatory checkpoint, and nothing is accepted without approval, correction, or rejection.
• Human on the loop. The human acts as a supervisor of a running system and intervenes only when necessary, such as on anomalies, edge cases, low confidence predictions, or sampled audits.

HITL suits high-stakes decisions and ambiguous tasks, where the cost of an error justifies the cost of mandatory review, and HOTL suits scaled operations, where oversight and monitoring must coexist with throughput.

The transition between the two paradigms can be managed progressively. Microsoft’s Crawl-Walk-Run framework [11] describes a maturity path for increasing AI automation in products. In Crawl Mode, human involvement is mandatory (pure HITL), in Walk Mode, the AI can interact directly with internal employees, in Run Mode, automation increases further, potentially including direct AI interaction with external users [3]. Each step relaxes the human checkpoint while, ideally, strengthening the monitoring system that replaces it.

A third position closes out the spectrum, Human Out of the Loop (HOOTL). Here the system runs fully autonomously inside boundaries a human set in advance, objectives, constraints, success criteria, and no one intervenes in routine operation. Oversight stops being a checkpoint and becomes an audit, humans review outcomes after the fact instead of steering them as they happen. HOOTL is defensible only where errors are cheap, the operating environment is well-characterized, and the boundaries themselves have been stress-tested, because once the system is running, there is no human left in the moment to catch a mistake [4].

Agentic AI pushes this trajectory to its current frontier. Agentic systems do not merely predict; they decide and act, which changes the design requirement. Oversight shifts from reviewing every model output to governing actions. Supervision then takes the form of checkpoints (approval gates, rollback mechanisms, and monitoring triggers), a configuration sometimes described as “human over the loop.” This makes human-computer interaction a safety property in itself. The workflow must make it easy for the supervisor to intervene quickly and correctly. The resulting division of labor is symbiotic rather than adversarial. AI handles routine tasks; humans handle exceptions, policy boundaries, and strategy; continuous human feedback keeps the system aligned as real-world conditions shift. Cloud-scale model stacks and platform HITL layers are complementary in this configuration. The model runs at scale while the human layer enforces governance at the right moments.

Both paradigms, however, share a hidden dependency. They assume the human performs consistently. The next section examines why that assumption is fragile.

5. Cognitive Load and Task Fatigue: The Hidden Failure Mode

5.1 Cognitive load

Cognitive Load Theory [5] models the mental effort imposed on working memory during learning and problem solving. Load is commonly decomposed into intrinsic load, originating from the inherent complexity of the material, and extraneous load, arising from the way information is presented [5, 6]. Poorly constructed materials add unnecessary workload and complicate learning and decision-making, whereas well-structured information balances cognitive demands and aids retention, and excessive load overloads working memory and degrades performance [6].

The consequences for oversight work are direct. High cognitive load increases errors and distractions, lowers processing efficiency, and degrades trust calibration with decision-support systems in high-stress situations. Precisely the failure modes an oversight role cannot afford [6]. It also constrains cognitive flexibility, limiting creativity, and slows decision-making under time constraints, while an optimal load promotes engagement [6]. Adaptive systems that monitor and modify task difficulty in real time have been shown to keep mental effort within acceptable limits and improve learning and decision outcomes [6].

Beyond the classical theory, recent work adds distributed-cognition and motivational perspectives. Human–AI interaction redistributes mental effort across the joint system, affecting autonomy and decision quality, and Self-Determination Theory [7] explains how perceived autonomy and competence shape whether people genuinely engage with AI tools rather than defer to them [6].

5.2 Task fatigue and mental exhaustion

Mental fatigue is a psychobiological state produced by prolonged cognitive work, resulting in diminished cognitive capacities such as memory and attention [6, 8]. It arises from insufficient sleep, extended concentration, or repetitive tasks, and it manifests as increased error rates, slower reaction times in tasks requiring sustained attention, impaired attentional control and task switching, and reduced motivation and cognitive flexibility [6, 8]. Recovery is slow, and chronic fatigue is associated with burnout and stress [6].

For HITL and HOTL systems alike, the implication is that oversight quality is not a constant, in the absence of motivation, recovery, or other supporting conditions, it decays over a shift, over a queue, over a career. A review process that is reliable at item 10 may be a rubber stamp by item 400. Designing the human task is therefore as consequential as designing the model.

6. Reducing Cognitive Load Fatigue in Human-Computer Interaction Systems

6.1 Solicitation design: matching the task to the human

How a task is presented to the human, its solicitation design, determines the load it imposes. A discreet and effective example is the image-identification CAPTCHA. The user selects images containing a target object to verify they are human. The task is trivially simple, takes seconds, and occurs only occasionally for any individual. The output per annotator is negligible, but the strength of the approach lies in its distribution. The reCAPTCHA system demonstrated that such micro-contributions aggregate at scale, with over 100 million CAPTCHAs solved daily, it channelled human verification effort into digitizing books with word-level accuracy exceeding 99% [9].

Annotation tasks can be positioned along two axes: Task Annotation Duration and Task Annotation Complexity, and each quadrant calls for a distinct solicitation strategy as shown in Figure 1.

Here’s how each solicitation zone plays out:

• The Verification Zone. A single glance settles the case, so the design challenge is distribution, not depth, as with the CAPTCHA model.
• The Batch Labeling Zone. It keeps that same low complexity but stretches it across a session, so the strain comes from duration rather than difficulty.
• The Validation Zone. It accepts real complexity in exchange for keeping each judgment short, which is exactly what makes it viable as a checkpoint rather than a bottleneck.
• The Deep Review Zone. It is where both costs compound, reserved for cases where the price of a miss is high enough to justify it.

The matrix sorts tasks by how long they take and how hard they are, but a further variable cuts across every cell: how much of the system’s own reasoning to expose to the human doing the work. Transparency builds trust, but the explanation behind it consumes attention, and that cost does not show up on either axis above. The case study below measures it directly.

you can’t fix cognitive load purely by hiring better or more experienced people. You also have to design the task, its length, how it’s chunked, how much is asked per sitting, because duration and complexity impose their own cost independent of who’s doing the work.

The matrix sorts tasks by how long they take and how hard they are, and neither axis can be solved by hiring better or more experienced people: duration and complexity impose their own cost independent of who’s doing the work, which is why the task itself, its length, its chunking, how much is asked per sitting, has to be designed into the task, not solved by who you assign to it. A further variable cuts across every solicitation zone (Figure 1) in that design space: how much of the system’s own reasoning to expose to the human doing the work. Transparency builds trust, but the explanation behind it consumes attention, and that cost does not show up on either axis above. The case study below measures it directly.

Recommendation Agent Adoption: How Recommendation Presentation Influences Employees’ Perceptions, Behaviors, and Decision Quality

Bigras, Léger, and Sénécal [10] tested to balance transparency against effort, in the setting where it bites hardest: an organization where employees must not only trust an AI recommendation agent (RA) but justify their decisions to superiors. Transparency, explaining the reasoning behind a recommendation, is what makes that justification possible, but the explanation itself is a cognitive cost, and when it exceeds the user’s processing capacity the effect reverses. Overload breeds acceptance without understanding, and hence heavier reliance on the RA rather than better judgment [10]. To test this, twenty retail professionals used an AI-based RA for an assortment-planning task under three presentations (1. no explanations, 2. low-effort explanations, and 3. high-effort explanations), while researchers tracked their perceptions, eye movements, and actual decision quality [10].

Perception and performance diverged. Employees felt most in control, and rated the RA as more credible and satisfying, whenever it was transparent, and felt this regardless of how much effort the explanation took to parse, effortful transparency scored just as well subjectively as effortless transparency [10]. Objective decision quality told a different story. It improved only under low-effort transparency; the high-effort version was, if anything, consulted more often, the overload driving dependence rather than resistance [10]. Tellingly, nearly half the participants spontaneously asked for a customizable RA that would let them fold in their own tacit expertise rather than accept the recommendation as given. It was a request to remain a collaborator, not a rubber stamp [10]. The guideline this supports is that exposing reasoning partially, through condensed and easily accessible explanations, is enough to build trust and support justification, and light enough to protect the accuracy of the decision itself.

6.2 Design principles

The evidence reviewed above suggests a set of principles for reducing cognitive load fatigue in HCI systems:

1. Reduce extraneous load first. Most avoidable workload comes from presentation, not content; condensed visual representations, progressive disclosure, and consistent layouts lower effort without concealing information [5, 6, 10].
2. Route by complexity. Automate routine checks and direct humans to ambiguous, high-stakes, or low-confidence cases, in line with the quadrant model in Figure 1, humans as exception handlers [1, 2].
3. Design for recovery. Rest intervals and deliberate task design are effective countermeasures to mental fatigue [6, 8], combined with annotator rotation and session limits, they function as quality controls rather than perks.
4. Adapt in real time. Systems that monitor performance signals and adjust task difficulty accordingly maintain mental effort within acceptable bounds [6].
5. Calibrate continuously. Clear guidelines, inter-annotator agreement metrics, and regular calibration sessions keep quality measurable and drift visible [2].
6. Preserve agency. Perceived autonomy and competence drive sustained engagement with AI tools [7], allowing humans to adjust and override system outputs also captures tacit domain knowledge.
7. Balance transparency against effort. Explanations build trust but consume attention, the balance point is an empirical question.

Conclusion

Human in the Loop is often treated as a checkbox. A human reviews the output, therefore the system is safe. The evidence assembled here says otherwise. Oversight quality is a function of workload routing, presentation design, and fatigue management, and it degrades predictably when those are ignored. As systems move from in-the-loop review toward on-the-loop supervision and agentic autonomy, the human’s role concentrates precisely where cognitive load matters most, like with exceptions, judgment calls, and intervention under time pressure. The human is not a fail-safe bolted onto the system; the human is part of the system, with a performance curve of their own. Design for that curve (route by complexity, minimize extraneous load, balance transparency against effort, preserve agency) and the loop works. Ignore it, and you have built an expensive rubber stamp.

Key Takeaways

• The human is the most trustworthy part of an AI system and the least scalable
• HITL suits high-stakes, ambiguous cases, and HOTL suits scaled operations
• In the absence of motivation, recovery, or other supporting conditions, oversight quality decays with fatigue over a shift, a queue, or a career
• Task duration and complexity shape cognitive load, apart from reviewer skill
• Only low-effort transparency actually improves decision quality
• Preserving human agency keeps people collaborators, not rubber stamps
• Design for the human’s performance curve, not just the model’s

References

[1] Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022). A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135, 364–381. https://doi.org/10.1016/j.future.2022.05.014

[2] Monarch, R. (2021). Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-Centered AI. Manning Publications.

[3] Huyen, C. (2025). AI Engineering: Building Applications with Foundation Models. O’Reilly Media. (Includes Microsoft’s 2023 Crawl-Walk-Run framework for gradually increasing AI automation.)

[4] LUKIANOV, D. Optimal Trust in AI for Project Management: A Decision-Theoretic Analysis of HITL, HOTL, and HOOTL Based on an Extended IPMA ICB 4.0 Competence Model.

[5] Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4

[6] Generative AI and Cognitive Challenges in Research: Balancing Cognitive Load, Fatigue, and Human Resilience. (2025). Technologies, 13(11), 486. https://www.mdpi.com/2227-7080/13/11/486

[7] Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78.

[8] Boksem, M. A. S., & Tops, M. (2008). Mental fatigue: Costs and benefits. Brain Research Reviews, 59(1), 125–139. https://doi.org/10.1016/j.brainresrev.2008.07.001

[9] von Ahn, L., Maurer, B., McMillen, C., Abraham, D., & Blum, M. (2008). reCAPTCHA: Human-based character recognition via web security measures. Science, 321(5895), 1465–1468. https://doi.org/10.1126/science.1160379

[10] Bigras, É., Léger, P.-M., & Sénécal, S. (2019). Recommendation Agent Adoption: How Recommendation Presentation Influences Employees’ Perceptions, Behaviors, and Decision Quality. Applied Sciences, 9(20), 4244. https://doi.org/10.3390/app9204244

[11] Microsoft Security Blog. (2026). Crawl, Walk, Run: A Practitioner’s Guide to AI Maturity in the SOC. Microsoft Community Hub. https://techcommunity.microsoft.com/blog/microsoft-security-blog/crawl-walk-run-a-practitioners-guide-to-ai-maturity-in-the-soc/4500433

---

Human In The Loop and Cognitive Load Fatigue was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.