I previously wrote an article titled Seven hypotheses on why educational institutions cannot shift to active learning.

In that article, I asked why educational institutions have not made a major shift toward active learning, even though ideas such as active learning and personalized learning have existed for a long time. One of the hypotheses I raised was that measuring learning effects is difficult in the first place, or that it is hard to measure whether active learning or passive learning is better. Another was that teachers may not be able to adapt to active learning, or that securing teacher resources may be difficult.

This time, I worked on a study that is close to a continuation of that question.

Agent-Based Theory Testing: Bloom’s 2-Sigma Problem (GitHub)

Can LLM Agents Test Social-Science Theory? (zenode)

The main theme was whether LLM agents can be used to build a test bed for social-science theory. The topic I chose was Benjamin Bloom’s 2-sigma problem. Roughly put, this is the famous claim that one-to-one mastery learning with tutoring produces much larger learning gains than conventional group instruction.

I did not think from the beginning that LLMs could reproduce and verify human society as it is. Part of the purpose was to investigate the limits and possibilities of LLM agents through this kind of work. What I wanted to do was build a small test bed where educational theories could be tried safely, and then examine what that test bed can and cannot measure.

In that sense, I felt again that a powerful technological shift is underway. A question that, a few years ago, I could only leave as a hypothesis, or expected to test through practice over several years, could be moved forward a little through this experiment.

The result was somewhat surprising

At first, I expected one-to-one tutoring to show a fairly strong advantage. Since the target was Bloom’s 2-sigma problem, that was natural. When people talk about AI tutors, the image of every learner having their own teacher is also strong.

However, in this LLM-agent experiment, one-to-one tutoring did not clearly win.

I used a synthetic rule domain called Zarn Tokens, a new game created for this study, and rebuilt the experiment across seven generations. In the earlier experiments, some results made discussion or certain instructional formats look strong. But there were various confounds, meaning other factors that distorted the results. Prerequisite knowledge was not sufficiently aligned. Multiple learners were reading a single generated conversation. Learning opportunities and token budgets differed by instructional format. The scorer did not handle free-text answers well, and so on.

After removing those distortions, the final experiment showed only a 10-point spread across instructional conditions. Meanwhile, the spread across learner cognitive profiles was 38 points. In other words, in this experimental environment, the learner’s memory constraints and cognitive tendencies appeared to matter more than the instructional format.

This is not a conclusion that one-to-one tutoring is not very effective for humans. The subjects in this experiment were LLM agents, not people, and the task was an artificial rule domain. Misreading that point would distort the meaning of the study.

It is also possible that one-to-one tutoring works by acting on the learner’s characteristics or state. Perhaps that is closer to the essential part of the 2-sigma problem.

Still, I think we can say at least this: when trying to experiment with educational theory, it is easy to produce plausible-looking effect tests, but it is risky to use those results as they are. Through that process, however, we can obtain many clues about factors such as prerequisite knowledge, evaluation methods, sample independence, differences in learning opportunities, and learner characteristics, rather than instructional format alone.

Measuring learning is still hard

In the earlier article, I wrote that measuring learning effects is difficult. This study made that point more concrete.

For example, if scores rise after a certain instructional format, was it because of the format? Or because prerequisite knowledge happened to be aligned? Because the conversation quality happened to be good? Because the learner happened to fit that format? Because the scoring method favored that answer style?

This is difficult in human studies, and it remains difficult with LLM agents. In some ways it is even riskier, because all the logs remain and the conditions can look cleanly separated, making it easy to feel as if the construct has been measured.

The most important part of this study was probably not deciding whether tutoring or group instruction won. It was continuing to doubt what each of our experiments was actually measuring.

One-to-one is not automatically better

Another interesting point was that tutoring seems to demand a higher level of preparation and knowledge from the teacher side.

In group instruction, a teacher can design a coherent explanation as a lesson and deliver it. Of course, making good group instruction is not easy. But at least there is a central task: design the explanation and communicate it.

In one-to-one tutoring, the teacher needs to identify each learner’s misunderstanding on the spot, rephrase appropriately, add missing context, and keep checking understanding. If the teacher cannot diagnose what the learner does not understand, spending time one-to-one does not mean much. It may even reinforce a wrong understanding.

So the point is not that one-to-one is automatically good. To make one-to-one work, the teacher’s design and diagnostic ability matter a lot. The same applies to AI tutors. Returning personalized responses is not enough. The system must detect learners’ misunderstandings, decide in what order to intervene, and decide how much the learner should still think through.

This also fits very well with what I have experienced in practice.

This connects to the earlier hypothesis that teachers may not be able to adapt to active learning, or that teacher resources may be hard to secure. Active learning and personalized learning do not simply reduce teachers’ work. In some cases, they substantially change the abilities teachers need.

Think in terms of design, diagnosis, and intervention, not format

Overall, this experiment changed my thinking about active learning a little.

It also gave me a more practical feel for both the possibilities and the current limits of LLMs.

Active learning and personalized learning are not magic formats that replace conventional learning. It is not simply a matter of changing the format. What matters is designing learning so that learner differences become visible, and then enabling teachers and systems to diagnose and intervene based on those differences.

Seen this way, educational institutions may not fail to shift to active learning simply because they do not understand the new philosophy. Measurement for handling learner differences, teachers’ diagnostic skills, lesson design, evaluation, and cost all need to change together. Changing only the format does not accomplish much.

This is a modest conclusion. It is easier to talk in oppositions: one-to-one tutoring versus group instruction, active learning versus passive learning. But in practice, what matters more than the chosen format is what we diagnose inside that format, how we intervene, and what we count as learning. I rather like this modest conclusion.