Research notes · The scientific basis

Where Our Method Comes From.

Arclab's teaching design is not original. It implements conclusions that three separate research groups published in 2025. For each paper we list what it found, what we built because of it, and what it does not establish.

That last part matters most. A method is only as trustworthy as its weakest citation, so we would rather you saw ours than discovered them. None of these studies tested Arclab.

Concepts for Teaching Software Development in the Age of AI Tools

Axel Böttcher, Veronika Thurner, Benedikt Zönnchen

IEEE Global Engineering Education Conference (EDUCON), 2025

Read the paper on IEEE Xplore ›

What it found: Programming fundamentals still matter. The authors argue that learners must understand core concepts before they can use AI tools effectively, because without that foundation AI answers risk "increasing their confusion, rather than helping to overcome it." They compare three options, banning AI, allowing it from day one, and starting conventionally then introducing it through deliberate interventions, and recommend the third. They also set out a ladder of AI specific learning objectives that rises to detecting semantic errors in generated code and judging whether a model's output is appropriate at all.
What we built because of it: Our first seven Python modules are AI free. The mentor unlocks in stages after that, as an explainer first, then a comparator, then a collaborator. The modules on supervising AI and reading code sit near the end of the track rather than the start. Every module carries an explicit AI phase, and the product enforces it at runtime instead of leaving it to the learner's self control.
What it does not prove: This is a position and framework paper, not an outcome study. Its evidence base is roughly 40 students over a single academic year, and the authors themselves describe it as anecdotal. They explicitly leave open the question of whether the ability to write code can be acquired by only reading code, and note that the experiment to settle it would be unethical to run on real students. No learning outcomes were measured.

Designing CS Assignments to Foster Cogitation in the AI Era

Jaya Veera Surendra Gupta Kurivella, Rajeshwari Deoraj, Marcia Moraes

IEEE Frontiers in Education Conference (FIE), 2025

Read the paper on IEEE Xplore ›

What it found: The shape of an assignment decides how easily AI can finish it. Testing ChatGPT-4 against 11 introductory Python labs, the authors recorded 100% accuracy on the simplest text based problems and 35% on the hardest, with the number of prompts needed climbing as the problems got harder. Presenting a problem as a flowchart degraded the model's performance the most, and cost the least to author.
What we built because of it: A module dedicated to reading and reviewing code rather than producing it, and task types that make you trace, predict and explain instead of only submitting. We are adding non textual tasks such as flowcharts for variety and dual coding.
What it does not prove: No students took part in this study. Two researchers acted as proxy learners prompting ChatGPT, so what it measures is how hard a problem is for a model, not how much a person learns. The authors concede that prompt count "is not sufficient as a metric for critical thinking," and that future models may handle these formats better. We also decline the paper's adversarial techniques, such as putting deliberately misleading arrows in a diagram. Those make sense when students are graded for credit and have a reason to cheat. Our learners are here to learn, and the only person they could fool is themselves.

Enhancing Novice Programming Education through AI-Driven Mentorship and Project-Based Learning

Sai Veda Prakash Masetty, Sreeja Vallamulla

IEEE Integrated STEM Education Conference (ISEC), 2025

Read the paper on IEEE Xplore ›

What it found: The strongest outcome evidence of the three. In a four week controlled pilot with 200 participants, learners using an AI mentor that gave "hints, explanations, and step-by-step guidance rather than direct solutions" completed 90% of final projects against 60% in the control group, resolved bugs in 18.2 minutes on average against 32.7, and improved 22.2 points on a post-test against 10.3. Help was deliberately pulled rather than pushed, sitting behind a button the learner pressed after already attempting the problem.
What we built because of it: Our mentor refuses to hand over answers. It points at the line, names the concept, and offers a similar but different example. We split help the way the study did: the general mentor is open whenever your module has unlocked it, while the task specific help that can read your own code stays closed until you have made an attempt. Hints are available the whole time, so nobody is stuck with no way forward. We also record attempts, hints and help requests per task, because that instrumentation is the only reason the study above could claim anything at all.
What it does not prove: Four weeks is a short intervention, the sample was self selected, and the authors name a possible novelty effect from using a new tool. Participants averaged around six months of prior programming experience, so the results say little about complete beginners. Knowledge retention was not measured. The authors are also the vendor of the platform under evaluation.

Our own numbers

What We Measure Ourselves.

We are not going to borrow anyone else's results. The figures on this page belong to the studies that produced them, on the platforms those studies tested.

We have started recording what actually happens here instead: how often a learner gets a task right on the first try, how many hints they open, how often they ask the mentor for help and what they ask it for, and how long each task takes. Those are the measurements that let the studies above say anything at all.

Until a full cohort has worked through a track, treat this page as a statement of where our design comes from, not as evidence that it works. When we have our own numbers, they will appear here, including the ones we would rather not publish.

Learn The Part AI Can't Do For You.

Start with Python fundamentals, then earn the mentor.

Start free →

Researching AI and programming education? Tell us about your work.