Bridging the gap between a design idea and working code

01Designers can sketch in hardware, but not in software

Interaction designers who prototype physical products often need to write code to make an idea work, but most have little to no programming experience. Lars Erik Holmquist’s “Sketching in Hardware” (Interactions, 2006) named and argued for exactly that idea: a way of sketching interactive products that’s distinct from a finished prototype. Source Lars Erik Holmquist’s “Sketching in Hardware” (Interactions, 2006) named and argued for exactly that idea: a way of sketching interactive products that’s distinct from a finished prototype. Source With little to no programming experience, the problem designers ran into wasn’t the syntax of a programming language. It was translating an idea into something a computer could execute at all.

My M.Sc. thesis picked up the flip side of a problem I’d already run into on Atreyu, an earlier TU Delft project that let designers sketch interactions directly in hardware, no code required. Sketching in hardware had a real answer by then. Sketching in software still didn’t.

02The gap between idea and execution is as old as programming itself

The thesis traces that gap all the way back through the history of programming itself: from the abacus and Pingala’s binary numeral system around 300 BC, through Leibniz’s binary theory and Babbage’s mechanical Difference and Analytical Engines, with Ada Lovelace, working alongside Babbage, writing what’s often called the first algorithm, to Alan Turing’s 1936 solution to the decision problem, the universal Turing machine every programming language still traces back to today. A. M. Turing, “On Computable Numbers, with an Application to the Entscheidungsproblem,” Proceedings of the London Mathematical Society, 1937. The universal machine Turing described in this paper, one machine able to compute any computable sequence, is the theoretical basis every modern computer and programming language still rests on. Source A. M. Turing, “On Computable Numbers, with an Application to the Entscheidungsproblem,” Proceedings of the London Mathematical Society, 1937. The universal machine Turing described in this paper, one machine able to compute any computable sequence, is the theoretical basis every modern computer and programming language still rests on. Source

Almost every step after Turing was really about the same thing: lowering the barrier to who could program. Von Neumann’s 1945 stored-program architecture moved instructions out of hardwired switches and into memory. FORTRAN, in 1954, let people write something a human could read and compile it down, rather than addressing the machine directly. Thousands of higher-level languages followed, some built specifically for beginners. Even with all these changes and novel approaches, the entry level for designers often remained too high.

My thesis argued that the gap between a design idea and working code was a continuation of that same problem: how to lower the barrier to who could program, this time for interaction designers specifically.

03Analyzing finished code by itself, based of interactive workshops

The first study was a three-day Arduino workshop in TU Delft’s Interactive Environments minor. Students got a tutorial on day one, walking through the code for a basic electronic dice step by step, then spent the remaining days building their own version and submitting the finished Arduino code. Eleven dice came out of it.

Printing and comparing all eleven codebases side by side was supposed to reveal how designers worked through a programming problem. It didn’t, not the way I’d hoped. Finished code turned out to say very little about the mental process behind it: students had already erased their errors and false starts, so all that remained was the polished, working result.

Most of that result was also strikingly uniform, closely following the tutorial example, which made sense given they’d just been shown exactly how to do it. The one place real variation did show up was the part nobody had explicitly taught: how to generate and display the random number. That’s where students had to select and combine code fragments on their own, rather than lean on what they’d just been handed.

The workshop was a good way to teach programming and a flawed way to study it. A tutorial followed by homogenous results couldn’t show the actual thinking I was after. That conclusion shaped the next study directly: strip away the tutorial, remove the computer entirely, and give people enough freedom that any variation in their approach had to come from them.

04A paper programming exercise revealed two different ways designers think

To study the gap without a compiler getting in the way, eleven students were asked to design their own fictional programming language and sketch on paper how they’d program two design briefs in it: an interactive lamp, and generating a large number of unique Space Invaders. With no computer to say a solution was wrong, the exercise showed how people thought through the problem, not just whether they could code.

The eleven students produced 22 A3 sheets between them, and two distinct patterns emerged. Some students used a trial-and-error model, breaking the problem down as they went, revising code fragments whenever they thought something didn’t work. Others used a path-finding model: define the requirements first, then the code fragments needed, then find a path from input to output using those fragments, revising the fragment list rather than starting over when something was missing.

An independent reviewer, given only the 22 sheets and no context from the sessions themselves, identified the same path-finding pattern on their own.

The lamp brief was easier for most students than the Space Invaders brief, and the reason was concrete rather than abstract: the lamp let them think in physical components they already understood, while Space Invaders asked them to reason about symmetry and repetition with nothing physical to anchor it to.

05The Blueprint turns path-finding into a paper tool

The path-finding model became the structure for the first of two tools: the Blueprint, an A3 paper sheet with four steps. First, describe the program in one line, a check the designer can verify their final sketch against later. Then define the inputs and outputs. Then define the tools, the transformers, conditions, and data, needed to get from one to the other. Finally, sketch the path connecting them.

The Blueprint was designed to break an idea down into smaller, more abstract chunks. Forcing designers to think in specific steps introduced that structure deliberately, rather than leaving it to chance. Thinking first about the specific interactive outcome, then the inputs, then the conditions under which those inputs should trigger it, broke an idea down in a far more considered way than sitting down and just starting to code.

06Testing the Blueprint with real design students

The Blueprint was tested in two workshops, one at the University College of Antwerp, one at the Glasgow School of Art. Antwerp went first: by the end of the five-day workshop, every team had a working interactive sketch, five in total, and the Blueprint had clearly helped students break their ideas into programmable pieces. But the sequence printed on the sheet didn’t match how students worked through it. Every student in Antwerp independently arrived at the same order: define the tools and data before the conditions, not after, since a condition only makes sense once you know what it’s conditioning.

Glasgow produced similar results, with two real differences. Students there also had to learn basic electronics alongside programming, and the added complexity showed: they struggled more, and the spread in how well individual students coped was noticeably wider than in Antwerp. Glasgow’s students also came from a more conceptual, artistic background than Antwerp’s more technical cohort, and that showed up directly in the results. Despite that, every student still walked away with a working interactive sketch.

Across both workshops, over 40 students consistently said the Blueprint gave them a clearer picture of what they had to program. But it also surfaced the next problem: students could sketch a pattern into the Blueprint, a timer, a toggle, a random selection, without knowing how to write that pattern in code.

07The Pattern Dictionary was designed to close the second gap

That second gap needed a second tool. The Pattern Dictionary, inspired by the book “The Practice of Programming,“ Kernighan and Pike’s “The Practice of Programming” collects common programming patterns and best practices; the Pattern Dictionary borrowed its format, not its content, applying the same idea to patterns interaction designers specifically needed. Source Kernighan and Pike’s “The Practice of Programming” collects common programming patterns and best practices; the Pattern Dictionary borrowed its format, not its content, applying the same idea to patterns interaction designers specifically needed. Source is a reference of common programming patterns, focused on Arduino specifically since that was already the default platform in the design program.

The idea behind patterns in the Pattern Dictionary is that they provide building blocks for programs. Each pattern, a timer, a toggle, sensor input, includes a plain-language description alongside the actual code, plus a synonyms section, since the paper programming study had already shown designers often use different names for the same underlying pattern.

08Reducing hardware and electronics as a barrier

Glasgow’s extra struggle with electronics pointed to a barrier the Blueprint and Pattern Dictionary weren’t built to solve: getting the hardware itself working in the first place. One direction the thesis pointed toward, more a proposal for future work than a finished tool, was a standardized hardware playground, a breadboard that comes preset with a fixed set of inputs and outputs already wired in.

Designers wouldn’t have to think about the initial hardware setup at all. They could go straight from Blueprint to working prototype, without electronics becoming a separate barrier of its own. The reference inputs and outputs matched what was in the Pattern Dictionary.

The three tools combined were dubbed the Code Companion, which was later published at EDULEAR12. Robert A. Paauwe, Walter A. Aprile, Aadjan van der Helm, “Teaching Interaction Designers to Sketch in Software,” EDULEARN12 Proceedings, 2012. Source Robert A. Paauwe, Walter A. Aprile, Aadjan van der Helm, “Teaching Interaction Designers to Sketch in Software,” EDULEARN12 Proceedings, 2012. Source

09What the Code Companion solved

This research revealed that the difficulty lies in translating an idea into something structured enough for a computer to execute. Breaking that translation into explicit steps, outcome, inputs, conditions, gave designers a process to work through instead of something they had to intuit from scratch.

Pairing that with simple, plain-language reference patterns closed the other half of the gap. With the Code Companion, designers now had a concrete way of understanding how to build and prototype their interactions.

The full thesis, including the electronic dice study, the complete paper programming results, and the final Pattern Dictionary, is available through TU Delft’s repository. Robert A. Paauwe, “Programming for Interaction Designers,” M.Sc. graduation thesis, Delft University of Technology, 2012. Source Robert A. Paauwe, “Programming for Interaction Designers,” M.Sc. graduation thesis, Delft University of Technology, 2012. Source