context
Learning to code by typing alone doesn't work for everyone. There's a disconnect between understanding a concept and internalizing it. Handwriting, on the other hand, has long been shown to improve retention and comprehension over typing, but there's no good way to handwrite code and actually run it — Puterbook automatically recognizes handwritten code blocks and executes them, while also letting users create annotations and doodles to learn how code works, combining the cognitive benefits of handwriting with the feedback loop of real code execution.
code detection test 1
learn code the way you learn everything else-
by hand.
introduction (for real)

vision & python server test
about puterbook
Puterbook is one of my first (and favorite) projects in my Selfish Design series: things not out of the kindness of my heart but out of personal need (or for fun), and was created for Hack Your Summer 2026 - a month-long design & development sprint hosted by Coding It Forward.
the Hack Your Summer (HYS) timeline
week 1
🌱🖍️
Discovery
Finding a problem
worth solving.
->
week 2
🛠️📝
Build
Prototyping!
week 3
💻👧🏽
Iteration
Gaining feedback for iteration
->
week 4
🎙️📚
Storytelling
Reflecting, documenting and sharing!
week 1 : problem discovery
Before building anything, I scoped the app with a feature audit, grounded it in a persona, sketched the core interaction, and tied it all together with a storyboard of the end-to-end flow :).

storyboard
My goal was to start a nice foundation and MVP for my final proto v.1 of puterbook! I identified the problem, our target user, my core value propositions, and my MVP feature set.
Problem: Learning to code on a screen breaks the note-taking habits that actually work for learning (doodling, annotating, seeing things change). Students bounce between separate notes and a separate code editor instead of learning in one place.
Target user: Students learning to code for the first time — especially visual/kinesthetic learners who learn well by writing/drawing by hand.
Core value proposition: Let people write Python by hand and actually run it — collapsing "take notes" and "write/test code" into one action instead of two separate tools.
MVP feature set:
Handwriting recognition that converts pencil-written Python into executable code
Run button that executes the recognized code and shows output
Basic infinite/scrollable canvas
Colored ink for annotations that's ignored by the code parser
week 2 : building, prototyping, and MVP

I built Puterbook in Swift, developed and tested live on a physical iPad (connected to Xcode and my build environment).
For recognition, I used Apple's Vision framework on-device — no server, no latency. Indentation (Python's whitespace sensitivity) is calculated from Vision's line bounding boxes rather than raw stroke measurement.
Execution went through a few pivots. I'd initially developed with a local Mac server, and finally landed on Wandbox, a free no-auth code execution API. All together this is enough infrastructure to prove the core loop without over-building.
Aside from the technical design decisions I made this week, a core pillar of Puterbook's design identity was accessibility — ease of access, simplicity, and adoptability. I want people who've never touched tech in their lives to have a meaningful entry into programming, without feeling limited by technical knowledge or hardware.
week 3 : Iteration
key decisions and trade-offs
1. Layers vs. lasso vs. auto-detect vs. color ink
how does the app know what's code and what's a doodle?
Layers: separate code and note layers mean users have to manage which layer they're on, which is extra UI and extra mental load for someone already new to code.
Lasso : select your code, then run it. It works, but it adds a step every single time you want to run something, it lost out as a way to separate code, but I kept it for moving code blocks.
Auto-detect: no extra steps, but handwriting is messy. If it guesses wrong, your doodle becomes a syntax error, and users lose trust fast.
Mode Toggle & Color ink: I started with separate Code / Annotate modes, but switching mid-thought was one more thing to remember, so I merged the mode into the pen itself, working with familiarity. Users already know how to pick a color; I've only added intention to the choice : grey is code.
before (week 2)
functional prototype: proof of concept for the core loop
grey code recognition, ignored annotations, developer-facing UI,
debug tools (stroke count, code recognition output)
after (week 4)
full toolbar with color bar, pen and brush types,
undo/redo, clear canvas,
and lasso for moving code blocks
2. Familiar UI vs. a new UI
how much should a beginner have to learn before they can write their first line of code?

lite feature audit
custom UI: more distinctive, but it's one more thing to learn for someone already intimidated by code.
Familiar UI : I kept the toolbar intentionally familiar. Learning to code is already hard, so the only new things users learn are that grey ink is code and Run runs it.


Puterbook Toolbar

Goodnotes Toolbar
familiar toolbar, two new pieces
3. Servers vs. Users
Through HYS, I connected with industry mentor Priya Poolavari, VP Product / Engineering at InSpatial Media, to walk through approaches to the Python execution pipeline! The local Mac server I'd started with worked, but it tied the whole app to having a nearby computer running which cuts out anyone who doesn't have their own machine set up to act as a server. We talked through how a sandboxed execution API solves that: it puts the tool within reach of people who just have an iPad, not just people with the technical setup to run their own backend. That's what pushed me toward Wandbox.
canvas prototype development progress
week 4 : Storytelling, Reflection, & "Final" Product
final product
Puterbook is an iPad app that lets you write Python by hand and actually run it — collapsing "take notes" and "write code" into a single action instead of two separate tools.
Handwriting Canvas
An infinite, scrollable canvas built for writing code and notes the way you'd write in any notebook — pencil options, colors, and all the basics.
Color-Coded Ink
Grey ink means code, any other color means annotation — no new interaction to learn, just pick a pen color like you always would.
On-Device Recognition
Apple's Vision framework reads handwritten Python directly on the iPad — no server, no latency, no waiting on a network call.
Run & See Output
Tap run, and recognized code executes through Wandbox, with real output displayed right on the canvas — no jumping between a notes app and a separate code editor.
for the future
Reflections & What's Next
I design with a working knowledge of what's actually feasible to build. Puterbook is what happens when a self-taught illustrator learns to code and builds the notebook she wished she'd had. It's a product born from my own experience learning to code, shaped by design instincts from illustration and UX, and built end-to-end by hand.

The biggest surprise wasn't the handwriting recognition or the code/notes detection — it was how much of the build ended up depending on infrastructure outside my control. Pivoting execution APIs mid-build, from a local server to Piston to Wandbox, taught me to treat third-party dependencies as a real risk to plan around, not just an implementation detail to pick once and forget.
Puterbook is now being revamped for open-source development (hooray!), so other developers and learners can build on it too. The original Hack Your Summer version is still archived on GitHub if you want to see where it started.
My vision for Puterbook's next phase is tackling a real production execution backend. While Wandbox's public API proved the core loop, it's not built for reliability or scale. Everything else (saved history, layer management) can wait; execution is the one piece the whole app depends on working.














