context
Learning to code by typing alone doesn't work for everyone. CS students, people grinding LeetCode, and first-time programmers struggle to retain concepts when they're just typing syntax into an IDE. There's a disconnect between understanding a concept and internalizing it. Handwriting, on the other hand, has long been shown to improve retention and comprehension over typing, but there's no good way to handwrite code and actually run it — Puterbook automatically recognizes handwritten code blocks and executes them, while also letting users create annotations and doodles to learn how code works, combining the cognitive benefits of handwriting with the feedback loop of real code execution.
code detection test 1
It allows users to learn code the way you learn everything else-
by hand.
introduction (for real)
vision & python server test
about puterbook
Puterbook is one of my first (and favorite) projects in my Selfish Design series, where I created things not out of the kindness of my heart but out of personal need and convenience - and was created for Hack Your Summer 2026 - a month long design & development sprint hosted by Coding It Forward.
the Hack Your Summer (HYS) timeline
week 1
🌱🖍️
Discovery
Finding a problem
worth solving.
->
week 2
🛠️📝
Build
Prototyping!
week 3
💻👧🏽
Iteration
Gaining feedback for
iteration
->
week 4
🎙️📚
Storytelling
Reflecting, documenting and sharing!
week 1: problem discovery
Before developing anything I mapped how a user would move through the app starting with a feature audit to scope what a handwritten code notebook would actually need. From there a persona grounded the design in a real use case, wireframe doodles carried the core interaction and a storyboard helped tie together the end to end use flow :).
storyboard
persona draft 1
lite feature audit
wireframe doodle
My goal was to start a nice foundation and MVP for my final proto v.1 of puterbook! I identified the problem, our target user, my core value propositions, and my MVP feature set.
Problem: Learning to code on a screen breaks the note-taking habits that actually work for learning (doodling, annotating, seeing things change). Students bounce between separate notes and a separate code editor instead of learning in one place.
Target user: Students learning to code for the first time — especially visual/kinesthetic learners who learn well by writing/drawing by hand.
Core value proposition: Let people write Python by hand and actually run it — collapsing "take notes" and "write/test code" into one action instead of two separate tools.
MVP feature set:
Handwriting recognition that converts pencil-written Python into executable code
Run button that executes the recognized code and shows output
Basic infinite/scrollable canvas
Colored ink for annotations that's ignored by the code parser
week 2 : building, prototyping, and MVP
The Core Problem: Code vs Notes
I built Puterbook in Swift, developed and tested live on a physical iPad (connected to Xcode and my build environment).
I started with the canvas basics, pencil options, color choices, undo/redo, clearing canvas, erasing, and most importantly- code vs annotation mode.
When it came to the handwritten code and notes and developing for their separate detection I considered layers, a lasso tool, and automatic detection, but landed on color-coded ink — grey for code, any other color for notes.
This created little to no user choice friction or interaction cost; users already know how to pick a pen color.
For recognition, I used Apple's Vision framework on-device — no server, no latency. Indentation (Python's whitespace sensitivity) is calculated from Vision's line bounding boxes rather than raw stroke measurement.
Execution went through a few pivots. I'd initially developed with a local Mac server, and finally landed on Wandbox, a free no-auth code execution API. All together this is enough infrastructure to prove the core loop without over-building.
Aside from the technical design decisions I made this week, a core pillar of Puterbook's design identity was accessibility — ease of access, simplicity, and adoptability. I want people who've never touched tech in their lives to have a meaningful entry into programming, without feeling limited by technical knowledge or hardware.
week 3 : Iteration
As a cool opportunity by HYS, I was able to connect with industry mentor, Priya Poolavari, VP Product / Engineering at InSpatial Media, to walk through approaches to the Python execution pipeline! The local Mac server I'd started with worked, but it tied the whole app to having a nearby computer running which cuts out anyone who doesn't have their own machine set up to act as a server. We talked through how a sandboxed execution API solves that: it puts the tool within reach of people who just have an iPad, not just people with the technical setup to run their own backend. That's what pushed me toward Wandbox.
before (week 2)
after (week 4)
canvas prototype development progress
week 4 : Storytelling, Reflection, & "Final" Product
Final Product
Puterbook is an iPad app that lets you write Python by hand and actually run it — collapsing "take notes" and "write code" into a single action instead of two separate tools.
Handwriting Canvas
An infinite, scrollable canvas built for writing code and notes the way you'd write in any notebook — pencil options, colors, and all the basics.
Color-Coded Ink
Grey ink means code, any other color means annotation — no new interaction to learn, just pick a pen color like you always would.
On-Device Recognition
Apple's Vision framework reads handwritten Python directly on the iPad — no server, no latency, no waiting on a network call.
Run & See Output
Tap run, and recognized code executes through Wandbox, with real output displayed right on the canvas — no jumping between a notes app and a separate code editor.
for the future
Reflections & What's Next
I design with a working knowledge of what's actually feasible to build. Puterbook is what happens when a self-taught illustrator learns to code and builds the notebook she wished she'd had. It's a product born from my own experience learning to code, shaped by design instincts from illustration and UX, and built end-to-end by hand.
The biggest surprise wasn't the handwriting recognition or the code/notes detection — it was how much of the build ended up depending on infrastructure outside my control. Pivoting execution APIs mid-build, from a local server to Piston to Wandbox, taught me to treat third-party dependencies as a real risk to plan around, not just an implementation detail to pick once and forget.
My vision for Puterbook's next phase is tackling a real production execution backend. While Wandbox's public API proved the core loop, it's not built for reliability or scale. Everything else (saved history, layer management) can wait; execution is the one piece the whole app depends on working.
made with love and some more fun stuff
©daisy fernandez-reyes 2026
last updated august 2026