september 2025
•6 min read
ctrl-arm at shellhacks 2025
a write-up on building ctrl-arm at shellhacks: the accessibility problem it was trying to solve, the hardware and ml tradeoffs that made it usable, and why the project landed with people.
why we built it
ctrl-arm started from a pretty direct question: what would computer control feel like if it leaned on the body instead of expecting a keyboard, mouse, or constant reach for a screen. shellhacks felt like the right place to explore that because the problem sits at the overlap of accessibility, interaction design, and just enough technical chaos to be interesting.
the goal was not to make a flashy demo that only worked once. the goal was to make something that felt immediately understandable: flex, move, speak, and have the machine respond in a way that feels natural instead of forced.
the stack we trusted
the core input came from myoware 2.0 emg sensors reading muscle activity, paired with motion data from a seeed studio xiao sense so the system had context for how the arm was actually moving. that combination mattered because raw emg on its own is noisy, personal, and easy to misread when the user gets tired or shifts position.
on the software side the backend handled the real-time processing in python, while an electron and react overlay kept the interface lightweight enough to stay out of the way. whisper handled speech-to-text and gemini handled natural language interpretation, which made the whole thing feel more multimodal than just gesture recognition with a voice add-on.
the real tradeoff
the biggest engineering lesson was that accuracy alone is not the right north star for body-driven interfaces. a heavier model can look amazing on paper and still feel bad in the hand if every command arrives late. what mattered most was keeping the interaction responsive enough that the system felt live.
that is why a lighter decision-tree pipeline ended up being the right call. it gave up some headline accuracy in exchange for faster inference, simpler calibration, and a version of the product that actually respected real-time use. for this kind of interface, trust comes from responsiveness first.
what made it land
the project resonated because it framed accessibility as a serious product problem instead of a side constraint. sensor fusion reduced false positives, calibration respected the fact that muscle signals vary from person to person, and voice made the whole interaction model feel more flexible instead of more complicated.
winning microsoft's shellhacks challenge mattered, but the better signal was that people immediately understood why it could be useful. the project was scrappy, but it showed a direction that feels bigger than a weekend build.
what i would push next
if i kept pushing ctrl-arm, the next step would be expanding the gesture vocabulary without making the system harder to learn. that probably means more thoughtful onboarding, better feedback loops, and more work on haptics or confirmation states so the user always knows what the model thinks just happened.
the broader idea still feels worth chasing: multimodal interfaces that combine muscle intent, motion, and voice can make computing more accessible, more immersive, and in some contexts just more human. ctrl-arm was a hackathon project, but it felt like the start of a real product conversation.