00 engagement
01 problem
Prototyping natural user interfaces (NUI) that rely on computer vision and conversational AI often creates a bottleneck between designers and developers. Designers have no easy sandbox for tweaking visual states, animations, and voice pacing, and developers end up writing and maintaining complex custom UI code just to test sensor integrations. That slows down feedback loops and stops teams from iterating together on experimental input mechanics.
02 approach
Built the animated 2D game interface ("Oriyo RPS") in ProtoPie Studio, with interactive round states, live scorekeeping, and dynamic dialogue bubbles.
Implemented a Python tracking service using OpenCV and MediaPipe to detect and classify Rock, Paper, and Scissors hand landmarks from a live webcam feed.
Connected the Python CV service to ProtoPie Connect over local Socket.IO / WebSocket channels to pass detected gesture events in real time.
Integrated OpenAI API endpoints to generate contextual character dialogue and stream real-time voice reactions for wins, losses, and ties.
03 stack
04 outcome
Built as an interactive showcase for ProtoPie headquarters offline events, the project showed that ProtoPie Connect can extend past traditional screen prototyping to drive computer vision pipelines and cloud AI models. It ran as a touchless multiplayer experience with no noticeable input lag.