signbridge

the problem

for many deaf and hard-of-hearing people, a doctor's visit can be a source of real anxiety. we were moved by the story of an asl instructor who was refused treatment because no interpreter was available. bringing a third-party interpreter can compromise medical privacy - we wanted something that lets people communicate on their own, privately.

design choices

• a wearable: 3d-printed glasses with a camera on the left arm, so the signer's hands stay free and the camera sees what they see.
• a raspberry pi as the processor - compact enough to wear, and our team was already strong in python.
• mediapipe hands for real-time 3d hand landmarks, feeding a tensorflow model that maps them to asl signs.
• our own dataset. public asl datasets didn't match a wearable camera's angle or the lighting of a clinic, so we recorded data from the device itself.
• a fingerspelling / word toggle. the model struggled to tell letter-by-letter spelling from single-gesture words, and forcing a word onto a letter produced gibberish - so we built logic to switch between the two modes.

challenges

• weight balance - fitting a camera and a raspberry pi onto someone's head took several redesigns of the glasses.
• soldering in tight spaces led to loose connections, and keeping the camera's power supply stable took real time.

what i learned

how to fit a machine-learning pipeline into constrained hardware - and that engineering matters most when it serves someone's independence and dignity.

team

built with harsit baral, akileash saravanan and dip khadka.