Wave to Prove You’re Human: A Gesture-Based Authentication System

Using AI is a common experience for students. But making and teaching an AI model was an absolute experience for me.  As a design student with only basic coding experience, I never thought I’d see myself training a LLM model. But for my UX and AI class, I had to do exactly that. 

What was the project?

The goal was to solve a real-world problem using AI — but the twist was that my end product couldn’t just be an idea or a prototype; it had to be a working AI model. That requirement alone felt intimidating. My mind was immediately flooded with questions: Can I even do this? Where do I start?

How Did I Choose the Problem?

I began by looking for everyday frustrations. One thing I know everyone, including me, hates: CAPTCHAs. Those squiggly letters, blurry images, endless “find all the traffic lights” puzzles — they feel outdated and honestly exhausting. Why haven’t we come up with a more creative, human-centered solution yet? This question became the heart of my project.

My Solution: A Gesture-Based Authentication System

An AI powered Gesture-based security authentication system that allow users to verify their identity using intuitive gestures, reducing reliance on passwords and traditional two-factor authentication. By leveraging computer vision and motion detection, a gesture-based system can recognize predefined gestures—such as a peace sign or a thumbs up in a specific pattern.

Here’s the idea: instead of solving distorted puzzles or clicking traffic lights, users could simply replicate a gesture shown on the screen — like a thumbs up or peace sign — to prove they’re human. Once the system recognizes the correct gesture, AI would trigger an interactive confirmation (like a visual animation or screen response), making the whole CAPTCHA and two-factor authentication process more playful and user-friendly.

Which Model Did I Use?

While exploring various machine learning libraries and pretrained models, I came across Google’s MediaPipe — an open-source framework. It’s widely used in applications like hand tracking, face detection, pose estimation, object detection, and most interestingly, gesture recognition. That’s when an idea struck me: what if CAPTCHA didn’t rely on distorted text or image clicks, but on gestures instead?

Here’s how gesture recognition with MediaPipe works:

1. Hand Detection: The system first checks if a hand is present in the frame.

2. Landmark Detection: It then maps 21 key 3D landmarks (x, y, z) on the detected hand.

3. Feature Extraction: These landmarks are translated into meaningful features such as angles or distances between fingers.

4. Gesture Classification: Finally, these features are passed into a trained model that classifies the gesture — like a thumbs up, peace sign, or raised index finger.

   
   

How Did I Use AI for This Project?

Instead of building a gesture recognition model from scratch, I chose to work with a pre-trained model and fine-tune it for my specific use case. When I initially tried using the model, it didn’t work — and after some troubleshooting (and a lot of help from a friend and ChatGPT), we discovered the issue: the model libraries were outdated. We debugged the patch path, replaced the old libraries, and got the model running again. As a non-coder, ChatGPT came to my rescue. I used it to understand the code and even identify bugs in the existing code. So basically, I used a lot of AI to build an AI.

What Were the Shortcomings?

My gesture CAPTCHA works — but it’s far from perfect. Sometimes it misreads gestures if the lighting is bad or if the camera quality is poor. Also, not everyone feels comfortable making gestures on camera, which raises questions about accessibility and privacy.

What Did I Learn?

This project helped demystify AI for me. I realized that with the right tools and support, even someone from a design background with minimal coding experience can create something impactful. More than just building a working prototype, it opened my eyes to the creative potential of AI — not just for solving technical problems, but for reimagining how people experience the world today.

Final Thoughts

I still hate CAPTCHAs — but now I hate them a little less. Working on this project showed me that with AI, there’s real potential to make authentication more intuitive, human-friendly, and even fun. It’s just the beginning, but it proves that better solutions are possible.

To try my AI model, click here. : http://192.168.1.159:8080/

Image Credit: Unknown author, Simple English Wikipedia, Public Domain