Skip to content

Product engineering

Talk to us Discuss a Project
Talk to us Discuss a Project

Case study · Computer vision

Handy

Ten hand gestures recognised on-device and mapped to phone controls, including the awkward part, doing it without showing the camera.

Prototype · In development

Engagement
Internal innovation project
Platforms
Android application
Role
Concept and engineering
Stage
Prototype Verify before publishing
The Handy app: a hand held in front of the phone with detection landmarks plotted across the fingers, the recognised gesture labelled, and the volume changing in response.
Industry
Applied computer vision
Users
Hands-free device control
Technique
On-device hand landmark detection
Platforms
Android
Role
Innovation build
Status
Prototype

01The business situation

Sometimes you cannot touch the screen.

There are plenty of moments when a phone is fixed in place and the person using it has full hands: a live stream, a kitchen, a workshop. Voice does not always suit. Our innovation team set out to see whether a phone could be controlled by gesture alone, reliably enough that it would not be a novelty.

Core challenge

Recognise ten distinct hand gestures reliably on-device, and do it without occupying the screen with a camera preview.

Before

Device control that requires a free hand

  • Touch Required
  • Fixed Devices
  • Occupied Hands
  • Voice Not Suitable

After

Control through gesture, screen left free

  • Hand
  • Detection
  • Gesture
  • Action

Handy App

02The system at a glance

One app. Four connected parts.

  • Action Mapping Gestures bound to controls
  • Detection View Hand landmarks tracked live
  • Gesture Classification Ten recognised shapes
  • Device Control Volume and sound mode
Core PlatformOn-device landmark detection and classification
  • Hand Detection
  • Landmark Plotting
  • Gesture Classification
  • Volume Control
  • Sound Mode Switching
  • Preview-free Operation

03Workflow 01

Ten shapes the phone can tell apart.

Counting gestures, a fist, an OK sign, a rock sign and more: each has to be distinguished reliably from the others, which is harder than detecting that a hand is present at all.

Image placeholder The gesture set: several recognised hand shapes with their detection landmarks plotted, each labelled with the gesture identified. assets/portfolio/handy/workflow-gestures.png · 16 / 11
  • Ten distinct gesturesThe classifier separates counting shapes, a fist, an OK sign and others rather than detecting a generic hand.
  • Landmarks, not silhouettesDetection plots points across the fingers, which is what makes similar shapes separable.
  • Bound to real actionsRecognised gestures adjust volume and switch the device between normal, silent and vibrate.

04Workflow 02

The camera preview was the hard part.

Detection worked, but it required the camera feed on screen, which defeats the purpose when the point is to free the display. Removing it broke detection, and getting both took real work.

  • Preview removal broke detectionTaking the camera view off screen initially stopped hand detection working at all.
  • Reduced, then eliminatedShrinking the preview was the interim fix; further work removed the need for it entirely.
  • The screen stays freeThat result is what makes the concept useful, since the display was the thing being reclaimed.
Image placeholder Before and after: detection working with a full camera preview occupying the screen, then the same detection running with the preview reduced and finally removed. assets/portfolio/handy/preview-problem.png · 16 / 10
Image placeholder A gesture controlling the device: the recognised shape and the resulting volume or sound-mode change. assets/portfolio/handy/control-action.png · 1 / 2.04

05Engineering behind the product

The system behind a recognised gesture.

A small application with one genuinely stubborn constraint, and the work is almost entirely in getting past it.

  • Landmark-based detection

    MediaPipe plots landmarks across the hand, which is what allows similar gestures to be distinguished.

  • Ten-gesture classifier

    Classification logic separates the recognised shapes from one another rather than detecting hand presence.

  • Preview-free operation

    Detection was made to work without the camera feed occupying the screen, which took iteration to achieve.

  • Device control bindings

    Recognised gestures are mapped to volume adjustment and sound-mode switching.

  • Native Android implementation

    Built for the platform directly so detection runs at usable frame rates on device.

  • Finger landmark accuracy

    Detecting individual finger landmarks reliably in Java was the main obstacle, and is what settled the choice on MediaPipe.

06Technical architecture

Under the hood

MediaPipe provides the hand landmarks; the classification and the preview-free operation were built on top of it.

Android

  • Java

Computer vision

  • MediaPipe (hand landmarks)

Recognised gestures

  • ONE–SIX
  • YEAH
  • ROCK
  • Fist
  • OK

07Delivery journey

How the prototype was delivered.

  1. Concept

    Innovation team frames the idea

  2. Library selection

    Choose the detection approach

  3. Detection logic

    Build classification for ten gestures

  4. Action mapping

    Bind gestures to device controls

  5. Preview removal

    Free the screen without losing detection

  6. Refinement

    Improve reliability

  7. Next stage

    Explore applied use cases

The prototype meets its initial requirements and work continues toward the next stage. Dates and durations are withheld until confirmed. Verify before publishing

08Where it stands today

What the prototype brings together

  • Ten distinct hand gestures recognised on-device
  • Finger landmark detection driving classification
  • Volume and sound mode controlled without touch
  • Detection running without a camera preview
  • A freed screen for the actual application
  • A validated base for applied use cases

Related systems

AR Furniture

Retail · Augmented Reality · Android

An augmented-reality shopping app placing true-to-scale 3D furniture in a real room, so a buyer can see whether it fits before it arrives.

View project overview , AR Furniture

Polar Attack

Gaming · Reflex Game · Android

A tilt-controlled reflex game built on an open-source Java engine, with three difficulty levels and a published Play Store release.

View project overview , Polar Attack

A prototype earns its keep by answering one question.

Ours answer whether a technique is ready, so a client project never has to find out the hard way.

Discuss your system