See. Translate. Learn.
-
Lingo lens is an augmented reality (AR) language learning app that transforms your surroundings into an interactive vocabulary builder.
-
Using your device’s camera, Lingo lens identifies everyday objects, allows you to anchor labels in 3D space, and when you tap a label it reveals the translation and plays the correct pronunciation.
-
This app, in its pre-mature state at commit
e163259a2cf234c037cc77b1eff6b222212c42e3, was submitted for the Apple Swift Student Challenge 2025, and it won the Swift Student Challenge. 🎉 -
Received a certificate signed by Susan Prescott (Apple’s VP of Worldwide Developer Relations), a pair of AirPods Max, and a free Apple Developer account.
Watch the demo video showcasing Lingo Lens in action.
Demo-Lingo-lens.mp4
Note: The demo was recorded before the iOS 26 update. For the updated UI, check the User Interface section.
- Select the language you want to learn.
- Point the camera at an object.
- Adjust the detection box and anchor a label in 3D space.
- Tap the anchored label to reveal its translation and hear the pronunciation.
- Long press a label to delete it.
- Save the word to your personal collection.
| Framework / Component | Purpose |
|---|---|
| ARKit | Spatial tracking and anchoring labels in the real world |
| Vision + CoreML | FastViT image classifier for real-time object recognition |
| Apple Translation Framework | Accurate translations for detected objects |
| AVFoundation | Speech synthesis for pronunciation playback |
| CoreData | Local persistence for saved words and settings |
Note: All processing happens on-device for privacy and offline usability.
| Section | Screenshot | Description |
|---|---|---|
| Translate Tab |  |
AR initializes and communicates its status to the user. |
 |
Shows the app in Detection Mode. | |
 |
Shows the anchored labels visible in 3D space. | |
 |
Translation pop-up for "coffee mug" and options to Listen or Save (orange, unsaved). | |
 |
Translation pop-up for "laptop"; shows it has been Saved (green checkmark) and allows Listen. | |
 |
Triggered when detection mode starts but the selected language isn’t installed (edge case). | |
 |
Shown on the Translate tab when camera access is not granted, prompting the user to enable permissions. | |
| Saved Words Tab |  |
Vocabulary list saved by the user. |
 |
Vocabulary list filtering by language. | |
 |
Vocabulary list sorted by date added. | |
 |
Saved Word Detail View | |
| Settings Tab |  |
Settings tab for selecting language and color scheme. |
 |
Language selection sheet. | |
 |
Color Scheme settings. | |
| Onboarding Screen |  |
Shown when the user downloads the app and opens it for the first time (Onboarding 1/4) |
 |
Onboarding 2/4 | |
 |
Onboarding 3/4 | |
 |
Onboarding 4/4 | |
| Instructions Sheet |  |
Shown when the user opens the Translate tab for the first time after downloading the app, and also appears when the info button is tapped on the Translate tab (Instruction Sheet 1/3) |
 |
Instruction Sheet 2/3 | |
 |
Instruction Sheet 3/3 |
Planned improvements:
- Enhanced object recognition accuracy.
- iCloud sync for saved vocabulary.
- Gamified progress tracking and achievements.
Note: Lingo Lens works best on Pro iPhones/iPads with a LiDAR sensor. Placing anchors on objects may take a few retries on other devices due to hardware limitations, and I'm actively working to improve this experience.
Lingo lens was developed as the final project for the course MPCS 51030 iOS Application Development (Winter 2025) at the University of Chicago.
Developed by: Abhyas Mall
Project: Lingo lens
Contact: [email protected]
Lingo lens is under the MIT License