Introducing Squiggly! Squiggly is a visionOS 2 application that explores drawing with physical utensils (crayons, pens, markers, and brushes) in immersive spaces. It is developed with RealityKit and SwiftUI in addition to the machine learning frameworks: BERT-SQuD, MNIST, and Apple's Vision Framework. This research prototype explores spatial user interfaces (UI) and graphics for 3D environments.
The physical objects, including crayon boxes (new box, broken box, condensed box), the individual crayon colors (yellow, blue, purple, orange, red, green, and pink), a stuffed Pokémon plushie, another 3D objects are trained on 3D models and collections of images before importing to CreateML.
- XCode, Reality Commposer, CreateML
- Hardware: Mac (ideally a computer with an M3/M4 chip) and an Apple Vision Pro
- CoreML (BERT-SQuAD, MNIST)
- Apple Vision Framework
This application can only be run on an Apple Vision Pro with visionOS 2.0 and higher. Created with an Apple Developer account (no enterprise APIs).
Install XCode beta (Version 26.0 beta 2) and visionOS beta. You will need to be running macOS Sonoma (version 14.0 or later).
Install GitLFS
git install lfs
Fork the repository and then clone the forked repository
$ git clone https://github.com/debbieyuen/spatialui.gitUpdate submodule folders to retrieve ML and Reality Composer files
git pull --recurse-submodules
git submodule update --init --recursive
Pull all LFS objects
git lfs pull
git submodule foreach 'git lfs pull'
Retrieve your Access Token for your app from this page. In your terminal, add your token to your environment variables.
$ export VISION_TOKEN='xoxb-XXXXXXXXXXXX-xxxxxxxxxxxx-XXXXXXXXXXXXXXXXXXXXXXXX'This lab course will be presented at SIGGRAPH in Vancouver in August 2025. Special thank you to the teams at Apple and NVIDIA for helping us make this all possible.