Guide

What is the challenge with prototyping smartwatch interactions?

I think of the Apple Watch as the little sibling to the iPhone. It was born eight years later. In the first few years it was dependent on the iPhone to do the heavy lifting. It was unable to process demanding applications in its own casing. Until recently, it could not speak for itself. It came without a SIM and channeled all connectivity through its older sibling.

Camera Remote app is typical of the device’ original role as an extension of the smartphone

How shall we approach prototyping smartwatch interactions?

To this day, the smartwatch is less mature and capable than the smartphone. In some cases, it extends another personal computer, rather than being one in its own right. With its touch screen and crown wheel, to the iPhone it acts as a secondary output and input device, much like a screen and mouse to a desktop computer.

When we design for an aspirational future, we may be inclined to treat the smartwatch as a stand-alone device. Imagine a scenario where the person you design for can leave their phone at home. To a certain extent this is already possible.

Some of the following approaches lean towards treating the smartwatch as a wearable personal computer, others towards treating it as an interface and extension to the smartphone or even desktop computer.

1 – Big sibling for help

Since the smartwatch was born as the little sibling of the smartphone, we may take the more mature smartphone as a substitute device for prototyping. They have many sensors and actuators in common.

Figma

Can we just use the most common UX/UI design tool to prototype smartwatch interactions? Figma is not only suitable for designing assets for the graphic user interface. It features a Watch OS app called ‘Watch Mirror’. This app supports no interactivity on the device. It merely mirrors, as the name suggests. Whereas Figma wireframes are clickable on the smartphone, not so on the Apple Watch. You can think of Watch Mirror as a testing tool for the appearance of graphic user interface assets.

Figma on smartphone is clickable, Figma Watch Mirror is not (von Loeper, 2023)

Play (Beta)

Can we create interactions that feel like a native app? With Play, we can create functional iOS apps and test them on a smartphone, without writing a line of code. As of today, Play does not support WatchOS. It is a fairly new app and so far, the functionality is very limited. We can hope for such no-code tools to evolve further and include smartwatch apps in the future. I suggest to watch out for them.

Building an iOS app in Play (Play, 2023)

Origami Studio by Meta

Which other design tools allow us to easily access the device’ sensors and actuators? Meta’s Origami Studio enables us to prototype a variety of smartphone interactions. This app can help verify interactions to be carried over to a smartwatch.

Pinching gesture tested with Origami Live (von Loeper, 2023)

2 – Strapping things to your wrist

Any smartwatch draws on the tradition of wearing a bracelet with interface on your wrist. We may emulate part of this experience without using an actual smartwatch.

Wear a smartphone

When Apple developed their first generation smartwatch, reportedly their designers strapped an iPod to the wrist. Instead, we can strap a smartphone to the wrist, as many Delivery Riders do. QuadLock offers phone cases and a sports armband with a twist lock. Pretty neat.

Smartphone worn with QuadLock armband (QuadLock, 2023)

Strap hot keys to your wrist

As with the Apple Camera Remote app, there are situations where the smartwatch acts as a remote control. Consider Elgato’s mobile app as a prototyping tool and strap the phone to your wrist.

Hotkeys are versatile and easy to configure. You drag and drop buttons from a repository directly onto the deck. You can create your own actions from scratch. They can replace key combinations that are already defined in an app or you may come up with your own web app, define keys and take control from your wrist.

Elgato Stream Deck mobile app controlling Figma with hotkeys (von Loeper, 2023)

Build a smartwatch interface with hand-picked hardware

Why not build my own smartwatch interface? I can pick a few hardware components and connect them with a microcontroller.

Absolutely! But it may get complicated and you need to verify the compatibility of all components. The delivery time, possibly from far-away places, may delay your project. For most cases, I would not recommend you to build your own smartwatch interface with a long bill of materials, but focus on what you really need. If you want to test part of a smartwatch’s functionality, such as dimming a light from the wrist with a rotary knob or scrolling down a pitch deck: That seems fairly easy, once it is set up.

When it comes to connecting hardware components with a visual interface, two approaches convinced me. If you prefer a setup that feels more like a design tool and requires little or no coding, you may consider ProtoPie. You can create UI assets in a UX/UI tool of your choice, such as Figma, carry them over to ProtoPie and add interactivity. This approach is open and ProtoPie Connect supports many integrations, such as Blokdots, Haptic Labs and many components.

Alternatively, you may look up Alex’ approach to prototyping physical interactions with visual interfaces. He describes a setup with a NodeJS backend and a React frontend that allows you to control a web-application with any sensor that you connect via a microcontroller and vice versa. The web app can control any actuator connected via the microcontroller. You don't need to fiddle with Processing or other embedded languages. You can write the web app with html, CSS, JS and add to it as you want.

For output you can order a small screen, strap it to your wrist and connect it as a secondary display to a small Windows laptop that you may hide in a backpack. A thin HDMI cable for signal and a USB cable for touch input can be threaded through any sleeves the test user may wear. This is likely to end up bulky and uncomfortable.

For input you can order a microcontroller that acts as a keyboard, connect any button or potentiometer and send signals to your PC via USB cable or wireless RF transmitters with fitting receivers. With Apple computers it is possibly risky to take over the keyboard. Preferably, you use a Windows PC and add a function to exit the program and reset the keyboard takeover.

A DIY interface for a wearable prototype may end up bulky

3 – Proper ways

Let’s look at the two most common operating systems for smartwatches: Google’s Wear OS and Apple’s WatchOS. To my knowledge, currently we lack a no-code design tool that would allow designers to create native apps for Apple smartwatches.

ProtoPie Player for Google’s Wear OS

For Google’s Wear OS there is a ProtoPie Player that enables you to build an interactive interface. You can use the player application to emulate the behaviour of a new app. This approach benefits from ProtoPie’s ease of use and its ability to import UI assets from others tools like Figma. Instead of following the logic of a real app, it follows the logic of ProtoPie, with its constraints. Common functionalities, UI assets and APIs have to be built from scratch. The ProtoPie ecosystem makes it easy to connect to other devices, things, and interfaces.

If you want to prototype apps for Wear OS and overcome the constraints of ProtoPie, you may look out for the Wear OS development framework.

ProtoPie Player for Wear OS (ProtoPie, 2023)

Write your own app in Swift

SwiftUI is Apple’s framework for app development. It is convenient to integrate existing components and connect to APIs from the Apple ecosystem. I find it is the easiest way to work with real-time sensor data from the device. You may hook into the signal stream of the gyroscope or compass. You can use the crown wheel as input, all while benefitting from the low-latency taptic feedback that makes the Apple Watch feel immediate and second to nature: almost part of your body.

Initially, writing code in Swift does not feel like using a design tool. The editor and development environment, X-Code, requires you to pay close attention to errors and declare permissions as if you launched an app on the market, even if you run it on your personal device only. In order to install an app on your smartwatch, you need to connect your iPhone with a cable to the computer on which you are running X-Code. When you disconnect your phone or quit X-Code, sooner or later the smartwatch app will uninstall itself and disappear, unlike apps that you install on iPhone. This unexpected constraint occurs in developer mode. As a result, you can’t easily walk around with your app and test it in daily life.

Writing Swift is fun and you can get a little help from the documentation, other projects, or Chat-GPT. I second Apple’s verdict that common ideas are easy to realise while uncommon ideas are possible. SwiftUI enables you to prototype and design apps – and even complications for the watch face. The Apple Watch still works as usual: It displays the time and sends notifications. You can receive another level of feedback, when a prototype feels real and people relate to it as a product that could soon hit the market.

Panning gesture for ‘Nectar’ app written in Swift (von Loeper, 2023)

4 – Do more with your smartwatch

It does not always require a designated tool to prototype interactions on the device. Here are a few work-arounds for the Apple Watch.

Browse a web application

When so few prototyping tools support the smartwatch, why not make a web app?

The Apple Watch comes with a hidden browser and other third-party browsers are available as well. To open a website on your Apple Watch you can send yourself a weblink, for example via iMessage. You can create a website in any way you can think of. You may use a no-code tool like Webflow or you put together a few lines of code, add them to a repository and deploy a website with for example Vercel.

How capable are web apps compared to WatchOS apps? Currently, browsing on the Apple Watch is very limited and the browser is hidden, as if Apple did not want it to be used at all. When you browse websites on the Apple Watch, you get surprised both by what is possible and what is not. Try to navigate your own portfolio on Apple Watch or click through your favourite web apps. Many websites respond well and it is fascinating to browse content in this aspect ratio and size. As the wrist gets tired quickly, the watch is less suitable for watching content for extended times. The browser does not support all file formats and integrations. In my trial, videos did not appear. Certain elements fall apart or behave differently. I noticed a custom font did not load properly. An element that was fixed in CSS did not stay in place while scrolling. This approach limits interactivity.

When browsing websites on Apple Watch, the menu bar takes away screen real estate for the content. If someone would design a web browser for Apple Watch that supports full screen mode, that could be beneficial for prototyping.

Browsing the Pokédex on the wrist (von Loeper, 2023)

Shortcuts

Can my smartwatch bark?

This should be fairly easy, I thought.If the smartwatch interaction you want to prototype essentially consists of one button that triggers a custom action or set of actions, you may try Shortcuts: Apple’s visual scripting application to create macros. Siri serves as an alternative way to trigger actions and automations are supported as well. They allow you to trigger actions based on events. Alternatively, try IFTTT on Apple Watch. As Shortcuts, this app allows you to create applets and trigger actions without opening the respective app. After all, many applications are synced across devices and the Apple Watch merely acts as the interface at hand or as the registration of your presence in the digital realm.

You can think of both tools as dashboards to combine actions from different apps in a new way. In theory both seem promising. In practice they come with constraints and poor implementations – on a case by case basis. Once again, on Watch Shortcuts are more limited than on iPhone.

Failed attempt to play recording of a barking dog from the wrist, while it works from the iPhone (von Loeper, 2023)

Video-prototype

In case you don’t need real-time interactivity, you may consider installing the WatchOS app Videwatch to play video. This way you can quickly verify the appearance of micro-interactions or wireframe flows in full screen on an Apple Watch.

Micro-interaction, unlocking the bike (von Loeper, 2023)

Connect peripherals

Bear with me. I may be the only person interested in this: Do you remember that glorious Spotify Car Thing? I thought: How would a Spotify Bike Thing integrate into a handlebar? How would it feel to switch songs and regulate volume while riding a bike? And since I like to bike with empty pockets, I may as well leave my phone at home. Can we treat the Apple Watch as a stand-alone computer? Can we connect a mouse, keyboard, or other peripheral to control the Apple Watch?

If you need the possibility to control an Apple Watch app from a tangible user interface that behaves as a peripheral to an Apple Watch, you can use an Apple Remote. Connect it to a Mac and wirelessly sync the controls across the apps instances on different devices. Since modern MacBooks don’t receive IR signals, you may need to add an external IR receiver, such as the Flirc USB infrared receiver. In this example, I mapped out the keys of a 2011 Apple Remote to the pre-defined keys in Spotify to play/pause tracks, skip forward/backward and control volume. To unassuming eyes, it seems as if the remote controlled the WatchOS app directly from my bike’s handlebar.

In the future I hope for proper ways to connect a smartwatch with peripherals, directly. This could open up for more scenarios where the phone can be left at home.

Spotify bike thing (von Loeper, 2023)

5 – Break out of technical constraints

For the major part of this article I looked out for technical constraints and possibilities. Restricting yourself to current possibilities may not always be the best way.

Lo-fi prototype for enactments

If you want to explore ideas more freely, you may engage in enactments with lo-fi props. This approach can be suitable for ideation of gestures. In the project ‘Cocoon’, my team and I conceived a virtual membrane to regulate availability in the context of remote work settings. This could have been done with a smartwatch, but we wanted to do without a screen. Following a first iteration with cardboard, we cast a wristband out of silicone. AfterEffects served to visualise the effect of the pinching gesture that opens and closes the membrane.

In a second iteration, this gesture can be verified and adapted with the interface of an existing smartwatch. Two years after this project, Apple introduced the tapping gesture, which does not require a touchscreen or crown wheel as input. When you deal less with technical constraints, you can put more emphasis on other aspects of design, such as exploring the context.

Enactment of pinching gesture for wearable devices in the setting of a home office (Hulling, von Loeper, Lu & Shivaraj, 2020)

Conclusion

I hope this guide provided you with an overview over different approaches to test ideas for smartwatch interactions and you can learn from my own moments of failure and success. Rather than choosing one approach over the other, often it is about cleverly combining them. I consider SwiftUI a design tool disguised as a development tool. It enables you not only to realise and test ideas, but take them to the next level of development – to launch an app. This can benefit your own independent competence as a designer and increase your empathy and sense of realism when you collaborate with developers.

In this article I considered the technical constraints and possibilities. Other constraints and possibilities stem from our conception of this device category. I am thinking of writing another article about the future role of smartwatches. The name ‘smartwatch’ suggests that this device is meant to be attended to, but the greater opportunity lies in sensing and adding meaning to what is already happening. This would allow people to immerse themselves in their ‘natural’ environment with their heads up and their hands free.

Please do not hesitate to reach out with feedback or questions!