I led the design and development of a collaborative augmented-reality game called Brick. The game has been recognized as a pioneer in multiplayer AR. Brick was launched at the CHI 2019 Conference on Human Factors in Computing Systems in Glasgow, Scotland.
Verizon Open Innovation partnered with Carnegie Mellon University to explore collaborative futures in augmented reality.
Subverting the Cellphone Zeitgeist
Smartphones get a bad rap, and rightly so. We are a society of noses glued to screens. We breathe in the spaces between push notifications. In the evenings, seated across from friends and family, we infinite scroll and half-listen.
But it doesn’t have to be this way. Imagine a future in which our mobile devices don’t come in the way of being truly present. Imagine a future in which these devices actually enable the creation of meaningful shared experiences. A year ago, we took a stab at prototyping this future.
[Figure 1] Nicole Eisenman’s Prince of Swords was an early inspiration for the project. (Courtney Linder/Carnegie Museum of Art)
To establish a clear scope for the project, the team made two early decisions. First, we decided to focus on the dynamics between pairs of people who know each other really well, such as close friends, romantic partners, and roommates. From experience, we knew that smartphones were especially disruptive to in-person interactions between two people, and we reasoned that the stakes were highest when these two people were intimately acquainted.
Second, we decided to build a game—an exercise without any explicit goal other than to have fun. We reasoned that a game would be a natural activity for our chosen audience, and it would also afford us plenty of flexibility to set our own implicit goals for the experience. We conducted a round of research and brainstorming to figure out what these goals might be. Here’s where we landed.
A relaxed, light-hearted atmosphere fosters intimacy.
Stronger communication leads to deeper bonds between people.
A spike in heart rate increases feelings of intimacy.
Collaboration contributes toward building resilient relationships.
[Figure 2] The main design goals for Brick.
Designing for the Opposite of Immersion
Our chosen interaction medium was augmented reality (AR), an interface that places digital objects in people’s physical surroundings. At the project’s inception in early 2017, AR was an emerging technology that was largely restricted to single-user interactions. In our work, we wanted to push the envelope on AR and make it a social experience for colocated human users. In order to do this, our game needed to empower our users to be less focused on their devices and more attuned to their partners and the physical world around them. In other words, we would have to design the opposite of an immersive experience.
We conducted several brainstorms using the round robin brainstorming technique [link], and we generated nine different game concepts. We playtested these concepts in low fidelity and scored each against our chosen design goals. Eventually, we landed on a concept called Brick. Brick is a collaborative game in which two players work together to fll in a pattern of empty slots in a grid, using AR bricks that are scattered all over the room. Some of the bricks need to be collected individually, while others must be collected by both players working together. To win, the players must fll in the entire pattern before time runs out. At some point during the game, a bomb appears in the pattern, and the bomb must be defused collaboratively within seven seconds. If the bomb isn’t defused in time, it sets the players back by knocking of adjacent bricks in the pattern. Here are some photos of Brick’s low-fidelity prototypes.
[Figure 3] A low-fidelity prototype of Ice-cream Truck, a game in which two players have to serve up ice-cream orders together.
[Figure 4] A low-fidelity prototype of Leaky Faucet, a game in which players offer solutions to each other's problems.
[Figure 5] A low-fidelity prototype of Brick, in which players collaborate to complete a challenge. This is the concept we decided to move ahead with.
[Figure 6] A game diagram showing the actors, elements, and interactions in Brick.
[Figure 7] Notes from an early brainstorm about Brick's in-game interactions.
[Figure 8] In progress, a playtest using the prototype in Figure 3.
Building on the Shoulders of Giants
Our early prototypes and playtests established a clear runway for the game’s development. The high-fidelity versions of Brick were built on the Unity game engine using Google’s ARCore software development kit. Synchronous networking across multiple Android devices was essential for Brick to work; Brick’s players need to keep track of their own progress as well as that of their partner. However, establishing a networked environment turned out to be a thorny problem, with little documentation and few precedents in the community. Using ARCore Cloud Anchors, we were able to create a stable body of code to support synchronous networking for Brick and other games.
How to Play Brick
[Figure 9] Brick requires two mobile phones compatible with ARCore.
[Figure 10] Players see a pattern with empty slots. They must complete the pattern before time runs out. The timer is shown on the top-left of the screen.
[Figure 11] Players collect bricks and place them in the appropriate slots.
[Figure 12] Each player is responsible for bricks of the two colors assigned to them. The color assignments are shown on the top-right of each screen.
[Figure 13] Some bricks, such as the one on the left, are collected individually. Others, such as the one on the right, must be collected collaboratively.
[Figure 14] During the game, a bomb might appear in the pattern. The players must work together to defuse the bomb.
A New Paradigm for Augmented Reality
In the early months of 2018, when Brick was being developed, the ecosystem of AR games offered limited opportunities for a truly multiplayer experience. At the time, multiplayer AR games typically reduced to single-player experiences with an asynchronous communication capability across multiple devices. We wanted to push the envelope by using AR to connect people not just with their own physical environments but also with each other. To do so, we developed a framework for shared-world AR, in which players inhabit a shared, real-time augmented environment and can engage in synchronous and collaborative interactions with other players.
For a two-player shared-world AR system such as Brick, we prototyped and iterated on five categories of interactions (see Figure 10): single-player interactions, which play out between individual users and the AR interface; intrapersonal interactions, which involve an individual user’s internal deliberations and experiences; multiplayer interactions, which play out between multiple users and the AR interface; interpersonal interactions, which involve multiple users but not the interface; and environmental interactions, which involve users and their physical environment.
[Figure 15] The five categories of interactions in a shared-world augmented reality system: (1) single-player, (2) intrapersonal, (3) multiplayer, (4) interpersonal, (5) environmental.
[Figure 16] An illustrated example of a single-player interaction: (A) pick-up, (B) carry, (C) drop-off.
[Figure 17] An illustrated example of a multiplayer interaction: (A) One player picks up, (B) Both players pick up, (C) Both players carry.
Trials by Fire
We playtested Brick with 14 participants, recruited in pairs to ensure that gameplay partners knew each other. Each session included a short introductory conversation, then multiple rounds of gameplay, and fnally an interview about the gameplay experience. The introductory conversation inquired into participants’ prior experience with AR and the players’ relationship with each other. Gameplay began with a short description of the rules, followed by multiple rounds of the game. The post-game interview was an extended conversation with four sections: general gameplay, inter-player dynamics during the game, main takeaways from the game, and an exit debrief. The questions in the post-game interview were adapted in part from Shawn Patton’s work at Schell Games [link].
1. What was the most frustrating moment/interaction in the game?
2. What was the most delightful moment/interaction in the game?
3. What was something you wish you could do during the game but couldn't?
4. If you had a magic wand, what would you change about the game?
Here are some snippets of footage from our playtests.
[Figure 18] Players pick up pieces by tapping them with a finger and holding the finger down.
[Figure 19] Players release pieces by lifting their finger.
[Figure 20] When a piece is dropped off close to an appropriate slot, it drifts into place automatically.
Design for simplicity. Our fndings indicate that phone-based mental models of interaction overrode physical models early in the game. For example, one participant said, "My first instinct was to tap on things. I didn’t know that I had to approach things." Even after learning how to play, players needed ongoing technologically mediated cues to move their bodies.
Design for physical proximity. When players are colocated, a shared-world AR system ofers numerous opportunities for physical proximity and contact. Brick’s design forced players to stand close to each other during gameplay, and some of our playtesters went so far as to expect collaboration to be associated with physical proximity. For example, one player said, "I can hold [a collaborative brick], and as soon as you’re holding it as well, that’s when we pick it up."
Design for exertion. Our fndings indicate that designers should account for exertion and fatigue while creating room-scale shared-world AR experiences. In the case of Brick, many players experienced a spike in heart rate while collecting bricks from around the room. While designing specifc interactions in AR, both gross-motor involvement and frequency of repetition may be used to intentionally control difculty and fatigue.
Design for the ecosystem. In shared-world AR, single-player and multiplayer interactions had intrapersonal, interpersonal, and environmental implications. For example, the apparently single-player interaction of collecting an individual brick involves intrapersonal cognition, in which a player comprehends the gestures necessary to pick up, carry, and drop of the brick; interpersonal conversation, as when players help each other locate bricks during gameplay; and environmental awareness, since players navigate the physical space as they move.
[Figure 21] A short video showing the highlights from a session of gameplay.
[CHI 2019 paper] Po Bhattacharyya, Radha Nath, Yein Jo, Ketki Jadhav, and Jessica Hammer. Brick: Toward A Model for Designing Synchronous Colocated Augmented Reality Games (in press).
[CHI 2019 demonstation] Po Bhattacharyya, Radha Nath, Yein Jo, Ketki Jadhav, and Jessica Hammer. Brick: A Synchronous Multiplayer Augmented Reality Game for Mobile Phones (in press).
[Open-source code] Synchronous networking using ARCore Cloud Anchors, and custom network transforms for augmented objects, available on Github.
[Client deliverable] Product strategy for developing AR experiences for the home (confidential).