A.I. as Auteur: Interactive ‘Frankenstein’ Experience Shows How Artificial Intelligence Could Be Storytelling’s Future
Each year, members of MIT’s Open Documentary Lab take a look at the interactive works at Sundance’s New Frontiers section. This dispatch explores the storytelling potential of artificial reality.
The projects in this year’s New Frontier section at the Sundance Film Festival ranged from the latest cutting-edge virtual reality to decidedly low-tech dance and theater performances. The most thought-provoking of these pieces incorporated media forms across the technological spectrum to explore the leading scientific development and debate of the day: the ascendance of artificial intelligence (A.I.).
As economists and social scientists debate the ramifications of A.I. on their respective disciplines, artists, unsurprisingly, are feeling compelled to do the same. Accordingly, the pieces at Sundance chose not to question issues like the future of work but rather cut to the philosophical core of the matter: in the era of AI, what does it mean to be human? As a recent Medium piece asked, riffing on critical theorist Walter Benjamin: What is the work of art in the age of algorithmic reproduction?
Combining installation, performance and a mobile augmented reality (AR) game, the piece “TendAR” probes the limits of facial recognition software and A.I. In the experience, two users are asked to help test a human emotion recognition software for marketing purposes. The format of this test is a mobile AR game in which a cutesy, A.I.-powered fish prods the users to help it learn how to identify objects and human emotions, until everything goes awry.
In a panel discussion during the festival, creator Samantha Gorman said that the piece was conceived of as a parody that sought to use comedy to question how A.I. related to human emotion. She said she and her co-creator Danny Cannizzaro wanted to imagine a future in which humans are neither the slaves nor slave masters of robots, rather something like their “cherished pets.” Compelling in theory, the ambitious premise was somewhat muddled in practice, a not-uncommon problem for similar genre-pushing projects. (Whether the fish is more aggravating than adorable is also up for debate).
More problematic, however, was the issue raised during the panel by New Frontier advisor Lisa Osborne that, in her experience, the A.I. fish was unable to recognize facial expressions from black skin. While she didn’t respond directly to the concern, Gorman suggested some of the software’s recognition difficulties may have stemmed from lighting issues. In any case, that a piece that seeks to investigate the societal implications of biometric data and A.I. may be (unintentionally) perpetuating the widely-discussed problem of algorithmic bias gives some pause. There may be time to sort out its kinks, however, as Gorman said “TendAR” will be available as an android app in April.
Perhaps the most ambitious piece at New Frontier, A.I. or otherwise, was the multifaceted transmedia work “Frankenstein AI: A Monster Made by Many” by Rachel Ginsberg, Nick Fortugno and Lance Weiler. Billed as a “participatory installation and performance,” the total experience, which will tour this fall, spanned over two hours and involved elements of theater, dance, collaborative storytelling, philosophy, Mary Shelley and, of course, artificial intelligence.
Inspired by the 200th anniversary of Shelley’s “Frankenstein,” the creators invented their own naïve monster — an A.I. who is seeking to understand humanity. Designed as an experience in “three acts,” the project has users share deeply personal memories and emotions — first in pairs, then in a small group, and lastly, in a single performance that took place on January 23, as part of a large audience — so that the AI could “learn” what it means to be human.
The resulting data from these interaction was fed into the machine, combined with a corpus that includes the text of Shelly’s “Frankenstein” and other human and online information, and then parsed for sentiment. Based on this information, which it assigned one of 12 pre-programmed emotional states, the machine asked the group questions ranging from “How do you talk to someone to fall in love?” to “Why do humans kill each other?” The answers were fed back into the machine, processed in real time by the A.I., and the cycle continued.
While not all users may understand the science behind the project, they will likely be swept up in its imaginative theatricality, an aesthetic that could be described as “futuristic Gothic.” The set pieces include a dimly lit room where red velvet chairs ring tables outfitted with touchscreens that resemble a Ouija board married to an iPad; a dance performance of “algorithmic choreography” that mimics the process of machine learning; and the physical manifestation of the monster itself: a large plexiglass tank filled with smoke and abstract visualizations of code and projections of bits of imagery culled from the internet.
The Q&As afterwards served to demystify some of the process and worked towards the project’s overarching goal to broaden public participation in the debate about what an A.I. future looks like outside the tech world — before it’s too late. The narrative device of “Frankenstein,” Ginsberg said in the panel conversation, was a timely and resonant metaphor. The machine, as in the novel, is meant to hold a mirror up to humanity. The “monster” of AI, Ginsberg said, is neither utopian nor dystopian and needs to be understood, not feared.
In that vein, filmmakers should not feel that their auteur role is threatened by A.I., both Ginsberg and Gorman said. In their view, A.I. should be seen as a useful collaborator, rather than usurper, whose artistic value is in what we learn about being human through interacting with machines.
To illustrate the point, despite the piece’s impressive technology and creativity, the most memorable moments emerged from the group dynamic involved in “teaching” the machine the meaning of the human condition. It was hard to decide what felt more absurd – the machine’s need to understand love, or the fumbling attempts by a group of mostly strangers to explain it.
At one point in my group the A.I., seemingly fed up with our inadequate explanations of human life, cut to the chase and asked: “What does it all matter?” After a pause, one man exclaimed to laughter, “That’s your job!” before we all went silent. No one, human or machine, could muster a sufficient answer to that one.
Sara Rafsky is a freelance journalist and researcher at MIT’s Open Documentary Lab.
