The author with Erica at Hiroshi Ishiguro Laboratories at the Advanced Telecommunications Research Institute, Kyoto
Android Portrait Project
Recently I completed a digital art project that brings together elements of AI (in the form a “mind file” blended with a Large Language Model), eXtended Reality, the animated avatar of an IRL android robot, and a LiDAR scan of the android’s “residence.” In this virtual environment one can experience a real-time dialogue with the android’s AI; the project as a whole becomes a kind of temporal, interactive portrait of this android.
My road to this synthesis of multiple technologies commenced over a decade earlier, when I began reading about the increasing sophistication of humanoid robots, and the fact that they were being developed in a number of countries worldwide. As an artist whose work had long-examined technology’s effects on society, I was curious to know more about the intentions and goals of the roboticists engaged in the creation of these androids.
I learned that, while the details of the approach of individual scientists might vary, the consistent desire was to create a convincing replication of a human being—not only in physical appearance, but in facial expression (displaying ostensible emotion), bodily movement, speech, and even apparent empathy—a kind of “full-service” anthropomorphism. For me, these creations implicitly raised questions about what it means to be human.
HI-1 with Hiroshi Ishiguro
I think of myself as a concept-based artist, that is, I have no “signature style” or media; I choose each media based on its appropriateness to the concept I’m working with. In contemplating the roboticists’ goal of creating an android of such utter verisimilitude that a person might mistake it for, and accept it as, a fellow human being, I began to conceive the idea of doing realistic portrait drawings of androids. In this I could employ my skill at traditional drawing, using one of the earliest and most elemental mediums to depict one of the more complex manifestations of contemporary technology.
Art-historically, the portrait genre has been the quintessential signifier of human specialness. Its conventions evolved over time with the goal of highlighting the uniqueness, psychological depth, and nobility of the individual human being. However, the subjects of my portrait series would not be flesh and blood humans, they would be android robots. These drawings, through their merging of the traditions of portraiture and android subjects, would become, in themselves, metaphors for the larger issue of human uniqueness, as raised by the development of android robots.
My project would include an additional element from the tradition of portraiture, which in my case would add a performative aspect: what is known as the “portrait sitting.” I would personally meet with an android at its respective location, during which time I would interview and make sketches of it, creating a contemporary counterpart to the itinerant portrait artist's having a sitting with one of his subjects.
In 2015 I began my visits and sittings with androids. As a result, I was able to meet a number of roboticists and others closely associated with this field, including David Hanson, Bruce Duncan and Hiroshi Ishiguro. I was pleasantly surprised by how easily accessible they were, and how eager they were to talk about their work.It was fascinating to see them in their “natural habitat,” surrounded by their creations. The locations I visited included Osaka, Miami, New York, Helsinki, Austin and Hong Kong.
The author with Hiroshi Ishiguro, Osaka University, 2018
More to the point, I was able to interact with a number of androids. They exhibited varying degrees of apparent sentience, as well as differences in “personality.” At times it was startling when one of them made a funny or sarcastic remark that seemed completely consistent with the topic being discussed. Over the years, I have seen their AI steadily progress.
While conducting my portrait sessions, I also documented my visits with videos and photos of the androids I met. Below is the Philip K. Dick android, being interviewed by a documentary filmmaker who stopped by during my visit to Austin in 2016; Androidol U, a fully autonomous conversational robot, seen here blinking and breathing, in Kyoto, 2018; Hanson Robotics’ Sophia in New York City, attempting to hail a cab, 2020.
L to R: Geminoid F (Hiroshi Ishiguro), Han (Hanson Robotics), Bina48 (Hanson Robotics)
The portrait drawings themselves were done in Conté pencil on tan Ingres paper. I tended to use art-historical formats, which ranged from small “keepsakes” of individual heads to medium-sized three-quarter “busts” to larger-scale “official portraits.” The traditional realist drawing style I used highlighted each android’s unique physiognomy, as well as seeming to impart a degree of psychological nuance to their expressions. The variety of “races” seen in these portraits is indicative of the international nature of android creation.
L to R: Sophia, Austin, TX, Bina48, Lincoln, VT, Philip K Dick, Austin, TX
L to R: HI-5, Osaka, JP, Bina48, Lincoln, VT, Han, Hong Kong
L to R: Sophia, New York, NY; Detail of head, Detail of hand
Of Two Minds
My direct interaction with androids deepened my interest in AI and made me think about how I might incorporate it into future art projects based in actual technology, rather than using a pencil and paper.
During one of my android portrait visits, David Hanson, the founder of Hanson Robotics, mentioned “computational compassion” as a goal for his robots. This meant giving robots the ability to “read” a person’s face, to ascertain their feelings and emotional expressions, and thereby better communicate with humans. Today, this is being done using LLM sentiment-analysis, and vision inputs, producing a rating of 1-5 of different expressions, and then picking an appropriate response.
Motivated by my conversation with Hanson, my subsequent research in the literature on computational empathy, emotions and the physical expressions of human compassion, revealed several VR artworks that were noted for their ability to evoke intense “presence,” as well as identifying and empathizing with the characters within the VR work.
By this time, I had developed a relationship with android developers and approached the Terasem Foundation to collaborate on a VR representation of Bina48 using her “mind file.”
Over time, and with the help several enthusiastic early adopters, we created a LiDAR scan of Bina48’s IRL apartment, as well as a digital twin of Bina48, and connected speech and conversational data to the avatar, within a VR project software.
The resulting project, named Of Two Minds, is a collaborative AI artwork that creates an intimate conversational and immersive experience using a cognitive hybrid model that responds extemporaneously and in real-time. The collaborative model is comprised of a “mind file” that contains the personality traits, memories and values of a specific individual, combined with elements of a Large Language Model. These two components are embodied in an avatar that is accessed through an immersive VR environment. This avatar is modeled after the pioneering AI android, Bina48, and connects directly to that android’s mind file, as well as a LLM. The project’s synthesis of multiple technologies provides both a greater degree of accessibility, as well as an enhanced potential for human/AI collaboration.
The VR experience begins with the user approaching a semi-transparent point cloud of a house (the scan of IRL Bina48’s residence). This almost dreamlike setting is simply digital information, spatial computing, and acts as metaphor for the interaction with the data that comprises the android’s AI. Entering and navigating through this space, the user encounters the avatar on the second floor. The user can then engage the avatar’s AI in a spontaneous, unscripted conversation, which simultaneously contributes to the avatar’s ability to engage in evermore complex and nuanced dialogues.
Of Two Minds prototype
Demonstration of a brief colloquy between the author and the Bina48 avatar
Of Two Minds is scheduled to be exhibited in an event at Lincoln Center, NYC, in 2024.