Facilitating the Design, Production and Research of Human-Centered Mixed Reality Experiences

I work and live on the unceded territories of the Musqueam, Squamish and Tsleil-Waututh Nations. My teaching, research, and service are rooted in an ongoing inquiry to understand how technologies shape our interactions in the interconnected digital and physical spaces we inhabit. I embrace the multifaceted persona of an educator, researcher, and connector. My roles—be it as an xR researcher, project-based educator, sound designer, writer or technology developer—are driven by ceaseless experimentation and an interdisciplinary spirit that seeks collaborations across disciplinary boundaries. Each endeavour, regardless of its centered discipline, is underpinned by a critical and ethical approach to the development of emerging technologies. This approach ensures that potential implications and responsible usage are considered from the outset and are ingrained within the design process. Technology development serves a greater purpose: to improve lives and contribute to the greater good.

To effectively realize my vision, I maintain active involvement within diverse communities of practice, including theatre, dance, xR, and digital media. That involvement helps me to forge bridges between colleagues at UBC, industry professionals, and peers at other universities. I offer service to the university by linking technology development to both teaching and research, and by facilitating interdisciplinary connections across departments and faculties. I bridge scholars from a broad spectrum of disciplines at UBC, including Medicine, Psychology, Law, Music, Forestry, Biochemistry, Computer Science, Engineering and Biomedical Visualization through my work at the Emerging Media Lab.

Types of Research

The types of research I engage in can be categorized under the umbrella of research creation, applied and experimental research as well as educational research. A larger scale example that integrated several types of research creations and experimental research processes within a mixed reality experience was the Fun Palace: Carnival of Mixed Realities produced in Vancouver.

The Fun Palace: Carnival of Mixed Reality debuted at the Centre for Digital Media

Recent Research Creations

I am currently leading several research creations in collaboration with UBC Library and the Emerging Media Lab (EML), leveraging artificial intelligence, motion and volumetric capture studios to catalyze the creation of new xR works that explore Shakespeare characters and scenes across different virtual stages. As a Faculty in Residence and Project Research Consultant at the EML, I direct my passion and expertise towards developing and supporting the exploration and advancement of emerging technologies such as VR, AR, MR and Artificial Intelligence to solve human problems. The collaboration with EML has resulted in a national IT award recognizing our collaboration in the development and implementation of a project-based course focused on developing emerging technologies.

An introduction to the Emerging Media Lab at UBC

Sound Research Creations

Other research creations are focused on the design and composition of sound for physical and mediated productions. Currently, I am involved in several interrelated research creation projects exploring the corpus of Shakespeare’s First Folio to create new meanings using VR, Augmented Reality, AI and interactive touchtable design. Recently, I completed Shakespeare in the Metaverse in collaboration with volumetric capture studio Departure Lounge and VR Theatre focused Kreis Immersive that debuted a public prototype at BCIT’s Tech Collider in July 2023. That collaboration also involved numerous Centre for Digital Media graduates and newly graduated UBC Theatre and Film students along with research support from UBC’s Emerging Media Lab.

A previous and noteworthy research creation debuted at the Vancouver Art Gallery in 2022 for 8 months in an installation entitled The Imitation Game: Visual Culture in the Age of Artificial Intelligence where a number of mixed reality sound experiences were featured. These include customized Hypersonic speaker technology mounted on a motorized and programmed system that is automated to respond to human movement and direct sounds on a tight laser path in order to draw attention to specific locations in the rotunda at the Vancouver Art Gallery. A more complete picture of different sound research creations can be found here: sound research creation page.

Hypersonic Speaker at the Vancouver Art Gallery

The results of different collaborative efforts are disseminated internally at UBC, and to intersecting communities of practice including education, the live, blended, hybrid and virtual performing arts, the digital media industry (games, xR, VFX, animation) in addition to the field of biomedical visualization

Customizable virtual anatomy lab first prototype at the xR Prototyping Lab @ the CDM scaled at Hive lab @ UBC and deployed to students internationally during the Covid-19 pandemic.

Membership with the Association of Computer Machinery’s Special Interest Group on Computer Graphics (SIGGRAPH), the largest and most renown computer graphics conference internationally, has resulted in musical composition to accompany animation featuring Taiko drummers, moderation with industry experts, committee membership on the VR Theatre, Art Paper and General submissions groups, sound design for the VR theatre, Birds of a Feather symposia, and publication in Art Paper proceedings in addition to Leonardo’s MIT journal.

Teaching

The success of these research projects is, in part, dependent on the facilitation and mentoring of a diverse collective of undergraduate and graduate learners. Attention is paid at every step in the learner experience to ensure diversity in team make-up, provide equal access to technological resources, integrate supplementary course materials from a wide range of authors and creators, and attend to the individual learning needs of each student. Through every course that I teach, students learn 21st-century competencies such as collaboration, critical thinking, and self-regulation. These competencies are activated through their experience of researching, co-constructing, and reflecting on projects that focus on the design of audience and user experiences. Leveraging the transformative power of human centered design, I mentor and facilitate design thinking tools for teams of students in the co-development of immersive installations, AR, and VR experiences. Projects that explore audio also integrate spatial sound to further enhance a sense of immersion. My passion for a critical, equitable and practical approach to project-based learning has resulted in a recent book to support a growing number of sessional instructors who come from the digital media industry and want to share their knowledge in post-secondary settings. The learner-centred approach to teaching has also been shared at international conferences such as SIGGRAPH and South by South West (SXSW).

Service

My research is not separate from what I teach. My service is equally directed towards supporting students in the development of their careers, expanding collaborative research relationships between students, staff, and faculty within the university, and initiating professional, service and research contracts between UBC departments and units, and companies in the digital media industry in Vancouver, Canada and internationally. Some examples include Sawmill Motion Capture, Departure Lounge Metaverse, Arts Club Theatre, Small Stage, Vancouver Art Gallery, OPPO, and Kreis Immersive VR Theatre. Early exposure to members of the digital media industry has provided learners with mentoring opportunities, internships, coop placements and work-integrated learning. Maintaining ties with alumni who have transitioned into emerging technology driven companies has been an ongoing priority.

Students who develop technologies to solve human problems in courses develop prototypes that can then be used for research. Development of emerging technology has two integrated outcomes; it results in students developing competencies that will be essential as they transition into collaborative communities of practice while contributing to the advancement of research creations in the form of different media that also act as a focal centre of research. I impart knowledge about emerging technologies, cultivating in my students a deep understanding of the field, and equipping them with skills that are essential for navigating and influencing the rapidly evolving technology development landscape. This, while mentoring them on small teams to co-construct emerging technologies that result in proof-of-concepts that can then be further developed as research objects. For example, students in work-learn positions are mentored to co-create a Shakespeare augmented reality project in collaboration with UBC Library and EML. They learn how to develop the technology required to make it work, and the research creation that results can then be used to design human experiences that can be further investigated and used to apply for internal funding or external tri-council funding.

"I am more than the sum of my genetic parts. Soy más que mexicano, canadiense irlandes. No soy solo un maestro sino un estudiante de transformación."
Sample Music