Machine Learning Researcher

Machine Learning Engineer, September 2022 - April 2024

BMOLab, Dir. David Rokeby, University of Toronto, Toronto, Canada

Description:

Dir. David Rokeby and I explored ways to apply deep learning to the performing arts in interesting and novel ways.

In this position, we worked on using freeze maps and developed a system to use an archive of latents for the VoiceScroll project. This allowed the final system to have an interactive component for the user to decide where the model was generating on the canvas without breaking continuity.

After that, I worked on finetuning the Llama 2 models for generating plays in the style of playwrights in real time. The work targeted the works of Shakespeare as a backbone. I accomplished this by developing a new weighted sampling mechanism for the dataset, to make up for the small quantity of data being finetuned to.

Finally, I moved to exploring the use of diffusion models and the VoiceScroll project for real time audio generation using freezemaps, based on Riffusion.