Hello! I’m a Staff Research Engineer at Google Research NY where I develop methods for sequence modeling / NLP, machine learning, and AI. My research interests are in all things related to improving, expanding, or rethinking the functionality of Transformers and other state-of-the-art sequence models. At Google, I have made significant contributions to efficient token-free models, generative retrieval, safety, and Bard creativity.

Prior to research, I worked on a wide variety of projects across Google including natural language generation for news, fact checking, database visualization, and distributed systems. In a previous life I was a full-stack software developer, and prior to that I attended Brown University as an undergrad (‘16).

Selected Publications

Transcending scaling laws with 0.1% extra compute (2022)
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q. Tran, David R. So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc V. Le, Mostafa Dehghani

UL2: Unifying Language Learning Paradigms (2022)
Yi Tay*, Mostafa Dehghani*, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, Donald Metzler
ICLR 2023

Transformer Memory as a Differentiable Search Index (2022)
Yi Tay*, Vinh Q. Tran*, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, Donald Metzler
NeurIPS 2022

A New Generation of Perspective API: Efficient Multilingual Character-level Transformers (2022)
Alyssa Lees*, Vinh Q. Tran*, Yi Tay*, Jeffrey Sorensen, Jai Gupta, Donald Metzler, Lucy Vasserman
KDD 2022 ADS

ExT5: Towards extreme multi-task scaling for transfer learning (2021)
Vamsi Aribandi*, Yi Tay*, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler.
ICLR 2022

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021)
Yi Tay*, Vinh Q. Tran*, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler.
ICLR 2022