Seminar by Ms. Bingbin LIU from Machine Learning Department, Carnegie Mellon University

Machine learning systems are becoming increasingly powerful and complex, which poses challenges for performing diagnostics and making targeted improvements. In this talk, we will see how simple testbeds amenable to theoretical analyses can yield insights to guide practical advancements, with a specific focus on algorithmic reasoning. We will discuss how Transformer, designed for parallel computation, can perform algorithmic reasoning tasks that are sequential in nature. By representing reasoning with finite-state automata, we first provide formal results that Transformers can simulate reasoning using far fewer layers than the number of sequential reasoning steps. We will then discuss the gap between these theoretical constructions and practical considerations such as generalization and interoperability.

Bingbin Liu is a PhD student in the Machine Learning Department at Carnegie Mellon University, advised by Andrej Risteski and Pradeep Ravikumar. Her research interests include the theoretical and scientific understanding of machine learning, with a focus on self-supervised learning and more recently language models. Prior to CMU, she earned an MSc degree from Stanford University and is a proud graduate of the Department of Computer Science at HKU.