I will start by giving a brief overview of my DeepSPIN ERC project (https://deep-spin.github.io), whose goal is to develop new deep learning methods, models, and algorithms for structured prediction in natural language processing (NLP). Then, I will cover in more detail some recent work done in my group on sparse attention mechanisms. Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this talk, I will introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with alpha-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the alpha parameter—which controls the shape and sparsity of alpha-entmax—allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations. Joint work with Ben Peters, Gonçalo Correia, Vlad Niculae, Chaitanya Malaviya, Pedro Ferreira, Julia Kreutzer, Mathieu Blondel, Claire Cardie, Ramon Astudillo.

face  Speaker: André Martins  

Biografia: https://www.it.pt/Members/Index/4017