Creating and sharing knowledge for telecommunications

Interpretable Structure Induction via Sparse Attention

Peters, B. ; Niculae, V. ; Martins, A.

Interpretable Structure Induction via Sparse Attention, Proc BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, Vol. , pp. - , November, 2018.

Digital Object Identifier: 10.18653/v1/W18-5450

Download Full text PDF ( 239 KBs)


Neural network methods are experiencing wide adoption in NLP, thanks to their empirical performance on many tasks. Modern neural architectures go way beyond simple feedforward and recurrent models: they are complex pipelines that perform soft, differentiable computation instead of discrete logic. The price of such soft computing is the introduction of dense dependencies, which make it hard to disentangle the patterns that trigger a prediction. Our recent work on sparse and structured latent computation presents a promising avenue for enhancing interpretability of such neural pipelines. Through this extended abstract, we aim to discuss and explore the potential and impact of our methods.