MAMI: Multi-Attentional Mutual-Information for Long Sequence Neuron Captioning

Neuron labelling is an approach to visualize the behaviour and response of a certain neuron to a certain pattern that activates the neuron. Previous work, namely MILAN (Mutual Information-guided Linguistic Annotation of Neuron), has tried to visualize the neuron behaviour using modified Show, Attend, and Tell (SAT) model in the encoder, and LSTM added with Bahdanau attention in the decoder. MILAN can show great results on short-sequence neuron labelling, however, it does not show great results on long-sequence neuron labelling. In this research, we would like to improve the performance of MILAN especially for long sequence captioning by utilizing different kinds of attention mechanisms and additionally adding several attention results into one, to combine all the advantages of several attention mechanisms. The main contribution to this research is we propose a new method for attention mechanism by combining the three existing attention: Bahdanau, Luong, and Self Attention for neuron labelling. Using a compound dataset, we obtained higher BLEU and F1-Score on our proposed model, achieving 17.742 and 0.4811 respectively. At some point where the model converges at the peak, our model obtained a BLEU of 21.2262 and BERTScore F1-Score of 0.4870.