Architecture

Multi-Head Attention

Multi-head attention applies the attention mechanism multiple times (e.g., 8 or 16 heads) in parallel. Each head learns to focus on different aspects or features. For instance, one head might capture semantic relationships while another captures syntactic structure. The outputs of all heads are concatenated and projected. Multi-head attention is more expressive than single-head attention. The number of heads is a design choice balancing computational cost and model expressiveness. Empirically, more heads generally help but have diminishing returns.