Although hybrid approaches have been proposed to combine the advantages of convolutions and self-attention for a better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. However, its quadratic computational complexity with respect to image resolution limits its use in real-time applications, especially for deployment on resource-constrained mobile devices. Self-attention has become a defacto choice for capturing global context in various vision applications. Then, the matrix K is element-wise multiplied by the broadcasted global queries, resulting the global context representation. Here, the query matrix is multiplied by learnable weights and pooled to produce global queries. (d) Our proposed efficient additive self-attention. Then, the context vector is multiplied by V matrix to produce the final output. (c) is the separable self-attention of MobileViT-v2, it uses element-wise operations to compute the context vector from the interactions of Q and K matrices. (b) is the transpose self-attention, where the self-attention operation is applied across channel feature dimensions (d×d) instead of the spatial dimension (n×n). The latency is measured on iPhone 14 Neural Engine (iOS 16).Ĭomparison with different self-attention modules. (Mar 27, 2023): Classification training and evaluation codes along with pre-trained models are released.Ĭomparison of our SwiftFormer Models with state-of-the-art on ImgeNet-1K.SwiftFormer SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications
0 kommentar(er)
