ReGLA: Efficient Receptive-Field Modeling with Gated Linear Attention NetworkAuthors:Junzhou Li, Manqi Zhao, Yilin Gao, Zhiheng Yu, Yin Li, Dongsheng Jiang, Li XiaoDeep-Dive Summary:Attention Is All You Need (Transformer) 论文总结1. 摘要与背景本文提出了Transformer模型这是一种全新的神经网络架构完全摒弃了传统的循环神经网络RNN和卷积神经网络CNN仅依靠**注意力机制Attention Mechanism**来建立输入与输出之间的全局依赖关系。传统序列模型如 LSTM 和 GRU的主要局限在于其计算的并行性较差且难以捕捉长距离的依赖关系。Transformer 通过自注意力机制解决了这些问题并在 WMT 2014 英德和英法翻译任务中取得了当时的 SOTAState-of-the-Art性能。2. 模型架构Transformer 采用了经典的Encoder-Decoder编码器-解码器结构。Encoder (编码器)由 6 个相同的层堆叠而成。每一层包含两个子层多头自注意力机制Multi-Head Self-Attention和全连接前馈网络Feed Forward Network。每个子层周围使用了残差连接Residual Connection和层归一化Layer Normalization。Decoder (解码器)同样由 6 个相同的层堆叠而成。除了编码器中的两个子层外解码器还插入了第三个子层用于对编码器的输出执行多头注意力。此外解码器采用了掩码Masking机制以确保预测当前位置时不会看到未来的信息。3. 注意力机制 (Attention Mechanism)论文详细介绍了两种注意力组件。3.1 缩放点积注意力 (Scaled Dot-Product Attention)其计算公式为A t t e n t i o n ( Q , K , V ) softmax ( Q K T d k ) V Attention(Q, K, V) \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)VAttention(Q,K,V)softmax(dk​​QKT​)V其中Q QQ为查询K KK为键V VV为值。引入1 d k \frac{1}{\sqrt{d_k}}dk​​1​缩放因子的目的是防止点积结果过大导致梯度消失。3.2 多头注意力 (Multi-Head Attention)相比于单一的注意力多头注意力允许模型在不同的表示子空间中共同学习不同位置的信息。公式表示为M u l t i H e a d ( Q , K , V ) Concat ( h e a d 1 , . . . , h e a d h ) W O MultiHead(Q, K, V) \text{Concat}(head_1, ..., head_h)W^OMultiHead(Q,K,V)Concat(head1​,...,headh​)WO其中每个h e a d i A t t e n t i o n ( Q W i Q , K W i K , V W i V ) head_i Attention(QW_i^Q, KW_i^K, VW_i^V)headi​Attention(QWiQ​,KWiK​,VWiV​)。4. 位置编码 (Positional Encoding)由于 Transformer 不包含循环或卷积为了让模型利用序列的顺序信息必须加入位置编码。作者使用了不同频率的正弦和余弦函数P E ( p o s , 2 i ) sin ⁡ ( p o s / 10000 2 i / d m o d e l ) PE_{(pos, 2i)} \sin(pos / 10000^{2i/d_{model}})PE(pos,2i)​sin(pos/100002i/dmodel​)P E ( p o s , 2 i 1 ) cos ⁡ ( p o s / 10000 2 i / d m o d e l ) PE_{(pos, 2i1)} \cos(pos / 10000^{2i/d_{model}})PE(pos,2i1)​cos(pos/100002i/dmodel​)5. 为什么选择自注意力论文对比了自注意力层、循环层和卷积层的计算复杂度。自注意力在序列长度n nn小于表示维度d dd时这是常见情况具有更低的每层计算量且最大路径长度为O ( 1 ) O(1)O(1)极大地提升了并行计算能力。6. 训练与实验结果模型使用了 Adam 优化器并配合学习率预热Warmup策略。实验结果显示Transformer 在英德和英法翻译任务上的 BLEU 分数显著优于之前的模型且训练成本大幅降低。7. 结论Transformer 是第一个完全基于注意力的序列转录模型。它不仅在翻译质量上取得了突破而且在训练效率和扩展性方面展现了巨大潜力为后来诸如 BERT、GPT 等模型的发展奠定了基础。Original Abstract:Balancing accuracy and latency on high-resolution images is a critical challenge for lightweight models, particularly for Transformer-based architectures that often suffer from excessive latency. To address this issue, we introduce \textbf{ReGLA}, a series of lightweight hybrid networks, which integrates efficient convolutions for local feature extraction with ReLU-based gated linear attention for global modeling. The design incorporates three key innovations: the Efficient Large Receptive Field (ELRF) module for enhancing convolutional efficiency while preserving a large receptive field; the ReLU Gated Modulated Attention (RGMA) module for maintaining linear complexity while enhancing local feature representation; and a multi-teacher distillation strategy to boost performance on downstream tasks. Extensive experiments validate the superiority of ReGLA; particularly the ReGLA-M achieves \textbf{80.85%} Top-1 accuracy on ImageNet-1K at224 p x 224px224px, with only \textbf{4.98 ms} latency at512 p x 512px512px. Furthermore, ReGLA outperforms similarly scaled iFormer models in downstream tasks, achieving gains of \textbf{3.1%} AP on COCO object detection and \textbf{3.6%} mIoU on ADE20K semantic segmentation, establishing it as a state-of-the-art solution for high-resolution visual applications.PDF Link:2602.05262v1部分平台可能图片显示异常请以我的博客内容为准