【如果笔记对你有帮助，欢迎关注&点赞&收藏，收到正反馈会加快更新！谢谢支持！】

上期笔记

算法面试常见手撕题整理笔记（一）：K-Means, ​计算检测框IoU, NMS (非极大值抑制)_k-means计算题-CSDN博客

一、多头自注意力

• 方法：输入数据(N, L, embed_dim) 分成多个子空间（heads），在每个子空间中独立地计算注意力权重，然后将这些子空间的结果拼接起来，最后通过一个线性变换进行整合。
• 操作流程：
  
• 代码：

  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  
  class MultiHeadAttn(nn.Module):
      def __init__(self, embed_dim, num_heads):  # 
          super(MultiHeadAttn, self).__init__()
          self.embed_dim = embed_dim
          self.num_heads = num_heads
          self.head_dim = embed_dim // num_heads  # 每个头的dim
  
          self.w_q = nn.Linear(embed_dim, embed_dim)  # query投影层
          self.w_k = nn.Linear(embed_dim, embed_dim)  # key投影层
          self.w_v = nn.Linear(embed_dim, embed_dim)  # value投影层
          self.w_o = nn.Linear(embed_dim, embed_dim)  # 输出投影层
  
      def forward(self, x):
          N, L, dim = x.shape
          q = self.w_q(x).reshape(N, L, self.num_heads, self.head_dim).transpose(1, 2)
          k = self.w_k(x).reshape(N, L, self.num_heads, self.head_dim).transpose(1, 2)
          v = self.w_v(x).reshape(N, L, self.num_heads, self.head_dim).transpose(1, 2)
  
          score = torch.matmul(q, k.transpose(-2,-1)) / (self.head_dim**0.5)
          attn_weights = F.softmax(score, dim=-1)
          output = torch.matmul(attn_weights, v).transpose(1, 2).reshape(N, L, self.embed_dim)
  
          output =  self.w_o(output)
          return output
  
  
  if __name__ == '__main__':
      x = torch.randn(2, 10, 32*4)
      attn = MultiHeadAttn(32*4, 4)
      output = attn(x)
          
  

二、2D卷积

• 实现：
  
• 公式：
  
  
   
• 卷积操作 for 循环顺序：bs【批量大小】 → out_channels【输出通道数】 → h_out【输出高度】 → w_out【输出宽度】 → in_channels【输入通道数】 → kh【卷积核高度】 → kw【卷积核宽度】 
• 代码：

  import numpy as np
  
  def CNN(x, in_channels, out_channels, kernel, stride, padding):
      bs, c_in, h_in, w_in = x.shape
      c_out, c_in, kh, kw = kernel.shape
      h_out = (h_in + 2*padding - kh) // stride + 1
      w_out = (w_in + 2*padding - kw) // stride + 1
      
      x = np.pad(x, [(0, 0), (0, 0), (padding, padding), (padding, padding)]) # 表示在第一、二个维度（批量大小、通道维度）不填充，第三、四个维度（高度、宽度）前后各填充 padding 个单位
  
      out = np.zeros((bs, out_channels, h_out, w_out))
  
      for i in range(bs):
          for j in range(out_channels):
              for h in range(h_out):
                  for w in range(w_out):
                      for c in range(in_channels):
                          for k_h in range(kh):
                              for k_w in range(kw):
                                  out[i, j, h, w] += x[i, c, h * stride + k_h, w * stride + k_w] * kernel[j, c, k_h, k_w]
      
      return out
  
  
  if __name__ == '__main__':
      # 输入张量
      x = np.random.rand(1, 3, 5, 5)
  
      # 卷积核
      kernel = np.random.rand(2, 3, 3, 3)
  
      # 参数
      in_channels = 3
      out_channels = 2
      stride = 1
      padding = 1
  
      out = CNN(x, in_channels, out_channels, kernel, stride, padding)
  
      print("Output shape:", out.shape)  # Output shape: (1, 2, 5, 5)

三、Batch Norm 批归一化

• Batch Norm 批归一化：对批次的每个通道分别计算均值和方差
  
• 参数说明：
  • gamma:（可学习）缩放参数，用于对归一化后的数据进行缩放
  • beta:（可学习）偏移参数，用于对归一化后的数据进行偏移（添加一个常数偏移量）
  • momentum: 动量参数，用于更新移动平均值和移动方差，控制了新统计值对移动平均值和移动方差的贡献程度【使移动平均值和移动方差更平滑地更新，减少因单个批量数据的波动对全局统计值的影响】
  • moving_mean: 移动平均值。在训练阶段，根据当前批量的均值动态更新，用于推理阶段的均值估计。
  • moving_var: 移动方差。在训练阶段，根据当前批量的方差动态更新，用于推理阶段的方差估计。
• 代码：

  import torch
  import torch.nn as nn
  
  def batch_norm(x, gamma, beta, momentum, eps, moving_mean, moving_var, is_train=True):
      # gamma: 缩放参数; beta: 偏移参数; momentum: 动量参数; eps: 小常数，用于防止除以零的错误; moving_mean: 移动平均值，用于推理阶段的均值估计; moving_var: 移动方差，用于推理阶段的方差估计。
      x_mean = torch.mean(x, dim=(0, 2, 3), keepdim=True) # 当前批次均值
      x_var = torch.mean((x - x_mean) ** 2, dim=(0, 2, 3), keepdim=True)  # 当前批次房哈
  
      if is_train:  # 训练
          x_hat = (x - x_mean) / torch.sqrt(x_var + eps)
          moving_mean = momentum * moving_mean + (1 - momentum) * x_mean  # 更新移动均值
          moving_var = momentum * moving_var + (1 - momentum) * x_var   # 更新移动方差
  
      else:  # 推理
          x_hat = (x - moving_mean) / torch.sqrt(moving_var + eps)
      
      out = gamma * x_hat + beta
      return out, moving_mean, moving_var
  
  
  class BnLayer(nn.Module):
      def __init__(self, feat_nums):
          super(BnLayer, self).__init__()
          self.feat_nums = feat_nums
          self.eps = 1e-5
          self.momentum = 0.9
  
          shape = (1, feat_nums, 1, 1)
          self.gamma = nn.Parameter(torch.ones(shape), requires_grad=True)
          self.beta = nn.Parameter(torch.ones(shape), requires_grad=True)
          self.moving_mean = torch.zeros(shape)
          self.moving_var = torch.zeros(shape)
  
      def forward(self, x):
          y, self.moving_mean, self.moving_var = batch_norm(x, self.gamma, self.beta, self.momentum, self.eps, self.moving_mean, self.moving_var, is_train=True)
          return y
  
      
  if __name__ == '__main__':
      tensor = torch.ones(2, 3, 50, 50)  # bs, num_channels, h, w
      bn = BnLayer(3)
      output = bn(tensor)