1. glu 计算

原理

demo

dropout

原理
训练过程中，输入张量的一些元素按从伯努利分布中采样的概率p随机置零。在每个前向调用过程中每个通道都能被独立置零。
Dropout方法证明被证明是正则化和防止神经元的互适应（co-adaptation）效应的有效技术。

形状：
输入: 输入可以是任意形状
输出: 输出和输入同形状

demo

pytorch 计算BN的坑

import torch
import torch.nn as nn
bn = nn.BatchNorm2d(3)
x = torch.randn(4, 3, 2, 2)
y = bn(x)
import pdb
pdb.set_trace()
print(bn.weight, bn.bias)
print(y)
print(x)
a = (x[0, 0, :, :] + x[1, 0, :, :] + x[2, 0, :, :] + x[3, 0, :, :]).sum() / 16
b = (x[0, 1, :, :] + x[1, 1, :, :] + x[2, 1, :, :] + x[3, 1, :, :]).sum() / 16
c = (x[0, 2, :, :] + x[1, 2, :, :] + x[2, 2, :, :] + x[3, 2, :, :]).sum() / 16
print('The mean value of the first channel is %f' % a.data)
print('The mean value of the first channel is %f' % b.data)
print('The mean value of the first channel is %f' % c.data)
print('The output mean value of the BN layer is %f, %f, %f' % (bn.running_mean.data[0], bn.running_mean.data[1], bn.running_mean.data[2]))

print(x[:,0,:,:].mean())
print(x[:,0,:,:].var(unbiased=False))  # 注意这里用无偏估计
print(x[:,0,:,:].var())

src=(x[0,0,:,:] - a.data) *1./torch.sqrt( x[:,0,:,:].var(unbiased=False)) # 同时注意是开方
print(src* bn.weight[0] + bn.bias[0])

BN是在每个channel的通道上学些两个参数，一个是斜率，两外一个偏差。
例如一个 64* 16* 32* 256 产生 16*2个数值

Parameter containing:
tensor([1., 1., 1.], requires_grad=True) Parameter containing:
tensor([0., 0., 0.], requires_grad=True)
tensor([[[[-0.1003,  0.1534],
          [-1.2012,  1.0044]],

         [[-2.0168,  0.4494],
          [ 0.8210, -0.2925]],

         [[-0.1713,  1.2344],
          [-0.6626, -0.2881]]],


        [[[-0.8748,  2.3160],
          [ 0.8673,  1.0198]],

         [[ 1.6093, -0.0789],
          [ 0.3001, -0.1935]],

         [[ 0.8850,  2.4405],
          [-1.6390, -0.6598]]],


        [[[-0.2210,  0.4729],
          [-1.6394, -1.2612]],

         [[-0.6592, -1.4095],
          [ 0.1998,  1.0182]],

         [[-0.1326,  0.2007],
          [-0.7224, -1.1040]]],


        [[[-0.3208, -0.0263],
          [-0.7730,  0.5842]],

         [[ 1.0196, -1.1605],
          [ 1.2658, -0.8723]],

         [[-0.7327,  0.7711],
          [-0.4042,  0.9850]]]], grad_fn=<NativeBatchNormBackward>)
tensor([[[[-0.0146,  0.1799],
          [-0.8585,  0.8322]],

         [[-1.8782,  0.8985],
          [ 1.3168,  0.0632]],

         [[-0.2204,  1.2631],
          [-0.7389, -0.3437]]],


        [[[-0.6083,  1.8375],
          [ 0.7271,  0.8440]],

         [[ 2.2044,  0.3037],
          [ 0.7304,  0.1746]],

         [[ 0.8944,  2.5360],
          [-1.7693, -0.7360]]],


        [[[-0.1071,  0.4248],
          [-1.1944, -0.9044]],

         [[-0.3496, -1.1945],
          [ 0.6174,  1.5389]],

         [[-0.1796,  0.1722],
          [-0.8020, -1.2048]]],


        [[[-0.1836,  0.0421],
          [-0.5302,  0.5101]],

         [[ 1.5405, -0.9141],
          [ 1.8177, -0.5896]],

         [[-0.8129,  0.7742],
          [-0.4662,  0.9999]]]])
The mean value of the first channel is 0.062279
The mean value of the first channel is 0.392515
The mean value of the first channel is -0.039619
The output mean value of the BN layer is 0.006228, 0.039252, -0.003962
tensor(0.0623)
tensor(0.5875)
tensor(0.6267)
tensor([[-0.1003,  0.1534],
        [-1.2012,  1.0045]], grad_fn=<AddBackward0>)