考察点:把注意力公式落地的工程能力,以及变长序列下 padding 掩码(含因果掩码)对 softmax 的影响。
要点:
import numpy as np
def scaled_dot_product_attention(Q, K, V, mask=None):
d_k = Q.shape[-1]
scores = Q @ K.transpose(-1, -2) / np.sqrt(d_k) # (..., Lq, Lk)
if mask is not None: # mask 中需要屏蔽处为 0
scores = np.where(mask[..., None, :] == 0, -1e9, scores)
scores -= scores.max(axis=-1, keepdims=True) # 数值稳定
w = np.exp(scores); w /= w.sum(axis=-1, keepdims=True)
return w @ V
L = 4
causal = np.tril(np.ones((L, L))) # 下三角可见(因果掩码)
print(scaled_dot_product_attention(np.random.randn(1, L, 8),
np.random.randn(1, L, 8), np.random.randn(1, L, 8), mask=causal).shape)
题目与答案就在本页;收藏、批注和错题重练在应用内