Code

MultiHeadAttention implementation#2146

Code

Merged

CarloLucibello merged 21 commits intomasterFluxML/Flux.jl:masterfrom cl/attentionFluxML/Flux.jl:cl/attentionCopy head branch name to clipboardMar 11, 2023

+270-17

Commits

Commits on Mar 4, 2023

• move multiheadattention from Metalhead

  CarloLucibello

  committed
  a7c5952

  Copy full SHA for a7c5952

• generic attention

  CarloLucibello

  committed
  40c706d

  Copy full SHA for 40c706d

• [ci skip] updates

  CarloLucibello

  committed
  59edf23

  Copy full SHA for 59edf23

• [ci skip] updates

  CarloLucibello

  committed
  3646956

  Copy full SHA for 3646956

• [ci skip] fix tullio impl

  CarloLucibello

  committed
  2644283

  Copy full SHA for 2644283

• causal mask

  CarloLucibello

  committed
  0c61daa

  Copy full SHA for 0c61daa

• [ci skip] mask

  CarloLucibello

  committed
  6e7f538

  Copy full SHA for 6e7f538

• [ci skip] updates

  CarloLucibello

  committed
  e212b6b

  Copy full SHA for e212b6b

• [ci skip] add native implementation

  CarloLucibello

  committed
  2df4d5e

  Copy full SHA for 2df4d5e

• support mask = :causal

  CarloLucibello

  committed
  30b22d7

  Copy full SHA for 30b22d7

• [ci skip] factor out impl

  CarloLucibello

  committed
  38b8bdf

  Copy full SHA for 38b8bdf

• [ci skip] remove jax

  CarloLucibello

  committed
  19fe8d9

  Copy full SHA for 19fe8d9

• [ci skip] more benchs

  CarloLucibello

  committed
  2b9b219

  Copy full SHA for 2b9b219

Commits on Mar 5, 2023

• finish up

  CarloLucibello

  committed
  5745555

  Copy full SHA for 5745555

• cleanup

  CarloLucibello

  committed
  a1e8365

  Copy full SHA for a1e8365

• add cuda tests

  CarloLucibello

  committed
  2ecf19b

  Copy full SHA for 2ecf19b

Commits on Mar 10, 2023

• cleanup tests

  CarloLucibello

  committed
  b2809b2

  Copy full SHA for b2809b2

• IntOrDims

  CarloLucibello

  committed
  bd28c54

  Copy full SHA for bd28c54

• cleanup

  CarloLucibello

  committed
  111d814

  Copy full SHA for 111d814

• remove with_scores

  CarloLucibello

  committed
  0b10823

  Copy full SHA for 0b10823

• improve docstring

  CarloLucibello

  committed
  29afec7

  Copy full SHA for 29afec7