Code

Handle char level tokenization#11

Code

Merged

Zymrael merged 4 commits intotogethercomputer:maintogethercomputer/stripedhyena:mainfrom brianhie:char_tokenizer1brianhie/stripedhyena:char_tokenizer1Copy head branch name to clipboardFeb 18, 2024

+138-11

Commits

Commits on Feb 18, 2024

• add char level tokenizer, modify generation code to handle batched inference with char tok

  brianhie

  committed
  4a1d35f

  Copy full SHA for 4a1d35f

• remove some comments

  brianhie

  committed
  f6a49aa

  Copy full SHA for f6a49aa

• respond to review, delete record_memory, cleanup args

  brianhie

  committed
  171fb71

  Copy full SHA for 171fb71

• black and isort

  brianhie

  committed
  047503a

  Copy full SHA for 047503a