Large Memory Layers with Product Keys
Augmenting transformer language models with sparse access of large memory matrices
Attention
Finetuning Transformers with JAX + Haiku
A practical, code-first look at DeepMind's new haiku library.
A Survey of Long-Term Context in Transformers
Exploring 6 noteworthy approaches for incorporating longer-term context in transformer models.
machine learning musings