Tag: MatMul-free language model

All the talks with the tag "MatMul-free language model".

• Scalable MatMul-free Language Modeling

  Sagar Prakash Barad
  Published:Jul 15, 2024 at  at 02:00 PM

  This talk presents a paper that proposes a scalable MatMul-free language model, challenging the assumption that matrix multiplications are essential for high-performing language models. The paper demonstrates that by using ternary weights and element-wise Hadamard products, MatMul operations can be completely removed from large language models while maintaining strong performance. The paper provides an optimized implementation of the MatMul-free language model, achieving significant reductions in memory usage and latency compared to conventional models.