TinyLlama-1.1B: Training a Compact Language Model from Scratch
©PaperWeekly Original · Author | StatNLP Institution | Singapore University of Technology and Design The TinyLlama project aims to pre-train on 3 trillion tokens, building a Llama model with 1.1 billion parameters. With careful optimization, we only need 16 A100-40G GPUs to complete this task in 90 days🚀🚀. The model training started on 2023-09-01 and … Read more