The AIIT-THRESHOLD research group has introduced Tessera 1B, a language model with approximately 1 billion parameters, trained from scratch. The project demonstrates the possibility of creating a high-quality base model on an extremely low budget, spending only $315 on computing resources.
What Happened
The Tessera 1B model was trained on the custom ProtoGPT architecture (a decoder-only transformer) over 146 hours on a single NVIDIA H100 (80GB) GPU. Training was conducted on a corpus of 24.5 billion tokens consisting of carefully selected web pages, books, and academic texts, without the use of synthetic dialogues. The developers have made the model weights and data publicly available.
Context
Unlike many modern approaches that rely on massive amounts of synthetic content, this project bets on data curation. Using the ProtoGPT architecture and focusing on cleaned corpora allows for achieving efficiency at significantly lower training costs.
Why It Matters for the Industry
The project confirms the viability of a low-cost/high-quality pipeline for creating small language models (SLMs). It proves that high-quality base models can be created with extremely low CAPEX, lowering the barrier to entry for independent researchers and stimulating the development of specialized vertical AI solutions.
Why It Matters for Users
For enthusiasts and developers, this serves as an example of how to build serious AI tools with a minimal budget. While Tessera 1B is not a ready-to-use chatbot, it serves as an excellent foundation for creating specialized AI assistants for specific tasks through subsequent fine-tuning.
Sources
Author
Look at AI, Editorial Staff