🌋 Specialized LLM nanoGPT-Seis Created for Seismology
The developer has introduced nanoGPT-Seis — a model with 113 million parameters, trained from scratch on seismological data and scientific corpora. The architecture includes GQA, RoPE, and SwiGLU to handle a context window of 4096 tokens.
🌍 The project proves the effectiveness of creating highly specialized small language models (SLMs) through careful selection of the data mix, ensuring high accuracy at low costs.
👤 This serves as an example of creating a high-quality scientific model using accessible hardware (2× NVIDIA A30).