Pre-training was conducted in three phases, covering long-horizon pre-training, mid-training, and a long-context extension phase. We used sigmoid-based routing scores rather than traditional softmax gating, which improves expert load balancing and reduces routing collapse during training. An expert-bias term stabilizes routing dynamics and encourages more uniform expert utilization across training steps. We observed that the 105B model achieved benchmark superiority over the 30B remarkably early in training, suggesting efficient scaling behavior.
Последние новости
。业内人士推荐新收录的资料作为进阶阅读
欢迎分享、点赞与留言。本作品的版权为南方周末或相关著作权人所有，任何第三方未经授权，不得转载，否则即为侵权。。新收录的资料对此有专业解读
A reliable technology for returning payloads to Earth from space is a necessity for several futuristic business models—testing new materials in orbit, manufacturing pharmaceuticals or high-end electronics in microgravity, or harvesting resources like metals from asteroids. The U.S. military has shown interest in the ability to provide logistics support with orbital deliveries or test components for hypersonic weapons.