AlterNet: Adaptive-Depth Mixture-of-Experts with Learned Halting for Elastic Inference
Most LLM architectures have fixed compute demands determined at training time.
arXiv propose a novel architecture that combines the Mixture-of-Experts architecture with shallow experts and a novel dynamic-depth sys.