Lighthouse Attention

Open-source implementation and training-time hierarchical attention method for efficiently pretraining long-context causal language models; provided as a patch and source files for PyTorch Torchtitan.