kyegomez repositories

My own implementation/experiments with a local softmax

Last commit Sep 29, 2023

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

Last commit Jan 7, 2024

A simplistic pytorch implementation of LongVit using my previous implementation of LongNet as a foundation.

Last commit Dec 21, 2023

Implementation of M2PT in PyTorch from the paper: "Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities"

Last commit Jan 29, 2024

Open source community's implementation of the model from "LANGUAGE MODEL BEATS DIFFUSION — TOKENIZER IS KEY TO VISUAL GENERATION"

Last commit Jan 3, 2024

Implementation of the model: "(MC-ViT)" from the paper: "Memory Consolidation Enables Long-Context Video Understanding"

Last commit Feb 11, 2024

An open source community implementation of the model MELLE from the paper: "Autoregressive Speech Synthesis without Vector Quantization"

Last commit Oct 8, 2024

The open source implementation of the multi grouped query attention by the paper "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"

Last commit Sep 28, 2023

Community Implementation of the paper: "Multi-Head Mixture-of-Experts" In PyTorch

Last commit Apr 27, 2024

"Multimodal Instruction Tuning", An SOTA video dataset

Last commit Jul 2, 2023

Simple Implementation of a Transformer in the new framework MLX by Apple

Last commit Feb 5, 2024

PyTorch Implementation of the paper "MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training"

Last commit Apr 26, 2024

The open source community's implementation of the all-new Multi-Modal Causal Attention from "DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention"

Last commit Nov 27, 2023

Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention

Last commit Sep 28, 2023

Implementation of "MORPHEUS-1" from Prophetic AI and "The world’s first multi-modal generative ultrasonic transformer designed to induce and stabilize lucid dreams. "

Last commit Feb 3, 2024

MOSS-RLHF

Last commit Jul 12, 2023

MPBS - Model PreTraining Benchmarking Suite to test various metrics for pytorch models before pretraining, training, finetuning, or inference.

Last commit Jul 13, 2023

Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta

Last commit Feb 16, 2024

MambaDecoderBlock is a novel decoder architecture that replaces traditional self-attention mechanisms with Mamba state space models, augmented by Mixture of Experts (MoE) layers.

Last commit Dec 2, 2025

Implementation of MambaFormer in Pytorch ++ Zeta from the paper: "Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks"

Last commit Apr 7, 2024