In this repository, we survey three crucial areas: (1) representation learning, (2) neural codec, and (3) language models that contribute to speech/audio large language models.

1.⚡ Speech Representation Models: These models focus on learning structural speech representations, which can then be quantized into discrete speech tokens, often refer to semantic tokens.

2.⚡ Speech Neural Codec Models: These models are designed to learn speech and audio discrete tokens, often referred to as acoustic tokens, while maintaining reconstruction ability and low bitrate.

3.⚡ Speech Large Language Models: These models are trained on top of speech and acoustic tokens in a language modeling approach. They demonstrate proficiency in tasks on speech understanding and speech generation.

Kai-Wei Chang	Haibin Wu	Wei-Cheng Tseng
Kehan Lu	Chun-Yi Kuan	Hung-yi Lee

• The challenge covers nowday's neural audio codecs and speech / audio language models.
  • Time: December 3 starting at 15:15
  • Detailed agenda: https://codecsuperb.github.io/
• Keynote speakers
  • Neil Zeghidour (Moshi): 15:15-16:00
    • slides | recording
    • Title: Audio Language Models
  • Dongchao Yang (CUHK): 16:00-16:35
    • slides | recording
    • Title: Challenges in Developing Universal Audio Foundation Model
  • Shang-Wen Li (Meta): 16:35-17:10
    • slides | recording
    • Title: VoiceCraft: Zero-Shot Speech Editing and TTS in the Wild
  • Wenwu Wang (University of Surrey): 17:40-18:15
    • slides | recording
    • Title: Neural Audio Codecs: Recent Progress and a Case Study with SemantiCodec
  • Minje Kim (UIUC): 18:15-18:50
    • slides | recording
    • Title: Future Directions in Neural Speech Communication Codecs
• Host
  • Hung-yi Lee (NTU)
  • Haibin Wu (Microsoft)
• Accepted papers (recording)
  • ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
  • Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
  • Investigating neural audio codecs for speech language model-based speech generation
  • Addressing Index Collapse of Large-Codebook Speech Tokenizer with Dual-Decoding Product-Quantized Variational Auto-Encoder
  • MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios

Webpage. The challenge will cover nowday's neural audio codecs and speech/audio language models. Agenda: To be determined.

Professor Hung-Yi Lee will be giving a talk as part of the Interspeech 2024 survey talk titled Challenges in Developing Spoken Language Models. The topic will cover nowday's speech/audio large language models.

I (Kai-Wei Chang) will be giving a talk as part of the ICASSP 2024 tutorial titled Parameter-Efficient and Prompt Learning for Speech and Language Foundation Models. The topic will cover nowday's speech/audio large language models. The slides from my presentation is available at https://kwchang.org/talks/. Please feel free to reach out to me for any discussions.

• https://github.com/liusongxiang/Large-Audio-Models
• https://github.com/kuan2jiu99/Awesome-Speech-Generation
• https://github.com/ga642381/Speech-Prompts-Adapters
• https://github.com/voidful/Codec-SUPERB
• https://github.com/huckiyang/awesome-neural-reprogramming-prompting

If you find this repository useful, please consider citing the following papers.

@article{wu2024codec,
  title={Codec-SUPERB@ SLT 2024: A lightweight benchmark for neural audio codec models},
  author={Wu, Haibin and Chen, Xuanjun and Lin, Yi-Cheng and Chang, Kaiwei and Du, Jiawei and Lu, Ke-Han and Liu, Alexander H and Chung, Ho-Lam and Wu, Yuan-Kuei and Yang, Dongchao and others},
  journal={arXiv preprint arXiv:2409.14085},
  year={2024}
}

@inproceedings{wu-etal-2024-codec,
    title = "Codec-{SUPERB}: An In-Depth Analysis of Sound Codec Models",
    author = "Wu, Haibin  and
      Chung, Ho-Lam  and
      Lin, Yi-Cheng  and
      Wu, Yuan-Kuei  and
      Chen, Xuanjun  and
      Pai, Yu-Chi  and
      Wang, Hsiu-Hsuan  and
      Chang, Kai-Wei  and
      Liu, Alexander  and
      Lee, Hung-yi",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-acl.616",
    doi = "10.18653/v1/2024.findings-acl.616",
    pages = "10330--10348",
}

@article{wu2023speechgen,
  title={Speechgen: Unlocking the generative power of speech language models with prompts},
  author={Wu, Haibin and Chang, Kai-Wei and Wu, Yuan-Kuei and Lee, Hung-yi},
  journal={arXiv preprint arXiv:2306.02207},
  year={2023}
}

@article{wu2024towards,
  title={Towards audio language modeling-an overview},
  author={Wu, Haibin and Chen, Xuanjun and Lin, Yi-Cheng and Chang, Kai-wei and Chung, Ho-Lam and Liu, Alexander H and Lee, Hung-yi},
  journal={arXiv preprint arXiv:2402.13236},
  year={2024}
}