PaddleSpeech Architecture

Design Philosophy

For rapidly iteration of speech model and simplify model development and maintain, overly unified architecture is not appropriate, so we organize code as model first.

For common module, such as audio and text feature transformation and augmentation, we implement as library. We want implement model training/validation process as compose of these fundamental modules.

For each speech task, we have an exp and models directory. Under it, we create an directory named by model name. So we can do specific process for model, e.g. more fine data and/or feature processing, complicate training and/or evaluation loop.

We implement some recipe for conventional speech task, which on common dataset with performance reported, user can apply these template to private data or implement new model follow your own opinion. The philosophy make it easy to use, development and extension，which will accelerating industrial and/or academic iteration.

Layout of Source File

Full directory structure as follows:

paddlespeech
├── cli
│   ├── asr
│   ├── cls
│   ├── st
│   ├── text
│   └── tts
├── cls
│   ├── exps
│   │   └── panns
│   │       └── deploy
│   └── models
│       └── panns
├── s2t
│   ├── decoders
│   │   ├── beam_search
│   │   ├── ctcdecoder
│   │   │   └── tests
│   │   └── scorers
│   ├── exps
│   │   ├── deepspeech2
│   │   │   └── bin
│   │   │       └── deploy
│   │   ├── lm
│   │   │   └── transformer
│   │   │       └── bin
│   │   ├── u2
│   │   │   └── bin
│   │   ├── u2_kaldi
│   │   │   └── bin
│   │   └── u2_st
│   │       └── bin
│   ├── frontend
│   │   ├── augmentor
│   │   └── featurizer
│   ├── io
│   ├── models
│   │   ├── ds2
│   │   ├── ds2_online
│   │   ├── lm
│   │   ├── u2
│   │   └── u2_st
│   ├── modules
│   ├── training
│   │   ├── extensions
│   │   ├── triggers
│   │   └── updaters
│   ├── transform
│   └── utils
├── t2s
│   ├── audio
│   ├── data
│   ├── datasets
│   ├── exps
│   │   ├── fastspeech2
│   │   ├── gan_vocoder
│   │   │   ├── hifigan
│   │   │   ├── multi_band_melgan
│   │   │   ├── parallelwave_gan
│   │   │   └── style_melgan
│   │   ├── new_tacotron2
│   │   ├── speedyspeech
│   │   ├── tacotron2
│   │   ├── transformer_tts
│   │   ├── voice_cloning
│   │   │   └── tacotron2_ge2e
│   │   └── waveflow
│   ├── frontend
│   │   ├── normalizer
│   │   └── zh_normalization
│   ├── models
│   │   ├── fastspeech2
│   │   ├── hifigan
│   │   ├── melgan
│   │   ├── new_tacotron2
│   │   ├── parallel_wavegan
│   │   ├── speedyspeech
│   │   └── transformer_tts
│   ├── modules
│   │   ├── conformer
│   │   ├── predictor
│   │   ├── tacotron2
│   │   └── transformer
│   ├── training
│   │   ├── extensions
│   │   ├── triggers
│   │   └── updaters
│   └── utils
├── text
│   ├── exps
│   │   └── ernie_linear
│   └── models
│       ├── ernie_crf
│       └── ernie_linear
└── vector
    ├── exps
    │   └── ge2e
    └── models

97 directories

Speech Task and Toolbox

speech task as follows:

paddlespeech
├── cls (audio classfication/detection, emotion/gender/age recognition and so on)
├── s2t (speech to text task, e.g ASR, ST)
├── t2s (text to speech, e.g TTS, Voice Cloning, Voice Conversion, Music, Sing Voice Synthesis)
├── text (speech related text task, e.g. punctuation restoration, text corrector, model based text front-end)
└── vector (speech task which need extracting vector feature, e.g Speaker Verification/Identification, Language Identification, Speaker Dirazation)

speech toolbox as follows:

paddlespeech
├── cli (CLI toolbox for multi speech task)
└── server (Server/Client for speech task)

Exps and Models

exps contains bins for model, models contains implementation and specific process. Each speech task has these directories.

├── t2s
│   ├── exps
│   │   ├── fastspeech2
│   │   ├── gan_vocoder
│   │   │   ├── hifigan
│   │   │   ├── multi_band_melgan
│   │   │   ├── parallelwave_gan
│   │   │   └── style_melgan
│   │   ├── new_tacotron2
│   │   ├── speedyspeech
│   │   ├── tacotron2
│   │   ├── transformer_tts
│   │   ├── voice_cloning
│   │   │   └── tacotron2_ge2e
│   │   └── waveflow
│   ├── models
│   │   ├── fastspeech2
│   │   ├── hifigan
│   │   ├── melgan
│   │   ├── new_tacotron2
│   │   ├── parallel_wavegan
│   │   ├── speedyspeech
│   │   └── transformer_tts

All model implementation and updater core process are put in models, like bellow:

paddlespeech/t2s/models/
├── __init__.py
├── fastspeech2
│   ├── __init__.py
│   ├── fastspeech2.py
│   └── fastspeech2_updater.py
├── hifigan
│   ├── __init__.py
│   ├── hifigan.py
│   └── hifigan_updater.py
├── melgan
│   ├── __init__.py
│   ├── melgan.py
│   ├── multi_band_melgan_updater.py
│   ├── style_melgan.py
│   └── style_melgan_updater.py
├── new_tacotron2
│   ├── __init__.py
│   ├── tacotron2.py
│   └── tacotron2_updater.py
├── parallel_wavegan
│   ├── __init__.py
│   ├── parallel_wavegan.py
│   └── parallel_wavegan_updater.py
├── speedyspeech
│   ├── __init__.py
│   ├── speedyspeech.py
│   └── speedyspeech_updater.py
├── tacotron2.py
├── transformer_tts
│   ├── __init__.py
│   ├── transformer_tts.py
│   └── transformer_tts_updater.py
└── waveflow.py

7 directories, 26 files

fastspeech2.py is the model implementation, fastspeech2_updater.py is for train and valid core process.

We put train/valid/test and other process in exps directory, like bellow:

paddlespeech/t2s/exps/
├── __init__.py
├── fastspeech2
│   ├── __init__.py
│   ├── gen_gta_mel.py
│   ├── normalize.py
│   ├── preprocess.py
│   ├── train.py
│   └── voice_cloning.py

paddlespeech/s2t/exps/
├── __init__.py
├── deepspeech2
│   ├── __init__.py
│   ├── bin
│   │   ├── __init__.py
│   │   ├── deploy
│   │   │   ├── __init__.py
│   │   │   ├── client.py
│   │   │   ├── record.py
│   │   │   ├── runtime.py
│   │   │   ├── send.py
│   │   │   └── server.py
│   │   ├── export.py
│   │   ├── test.py
│   │   ├── test_export.py
│   │   ├── test_wav.py
│   │   └── train.py
│   └── model.py
├── lm
│   └── transformer
│       ├── __init__.py
│       ├── bin
│       │   ├── __init__.py
│       │   └── cacu_perplexity.py
│       └── lm_cacu_perplexity.py
├── u2
│   ├── __init__.py
│   ├── bin
│   │   ├── __init__.py
│   │   ├── alignment.py
│   │   ├── export.py
│   │   ├── test.py
│   │   ├── test_wav.py
│   │   └── train.py
│   ├── model.py
│   └── trainer.py