Fish Audio S2 Technical Report: plain AI paper summary
AI-generated summary. Read the original paper before relying on it.
Hugging Face paper page · Original paper on arXiv
Fish Audio S2 is an open text-to-speech system with multiple speakers, multi-turn output, and natural-language control over voice behavior. Its training pipeline adds captioning, voice-quality checks, and reward modeling.
Key findings
- Multi-stage training prepares the model for controlled speech generation.
- A staged data pipeline supplies captions and voice-quality filtering.
- Streaming SGLang inference reaches a reported 0.195 real-time factor and below 100ms time to first audio.
- The authors release weights and fine-tuning code.
How you might use it
- Test a short, consented client-document narration and record pronunciation fixes, first-audio delay, and streaming stability.
- Try speaker changes and multi-turn prompts separately so voice-control errors are easy to identify.
Limits
The abstract does not report language coverage, voice-consent safeguards, or quality across accents and difficult names.
Related research
- AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing This paper on arXiv Related paper on arXiv: Both papers build controllable speech models. Fish Audio S2 handles multi-speaker, multi-turn synthesis from natural-language instructions, while AuK targets open speech generation and editing across broader audio tasks.
Business ideas
1. Multi-Speaker Dialogue Production
Offer: A service that delivers multi-speaker dialogue production for the target workflow.
Buyer: one audio production team
Technical link: Test instruction-following multi-speaker streaming text-to-speech as the mechanism behind multi-speaker dialogue production; the paper supports a hypothesis, not a market result.
Evidence
- competitor: ElevenLabs supports content creation customer support gaming education and accessibility use cases (year: 2025) (source)
- competitor: AssemblyAI provides pre-recorded and real-time speech-to-text and voice agent APIs (denominator: AssemblyAI homepage statement) (source)
- adoption: 7.2% of EU enterprises used AI to convert spoken language into machine-readable format (year: 2025 region: EU denominator: enterprises with 10 or more employees or self-employed persons in covered NACE sections) (source)
Differentiation: ElevenLabs supports voice generation and editing; this adds full-duplex or multi-speaker controls.
First test: Replay 40 licensed sessions or scripts and compare the proposed method with the current workflow.
Success measure: At least 90% of target events or speaker turns are correctly handled.
Constraints: Treat market sources as adjacent-tool or adoption proxies, use licensed or consented audio, and keep the pilot inside the named safety boundary.
2. Style-Controlled Voice Rendering
Offer: A service that delivers style-controlled voice rendering for the target workflow.
Buyer: one audio production team
Technical link: Test instruction-following multi-speaker streaming text-to-speech as the mechanism behind style-controlled voice rendering; the paper supports a hypothesis, not a market result.
Evidence
- competitor: ElevenLabs supports content creation customer support gaming education and accessibility use cases (year: 2025) (source)
- competitor: AssemblyAI provides pre-recorded and real-time speech-to-text and voice agent APIs (denominator: AssemblyAI homepage statement) (source)
- adoption: 7.2% of EU enterprises used AI to convert spoken language into machine-readable format (year: 2025 region: EU denominator: enterprises with 10 or more employees or self-employed persons in covered NACE sections) (source)
Differentiation: AssemblyAI provides audio platform capabilities; this adds a measured production boundary for the proposed method.
First test: Run 30 fixed scenarios across three latency or speaker settings and record failures.
Success measure: P95 first-audio or response latency stays below a pilot-selected 500 ms cap.
Constraints: Treat market sources as adjacent-tool or adoption proxies, use licensed or consented audio, and keep the pilot inside the named safety boundary.
3. Streaming Audio Latency Planning
Offer: A service that delivers streaming audio latency planning for the target workflow.
Buyer: one audio production team
Technical link: Test instruction-following multi-speaker streaming text-to-speech as the mechanism behind streaming audio latency planning; the paper supports a hypothesis, not a market result.
Evidence
- competitor: ElevenLabs supports content creation customer support gaming education and accessibility use cases (year: 2025) (source)
- competitor: AssemblyAI provides pre-recorded and real-time speech-to-text and voice agent APIs (denominator: AssemblyAI homepage statement) (source)
- adoption: 7.2% of EU enterprises used AI to convert spoken language into machine-readable format (year: 2025 region: EU denominator: enterprises with 10 or more employees or self-employed persons in covered NACE sections) (source)
Differentiation: ElevenLabs supports generated-audio use cases; this adds method-specific timing and turn-state evidence.
First test: Label 100 outputs or turns with two reviewers and compare against a baseline.
Success measure: Reviewer agreement reaches 85% on the primary interaction or audio label.
Constraints: Treat market sources as adjacent-tool or adoption proxies, use licensed or consented audio, and keep the pilot inside the named safety boundary.
4. Voice-Quality Acceptance Testing
Offer: A service that delivers voice-quality acceptance testing for the target workflow.
Buyer: one audio production team
Technical link: Test instruction-following multi-speaker streaming text-to-speech as the mechanism behind voice-quality acceptance testing; the paper supports a hypothesis, not a market result.
Evidence
- competitor: ElevenLabs supports content creation customer support gaming education and accessibility use cases (year: 2025) (source)
- competitor: AssemblyAI provides pre-recorded and real-time speech-to-text and voice agent APIs (denominator: AssemblyAI homepage statement) (source)
- adoption: 7.2% of EU enterprises used AI to convert spoken language into machine-readable format (year: 2025 region: EU denominator: enterprises with 10 or more employees or self-employed persons in covered NACE sections) (source)
Differentiation: AssemblyAI provides audio processing; this adds an annotated quality score for the proposed workflow.
First test: Seed 20 known failures in 50 runs and check whether the method flags them.
Success measure: At least 90% of seeded failures are detected.
Constraints: Treat market sources as adjacent-tool or adoption proxies, use licensed or consented audio, and keep the pilot inside the named safety boundary.
5. Consented Voice Release Gating
Offer: A service that delivers consented voice release gating for the target workflow.
Buyer: one audio production team
Technical link: Test instruction-following multi-speaker streaming text-to-speech as the mechanism behind consented voice release gating; the paper supports a hypothesis, not a market result.
Evidence
- competitor: ElevenLabs supports content creation customer support gaming education and accessibility use cases (year: 2025) (source)
- competitor: AssemblyAI provides pre-recorded and real-time speech-to-text and voice agent APIs (denominator: AssemblyAI homepage statement) (source)
- adoption: 7.2% of EU enterprises used AI to convert spoken language into machine-readable format (year: 2025 region: EU denominator: enterprises with 10 or more employees or self-employed persons in covered NACE sections) (source)
Differentiation: ElevenLabs supports generated audio; this adds a release gate for consent and quality defects.
First test: Run 30 candidate outputs through rights, safety, and quality approval before release.
Success measure: All prohibited or unconsented outputs are blocked and false blocks stay below 5%.
Constraints: Treat market sources as adjacent-tool or adoption proxies, use licensed or consented audio, and keep the pilot inside the named safety boundary.