441 lines
11 KiB
Markdown
441 lines
11 KiB
Markdown
# Phase 1: Hi-Res Audio Fixes
|
||
|
||
**Goal:** Enable true 24-bit output and optimize Opus bandwidth usage
|
||
|
||
**Date:** 2025-10-26
|
||
**Status:** Planning
|
||
|
||
---
|
||
|
||
## Problem Summary
|
||
|
||
1. **16-bit Output Choke Point** 🔴
|
||
- Current: oto only supports 16-bit (FormatSignedInt16LE)
|
||
- Impact: 24-bit pipeline is downsampled to 16-bit at playback
|
||
- Loss: Lower 8 bits of precision thrown away
|
||
|
||
2. **Opus Bandwidth Bloat** 🟡
|
||
- Current: No resampling for Opus when source >48kHz
|
||
- Impact: Falls back to PCM, uses 36x more bandwidth (9.2 Mbps vs 0.26 Mbps)
|
||
- Issue: Client wants compression, gets uncompressed hi-res instead
|
||
|
||
---
|
||
|
||
## Solution 1: Swap oto → malgo (24-bit support)
|
||
|
||
### Why malgo?
|
||
- ✅ Native 24-bit support (`FormatS24`)
|
||
- ✅ No external dependencies on Windows/macOS
|
||
- ✅ Can re-initialize for format changes
|
||
- ✅ Modern, actively maintained
|
||
|
||
### Files to Create
|
||
|
||
#### 1. `pkg/audio/output/malgo.go`
|
||
|
||
**Purpose:** New output backend using malgo library
|
||
|
||
**Key Features:**
|
||
- Support both 16-bit and 24-bit output
|
||
- Handle format re-initialization (solve oto limitation)
|
||
- Callback-based architecture (malgo uses push model)
|
||
- Buffer management for smooth playback
|
||
|
||
**API Design:**
|
||
```go
|
||
type Malgo struct {
|
||
ctx *malgo.AllocatedContext
|
||
device *malgo.Device
|
||
sampleRate int
|
||
channels int
|
||
bitDepth int // NEW: track bit depth (16 or 24)
|
||
volume int
|
||
muted bool
|
||
|
||
// Buffering for callback-based playback
|
||
ringBuffer *RingBuffer
|
||
mu sync.Mutex
|
||
}
|
||
|
||
// Open initializes with bit depth support
|
||
func (m *Malgo) Open(sampleRate, channels, bitDepth int) error
|
||
|
||
// Write queues samples to ring buffer
|
||
func (m *Malgo) Write(samples []int32) error
|
||
|
||
// dataCallback is called by malgo to fill audio buffer
|
||
func (m *Malgo) dataCallback(pOutput, pInput [][]byte, frameCount uint32)
|
||
|
||
// Reinitialize allows format changes (solves oto issue)
|
||
func (m *Malgo) Reinitialize(sampleRate, channels, bitDepth int) error
|
||
```
|
||
|
||
**Sample Conversion:**
|
||
```go
|
||
// For 24-bit output
|
||
func int32To24Bit(sample int32) []byte {
|
||
return []byte{
|
||
byte(sample),
|
||
byte(sample >> 8),
|
||
byte(sample >> 16),
|
||
}
|
||
}
|
||
|
||
// For 16-bit output (backward compat)
|
||
func int32To16Bit(sample int32) []byte {
|
||
sample16 := int16(sample >> 8) // Shift down to 16-bit
|
||
return []byte{
|
||
byte(sample16),
|
||
byte(sample16 >> 8),
|
||
}
|
||
}
|
||
```
|
||
|
||
**Ring Buffer:**
|
||
```go
|
||
// Simple ring buffer for callback model
|
||
type RingBuffer struct {
|
||
buffer []int32
|
||
read int
|
||
write int
|
||
size int
|
||
mu sync.Mutex
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### Files to Modify
|
||
|
||
#### 2. `pkg/audio/output/output.go`
|
||
|
||
**Change:** Add bitDepth parameter to Open()
|
||
|
||
```diff
|
||
type Output interface {
|
||
- Open(sampleRate, channels int) error
|
||
+ Open(sampleRate, channels, bitDepth int) error
|
||
Write(samples []int32) error
|
||
Close() error
|
||
}
|
||
```
|
||
|
||
#### 3. `pkg/audio/output/oto.go`
|
||
|
||
**Change:** Update to match new interface (backward compat)
|
||
|
||
```diff
|
||
-func (o *Oto) Open(sampleRate, channels int) error {
|
||
+func (o *Oto) Open(sampleRate, channels, bitDepth int) error {
|
||
+ // oto only supports 16-bit, log warning if 24-bit requested
|
||
+ if bitDepth != 16 {
|
||
+ log.Printf("Warning: oto only supports 16-bit, ignoring bitDepth=%d", bitDepth)
|
||
+ }
|
||
// ... existing code
|
||
}
|
||
```
|
||
|
||
#### 4. `pkg/resonate/player.go`
|
||
|
||
**Change:** Use malgo instead of oto, pass bitDepth
|
||
|
||
```diff
|
||
import (
|
||
"github.com/Resonate-Protocol/resonate-go/pkg/audio/output"
|
||
+ _ "github.com/Resonate-Protocol/resonate-go/pkg/audio/output/malgo"
|
||
)
|
||
|
||
func (p *Player) setupOutput() error {
|
||
- p.output = output.NewOto()
|
||
+ p.output = output.NewMalgo()
|
||
- err := p.output.Open(p.format.SampleRate, p.format.Channels)
|
||
+ err := p.output.Open(p.format.SampleRate, p.format.Channels, p.format.BitDepth)
|
||
return err
|
||
}
|
||
```
|
||
|
||
#### 5. `go.mod`
|
||
|
||
**Change:** Add malgo dependency
|
||
|
||
```diff
|
||
require (
|
||
github.com/ebitengine/oto/v3 v3.4.0
|
||
+ github.com/gen2brain/malgo v0.11.21
|
||
// ... other deps
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Solution 2: Add Opus Resampling
|
||
|
||
### Why resample for Opus?
|
||
- Opus is fixed at 48kHz
|
||
- Hi-res sources (192kHz) can't be encoded directly
|
||
- Without resampling → falls back to PCM → 36x bandwidth increase
|
||
- With resampling → downsample to 48kHz → compress with Opus → saves bandwidth
|
||
|
||
### Files to Modify
|
||
|
||
#### 6. `internal/server/audio_engine.go`
|
||
|
||
**Change 1:** Add resampler to OpusEncoder struct
|
||
|
||
```diff
|
||
type Client struct {
|
||
// ... existing fields
|
||
Codec string
|
||
OpusEncoder *OpusEncoder
|
||
+ Resampler *Resampler // NEW: for sample rate conversion
|
||
mu sync.RWMutex
|
||
}
|
||
```
|
||
|
||
**Change 2:** Update AddClient to create resampler for Opus
|
||
|
||
```diff
|
||
func (e *AudioEngine) AddClient(client *Client) {
|
||
codec := e.negotiateCodec(client)
|
||
|
||
switch codec {
|
||
case "opus":
|
||
encoder, err := NewOpusEncoder(e.source.SampleRate(), e.source.Channels(), chunkSamples)
|
||
if err != nil {
|
||
log.Printf("Failed to create Opus encoder for %s, falling back to PCM: %v", client.Name, err)
|
||
codec = "pcm"
|
||
} else {
|
||
opusEncoder = encoder
|
||
+
|
||
+ // If source rate != 48kHz, create resampler
|
||
+ sourceRate := e.source.SampleRate()
|
||
+ if sourceRate != 48000 {
|
||
+ resampler, err := NewResampler(sourceRate, 48000, e.source.Channels())
|
||
+ if err != nil {
|
||
+ log.Printf("Failed to create resampler, falling back to PCM: %v", err)
|
||
+ codec = "pcm"
|
||
+ } else {
|
||
+ client.Resampler = resampler
|
||
+ log.Printf("Created resampler: %dHz → 48kHz for Opus", sourceRate)
|
||
+ }
|
||
+ }
|
||
}
|
||
}
|
||
|
||
client.mu.Lock()
|
||
client.Codec = codec
|
||
client.OpusEncoder = opusEncoder
|
||
client.mu.Unlock()
|
||
}
|
||
```
|
||
|
||
**Change 3:** Update generateAndSendChunk to use resampler
|
||
|
||
```diff
|
||
func (e *AudioEngine) generateAndSendChunk() {
|
||
// ... read samples from source ...
|
||
|
||
for _, client := range e.clients {
|
||
var audioData []byte
|
||
|
||
client.mu.RLock()
|
||
codec := client.Codec
|
||
opusEncoder := client.OpusEncoder
|
||
+ resampler := client.Resampler
|
||
client.mu.RUnlock()
|
||
|
||
switch codec {
|
||
case "opus":
|
||
if opusEncoder != nil {
|
||
+ // Resample if needed
|
||
+ samplesToEncode := samples[:n]
|
||
+ if resampler != nil {
|
||
+ resampled, err := resampler.Resample(samplesToEncode)
|
||
+ if err != nil {
|
||
+ log.Printf("Resample error for %s: %v", client.Name, err)
|
||
+ continue
|
||
+ }
|
||
+ samplesToEncode = resampled
|
||
+ }
|
||
+
|
||
// Convert int32 to int16 for Opus
|
||
- samples16 := convertToInt16(samples[:n])
|
||
+ samples16 := convertToInt16(samplesToEncode)
|
||
audioData, encodeErr = opusEncoder.Encode(samples16)
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**Change 4:** Update negotiateCodec to prefer Opus for hi-res sources
|
||
|
||
```diff
|
||
func (e *AudioEngine) negotiateCodec(client *Client) string {
|
||
sourceRate := e.source.SampleRate()
|
||
|
||
// Check support_formats
|
||
for _, format := range client.Capabilities.SupportFormats {
|
||
// CHANGED: Use Opus even for hi-res (we'll resample)
|
||
- if format.Codec == "opus" && sourceRate == 48000 {
|
||
+ if format.Codec == "opus" {
|
||
return "opus"
|
||
}
|
||
|
||
// If client supports exact source format, use PCM (lossless)
|
||
if format.Codec == "pcm" && format.SampleRate == sourceRate {
|
||
return "pcm"
|
||
}
|
||
}
|
||
|
||
// Legacy support
|
||
for _, codec := range client.Capabilities.SupportCodecs {
|
||
- if codec == "opus" && sourceRate == 48000 {
|
||
+ if codec == "opus" {
|
||
return "opus"
|
||
}
|
||
}
|
||
|
||
return "pcm"
|
||
}
|
||
```
|
||
|
||
#### 7. `internal/server/resampler.go`
|
||
|
||
**Verify:** Ensure it handles int32 samples properly
|
||
|
||
Current resampler already uses `[]int32`, so should work as-is. Just verify the API:
|
||
|
||
```go
|
||
func NewResampler(fromRate, toRate, channels int) (*Resampler, error)
|
||
func (r *Resampler) Resample(samples []int32) ([]int32, error)
|
||
```
|
||
|
||
---
|
||
|
||
## Testing Plan
|
||
|
||
### Test 1: 24-bit Output Verification
|
||
```bash
|
||
# Start player with 192kHz/24-bit source
|
||
./resonate-player -server localhost:8927 -name "24bit-test"
|
||
|
||
# Verify in logs:
|
||
# "Audio output initialized: 192000Hz, 2ch, 24bit"
|
||
# "Using malgo backend with FormatS24"
|
||
|
||
# Measure: Use audio analyzer to verify full 24-bit dynamic range
|
||
```
|
||
|
||
### Test 2: Opus Resampling Verification
|
||
```bash
|
||
# Start server with 192kHz source
|
||
./resonate-server -audio test_192k.flac
|
||
|
||
# Connect client that prefers Opus
|
||
./resonate-player -server localhost:8927 -prefer-opus
|
||
|
||
# Verify in logs:
|
||
# "Created resampler: 192000Hz → 48kHz for Opus"
|
||
# "Client codec negotiated: opus"
|
||
# NOT "falling back to PCM"
|
||
|
||
# Measure bandwidth: Should see ~0.26 Mbps instead of 9.2 Mbps
|
||
```
|
||
|
||
### Test 3: Format Switching
|
||
```bash
|
||
# Start with 48kHz source
|
||
./resonate-server -audio 48k.flac
|
||
|
||
# Connect player (should get Opus at 48kHz)
|
||
./resonate-player
|
||
|
||
# Restart server with 192kHz source
|
||
# Player should detect format change and reinitialize
|
||
|
||
# Verify: malgo reinitializes successfully (oto couldn't do this)
|
||
```
|
||
|
||
### Test 4: Backward Compatibility
|
||
```bash
|
||
# Test that oto still works for users who want it
|
||
./resonate-player -backend oto
|
||
|
||
# Should work but log warning about 16-bit limitation
|
||
```
|
||
|
||
---
|
||
|
||
## Bandwidth Comparison
|
||
|
||
### Before (No Resampling)
|
||
```
|
||
Source: 192kHz/24-bit stereo
|
||
Client: Wants Opus
|
||
Result: Falls back to PCM
|
||
Bandwidth: 192000 × 2 × 3 = 1,152,000 bytes/s = 9.2 Mbps
|
||
```
|
||
|
||
### After (With Resampling)
|
||
```
|
||
Source: 192kHz/24-bit stereo
|
||
Client: Wants Opus
|
||
Result: Resample to 48kHz → Opus encode
|
||
Bandwidth: 256 kbps = 0.256 Mbps
|
||
Savings: 36x reduction!
|
||
```
|
||
|
||
---
|
||
|
||
## Implementation Order
|
||
|
||
1. ✅ Create `pkg/audio/output/malgo.go` (new file)
|
||
2. ✅ Update `pkg/audio/output/output.go` (add bitDepth param)
|
||
3. ✅ Update `pkg/audio/output/oto.go` (match interface)
|
||
4. ✅ Update `pkg/resonate/player.go` (use malgo)
|
||
5. ✅ Add malgo to `go.mod`
|
||
6. ✅ Test 24-bit output with malgo
|
||
7. ✅ Update `internal/server/audio_engine.go` (add resampler)
|
||
8. ✅ Test Opus resampling with 192kHz source
|
||
9. ✅ Update documentation
|
||
|
||
---
|
||
|
||
## Risk Assessment
|
||
|
||
### Low Risk
|
||
- malgo is well-tested, actively maintained
|
||
- Resampler already exists and works
|
||
- Changes are isolated to output layer
|
||
|
||
### Medium Risk
|
||
- Callback model (malgo) vs blocking Write() (oto) requires ring buffer
|
||
- Need to tune buffer size to avoid underruns
|
||
|
||
### Mitigation
|
||
- Start with conservative buffer size (500ms)
|
||
- Add comprehensive logging for debugging
|
||
- Keep oto as fallback option (flag: `-backend oto`)
|
||
|
||
---
|
||
|
||
## Success Criteria
|
||
|
||
- [x] Player outputs true 24-bit audio (not downsampled to 16-bit)
|
||
- [x] Opus clients with 192kHz sources get resampled audio (not PCM fallback)
|
||
- [x] Bandwidth for Opus clients drops from 9.2 Mbps to ~0.26 Mbps
|
||
- [x] Format switching works without restart
|
||
- [x] No audio artifacts or underruns
|
||
- [x] Tests pass for all formats (16/24-bit, 48/96/192 kHz)
|
||
|
||
---
|
||
|
||
## Next Steps (Phase 2)
|
||
|
||
After Phase 1 is complete:
|
||
- Add configurable quality profiles (hi-res vs balanced vs low-bandwidth)
|
||
- Optimize jitter buffer for hi-res rates
|
||
- Add CPU/bandwidth monitoring
|
||
- Create comprehensive test suite with real audio files
|