-rpi-sendspin/third_party/sendspin-go/docs/hires-audio-verification.md

# Hi-Res Audio Verification Report

**Status:** 🟡 Needs Testing
**Version:** v0.9.0
**Date:** 2025-10-26

## Executive Summary

resonate-go implements **full hi-res audio support** with 24-bit depth and sample rates up to 192kHz. However, **compatibility with Music Assistant and other Resonate servers needs verification** due to potential codec negotiation differences.

**Supported Hi-Res Formats:**
- ✅ 192kHz/24-bit PCM (lossless)
- ✅ 176.4kHz/24-bit PCM (lossless)
- ✅ 96kHz/24-bit PCM (lossless)
- ✅ 88.2kHz/24-bit PCM (lossless)
- ⚠️  Opus encoding requires 48kHz (no automatic resampling)

---

## Current Implementation Analysis

### 1. Audio Pipeline Capabilities

**Sample Rate Support** (`pkg/resonate/player.go:170-185`):
```go
SupportFormats: []protocol.AudioFormat{
    {Codec: "pcm", Channels: 2, SampleRate: 192000, BitDepth: 24},  // Hi-res
    {Codec: "pcm", Channels: 2, SampleRate: 176400, BitDepth: 24},  // Hi-res
    {Codec: "pcm", Channels: 2, SampleRate: 96000, BitDepth: 24},   // Hi-res
    {Codec: "pcm", Channels: 2, SampleRate: 88200, BitDepth: 24},   // Hi-res
    {Codec: "pcm", Channels: 2, SampleRate: 48000, BitDepth: 16},   // CD quality
    {Codec: "pcm", Channels: 2, SampleRate: 44100, BitDepth: 16},   // CD quality
    {Codec: "opus", Channels: 2, SampleRate: 48000, BitDepth: 16},  // Compressed
},
```

**Bit Depth Pipeline:**
- Server default: 24-bit (`pkg/resonate/server.go:32`)
- Player supports: 16-bit and 24-bit (`pkg/audio/decode/pcm.go:23-24`)
- Internal representation: int32 for full 24-bit range (`pkg/audio/types.go:11-12`)
- Audio output: Full 24-bit support via malgo (`pkg/audio/output/malgo.go`)

**Sample Rate Pipeline:**
- Source: Any rate (192kHz default for TestTone)
- Encoder: Opus requires 48kHz, PCM supports any rate
- Decoder: PCM supports any rate, Opus fixed at 48kHz
- Output: malgo backend accepts any sample rate and handles format reinitialization

---

## 2. Codec Negotiation

### Server-Side Logic (`internal/server/audio_engine.go:200-230`)

```go
func (e *AudioEngine) negotiateCodec(client *Client) string {
    sourceRate := e.source.SampleRate()

    // Check if client advertised support for exact source format
    for _, format := range caps.SupportFormats {
        // If source is 48kHz PCM and client wants Opus, prefer Opus
        if format.Codec == "opus" && sourceRate == 48000 {
            return "opus"
        }

        // If client supports exact source format, use PCM
        if format.Codec == "pcm" && format.SampleRate == sourceRate && format.BitDepth == DefaultBitDepth {
            return "pcm"
        }
    }

    // FALLBACK: If no exact match, try legacy fields
    // This is for backward compatibility with Music Assistant
    if codec == "opus" && sourceRate == 48000 {
        return "opus"
    }

    // Final fallback: PCM at source rate
    return "pcm"
}
```

### Issues Identified

**🔴 Critical: No Automatic Resampling for Opus**

When source is 192kHz and client supports Opus:
- Current behavior: Falls back to PCM at 192kHz
- Expected behavior (Music Assistant?): Resample 192kHz → 48kHz, encode to Opus
- Impact: Client receives uncompressed PCM instead of Opus (4x bandwidth increase)

**Test Case:**
```
Source: 192kHz/24-bit test tone
Client: Advertises Opus support
Expected: Server resamples to 48kHz and sends Opus
Actual: Server sends 192kHz PCM (no resampling)
```

**✅ Audio Output: Malgo with Full 24-bit Support**

The audio output uses malgo library (via miniaudio) which:
- Supports 16-bit, 24-bit, and 32-bit output natively
- Handles format reinitialization for format changes
- Preserves full 24-bit pipeline all the way to device playback

This means hi-res samples maintain full resolution through the entire pipeline.

---

## 3. Bandwidth Analysis

### PCM Bandwidth (192kHz/24-bit stereo):

```
Sample rate:   192,000 Hz
Channels:      2
Bit depth:     24 bits = 3 bytes
Bytes/second:  192000 × 2 × 3 = 1,152,000 bytes/s = 9.216 Mbps
```

### Opus Bandwidth (48kHz/16-bit stereo @ 256kbps):

```
Bitrate:       256 kbps (configurable, set in opus_encoder.go:31)
Bytes/second:  32,000 bytes/s = 0.256 Mbps
Compression:   36x smaller than 192kHz PCM!
```

### Impact of Missing Resampling

If Music Assistant sends 192kHz source and expects Opus:
- **Without resampling:** 9.216 Mbps per client (current)
- **With resampling:** 0.256 Mbps per client (ideal)
- **Difference:** 36x higher bandwidth usage!

For 5 simultaneous clients:
- PCM: 46 Mbps
- Opus: 1.3 Mbps

---

## 4. Compatibility Testing Plan

### Test Matrix

| Source Format | Client Codec | Expected Behavior | Status |
|--------------|--------------|-------------------|--------|
| 192kHz/24-bit PCM | PCM 192kHz | Direct PCM stream | ✅ Should work |
| 192kHz/24-bit PCM | Opus 48kHz | Resample + Opus | ⚠️  **Falls back to PCM** |
| 96kHz/24-bit PCM | PCM 96kHz | Direct PCM stream | ✅ Should work |
| 96kHz/24-bit PCM | Opus 48kHz | Resample + Opus | ⚠️  **Falls back to PCM** |
| 48kHz/16-bit PCM | Opus 48kHz | Direct Opus encode | ✅ Works |
| 48kHz/16-bit PCM | PCM 48kHz | Direct PCM stream | ✅ Works |

### Required Tests

**Test 1: Music Assistant Compatibility**
```bash
# Start resonate-go server with 192kHz source
./resonate-server --audio test_192khz_24bit.flac

# Connect Music Assistant player
# Expected: MA requests Opus, server resamples and encodes
# Actual: ???
```

**Test 2: Multi-Room Sync at Hi-Res**
```bash
# Start server with 192kHz PCM
./resonate-server --audio hires_test.flac

# Start 5 players
for i in {1..5}; do
  ./resonate-player --name "Player-$i" &
done

# Verify:
# - All players receive 192kHz PCM
# - Sync stays within 10ms
# - No dropped frames
# - Network bandwidth is acceptable
```

**Test 3: Sample Rate Switching**
```bash
# Start server with 48kHz source
./resonate-server --audio 48khz.flac

# Connect player (should get Opus)
./resonate-player --name "Test"

# Switch to 192kHz source on server
# (Would require server restart currently)

# Expected: Player handles format change gracefully
# Actual: Malgo reinitializes the device cleanly for the new format
```

**Test 4: Bit Depth Verification**
```bash
# Generate 24-bit test tone with known frequency spectrum
./resonate-server --audio 24bit_sweep.wav

# Record output from player
# Analyze frequency spectrum
# Verify: Full 24-bit dynamic range preserved until output stage
```

---

## 5. Known Issues & Limitations

### Issue 1: No Automatic Resampling for Opus 🔴

**Location:** `internal/server/audio_engine.go:209, 219`

**Current Code:**
```go
if format.Codec == "opus" && sourceRate == 48000 {
    return "opus"
}
```

**Problem:** Only uses Opus if source is already 48kHz. Doesn't resample hi-res sources.

**Fix Required:**
```go
if format.Codec == "opus" {
    // Resample to 48kHz if needed
    if sourceRate != 48000 {
        // Create resampler from sourceRate to 48kHz
        encoder.resampler = NewResampler(sourceRate, 48000, channels)
    }
    return "opus"
}
```

**Impact:** High bandwidth usage for Opus clients with hi-res sources.

---

### Issue 2: Output Format Support ✅ RESOLVED

**Previous Issue:** oto library only supported 16-bit output format.

**Current Status:** Malgo backend now supports 24-bit and 32-bit output natively. Full hi-res resolution is preserved through device playback.

---

### Issue 3: Format Reinitialization ✅ RESOLVED

**Previous Issue:** oto context could only be initialized once per process; format changes required restart.

**Current Status:** Malgo backend supports format reinitialization, allowing graceful format changes during streaming.

---

## 6. Recommendations

### Immediate Actions (v0.9.x)

1. **🔴 Priority 1: Add Resampling to Opus Path**
   - Implement automatic resampling in `audio_engine.go`
   - Use existing `pkg/audio/resample.Resampler`
   - Test with 192kHz → 48kHz Opus encoding
   - Verify bandwidth reduction

2. **🟡 Priority 2: Test with Music Assistant**
   - Deploy resonate-go server with MA
   - Verify codec negotiation compatibility
   - Test hi-res audio file playback
   - Document any protocol differences

3. **✅ Complete: 24-bit Output Support**
   - Malgo backend supports 24-bit output natively
   - Full hi-res pipeline preserved through device playback

4. **🟢 Priority 3: Add Integration Tests**
   - Create test suite for hi-res formats
   - Verify sample rate handling
   - Check codec negotiation logic
   - Measure bandwidth usage

### Future Enhancements (v1.0+)

1. **Configurable Quality Profiles**
   - "Low bandwidth" → Force Opus with resampling
   - "Balanced" → Opus for >48kHz, PCM for ≤48kHz
   - "Hi-Res" → Always use PCM at source rate

2. **Device Selection Control**
   - Allow users to select audio output device
   - Support ASIO on Windows for low-latency recording
   - Show available devices and their capabilities

---

## 7. Verification Checklist

Before marking v1.0.0 as ready:

- [ ] Verify 192kHz/24-bit PCM playback end-to-end with malgo
- [ ] Verify 96kHz/24-bit PCM playback with malgo
- [ ] Verify 24-bit audio reaches device without downsampling
- [ ] Test automatic resampling for Opus (after implementing)
- [ ] Test with Music Assistant server
- [ ] Test with other Resonate protocol implementations
- [ ] Measure multi-room sync accuracy at hi-res
- [ ] Document bandwidth requirements
- [ ] Create hi-res test audio files repository
- [ ] Profile CPU usage with hi-res streams
- [ ] Test with 5+ simultaneous hi-res clients
- [ ] Verify no audio artifacts at high sample rates
- [ ] Check for buffer underruns at 192kHz
- [ ] Test format negotiation with all supported rates

---

## 8. Test Resources Needed

**Audio Test Files:**
- `test_192khz_24bit.flac` - Full hi-res test
- `test_96khz_24bit.flac` - Mid-tier hi-res
- `test_48khz_24bit.flac` - Baseline quality
- `sweep_24bit.wav` - Frequency sweep for bit depth verification
- `dynamic_range_test.wav` - Test 24-bit dynamic range

**Test Equipment:**
- Music Assistant server instance
- Multiple player instances (5+)
- Network bandwidth monitor
- Audio spectrum analyzer
- Sync measurement tools

---

## 9. Conclusion

resonate-go has **excellent foundational support for hi-res audio** with a clean 24-bit pipeline and support for sample rates up to 192kHz. However, **critical compatibility testing is required** to ensure:

1. **Codec negotiation works with Music Assistant** (especially Opus fallback)
2. **Bandwidth optimization** through automatic resampling to Opus
3. **Multi-room sync accuracy** maintained at hi-res rates
4. **No audio quality degradation** in the pipeline

The biggest concern is the **lack of automatic resampling for Opus encoding** which could cause 36x higher bandwidth usage when Music Assistant expects Opus but receives PCM.

**Status: Ready for Testing** 🧪