"I kind of wonder about your "Hooked up a 1k resistor and red LED.." - If this LED wasn't a fast response part*, could this be the cause of the problem?" - FredM
Just tried a 220R and am seeing essentially the same spectra. I think if there were any digital issues at the serial interface they would show up as gross issues with the spectra. I'm actually kind of shocked the results are as good as this, with the second harmonic ~80dB down (the sine waveform has its LSBs simply truncated to fit 16 bits, rather than noise shaped or dithered). What I don't get is why feeding it a 0dB digital signal causes severe clipping, though a quick web search seems to indicate others have perhaps similar problems with SPDIF levels:
It's interesting that the value of -11dB mentioned in the first link is suspiciously close to the >> 2 bit / -12dB case I'm seeing. I have the huge advantage in that I know exactly what information the serial SPDIF contains. Those poor slobs are shooting in the dark behind a wall of drivers and other SW, trying to make sense of things. Yet I still don't have a clue. The data should be aligned so that MSB is in the MSB location, and any unused LSBs set to zero.
I see a Toslink output on my PC motherboard but not sure I've got the time nor the patience to see what's going on with that.
OK, the SPDIF 0dB clipping mystery is solved. After pouring over the Cirrus Logic 8416 datasheet and probing some pins it's clear that there are no errors in the serial stream the FPGA tester is providing via LED. Probing the Everest Semiconductor ES7144LV DAC analog right output pin with -12dB, -6dB, and 0dB SPDIF levels from the tester shows a ~0.5Vp-p, ~1Vp-p, and ~2Vp-p triangle waveform with no clipping. Probing the DAC box right side line level output shows clipping for the latter two cases. So the clipping is happening in the final DAC box line out conditioning / drive circuitry, which, for whatever crazy reason, has ~4x too much analog gain.
To be able to adapt to this kind of unforeseen nonsense I suppose I'll have to incorporate a sticky "SPDIF gain" user control setting somewhere in the bowels of my digital Theremin.