Thanks very much for the suggestion. You are probably right about the limitation of spdif (though I'd be surprised if optical was also limited in this way), and on top of that I'm just using the yellow leads from a cheap RCA AV cable as my coax. However, I tried HDMI but that didn't seem to help. It is of no real consequence. As long as I get my 5.1 movies and CD quality audio that is what counts, and 5.1 audio is a nice bonus. 96/24 is overkill IMO but unfortunately once you go to 5.1 audio it is usually par for the course, which means the files are truly massive.
the test file I have is confusing. It is a DTS 5.1 flac. Flac is supposed to be lossless, but as I understand it, in this case it is acting more like a container for the DTS which is lossy, though still very high bitrate. Honestly, it's a wonder that I can play it at all!
Really I just wanted to know if I was able to play these files and to see if I could hear a difference. Since I can't, I will just avoid this massive format! If anyone knows of any good sources for 5.1 gently compressed (DTS or AC3) 48/16 audio please pass it along.
I really enjoy being able to use my htpc with my receiver either turned off, or set to another audio input, so COAX is much better for me, with the HDMI going direct to the TV.