One aspect you have not covereed is the poor and dodgy software driving some usb interfaces - more hassle than it is worth. Another aspect is the less than perfect usb interfaces found on pcs. It can take a lot of messing about to get extraneous noise suppressed properly.
One last aspect has nothing to do with usbs; it's simply that noiseless computers are very difficult to implement. With 24/96 audio I expect better than the CD thd+n of 95dB or so. This is very hard to get on a computer as many sound cards are just not good enough.
