I recently got my 2006-era HTPC out of storage and realized it's just not cutting it any more. I installed Precise Pangolin and it won't smoothly play DVDs at 1920x1080 using XBMC. Everything else about the software stack is making me pretty happy, except my CPU runs at 70%, and memory similarly used up. I had to create a swap file (something I haven't done in years) to prevent running out of memory!

This is what I have:
8400GS passive video card
AMD single-core 2.2GHz 90W
mATX motherboard
1.5GB DDR 400 ram
OCZ SSD

I'm running the system with a single 120mm fan and it seems to be doing ok heat-wise. I don't do gaming, but I do have high standards for video.

My question is, what should I upgrade? Does it make sense to buy a new video card, or should I bite the bullet and upgrade the mobo-processor-memory?

If I upgrade to one of the new Sandy Bridge or Ivy Bridge I get Intel graphics. Are these going to be better than my 8400GS? If so, then it's like upgrading my CPU and graphics at the same time. If not, it doesn't bother me to use my 8400GS...I won't feel like I'm 'wasting' the Intel graphics...but would I will still be using the same videocard I'm using now. Does it make more sense to upgrade my graphics?

If I go with AMD Llano or Trinity or whatever, I get ATI graphics. I swore off ATI for Linux a long time ago, so I would be using my 8400GS no matter what if I buy AMD. Will my current video card cut it?

Any suggestions welcome. I also don't now what kind of ram or how much, I should be getting nowadays.

According to this site:

http://www.tomshardware.com/reviews/gam ... 107-7.html

an IVB or SB cpu will have a "better" graphics chip than your card. However, I don't think your card should be having trouble playing that content, but it might just be me. You should head over to the xbmc forums to get their advice. Is VDPAU working?

I believe that VDPAU is working. I have the latest NVIDIA driver, and mplayer reports VDPAU success, and XBMC has 'use VDPAU' checked in the menu.

I think the problem might be the bloated desktop and/or XBMC more than the video itself. I have my card set to 1920x1080 so I'm doing the scaling on the computer, but I really don't want to let the TV do the scaling.

Another thing is that I'm playing over a wired network, but at these bitrates I don't think network speed is causing a problem...is there any way to check for that?

I have nothing but good things to say about the Intel HD 4000 on my i5-3570K. Plays every kind of video AND video game I've thrown at it (even played through Spec Ops: The Line, which was nothing short of cinematic). The level of Linux support I have no idea about, but they released a patch fixing the issues I was having relatively quickly, in a week or two.

The CPU and GPU run very cool as well, despite the high Hz, although I have only experimented with this big tower heatsink I use.

If it was me upgrading, I would go and upgrade mobo+CPU(+RAM) and ditch the card as unnecessary. If price was a problem, then I'd look at AMD rather than Intel, but Intel is solid in both GPU and CPU, whereas AMD still seems to waver in the CPU department.

I think you should just figure out why it doesn't play smoothly. Well the high cpu load clearly is a good indication, so you'd need to figure out what causes this.
The box should certainly be fast enough for dvd playback - even without vdpau mpeg2 decode needs very few cpu cycles (your k8 could probably decode like 10 in parallel... - you could try switching vdpau off to see if that somehow doesn't work right). Scaling shouldn't need any cpu cycles at all (all handled by the graphic chip) unless xbmc does something unusual. Granted your graphic card is terribly slow but I'd think it should be fast enough for this.
You could try something like "sudo perf top" (after installing the perf stuff) which might give some good indication where those cpu cycles are spent. Maybe the distro is just failing to setup dma mode for the dvd drive or something, pio mode could for instance cause such terrible things easily (though that shouldn't happen unless your chipset is blacklisted somewhere). Or xmbc uses some "unreasonable" (for that system) post-processing.
Or maybe you're hitting https://bugs.launchpad.net/unity/+bug/993397 - in which case the solution seems to be to not use unity3d, disabling compositing (compiz) or using other tweaks. I'd think 1.5GB should be enough to run decently, though yes I'd not be completely surprised if you'd require a swap file - as long as only some barely used stuff lands in it there's not really any performance impact.

Actually with working vdpau (but not without) you should be able to play back full hd h.264 videos without problems.

btw for the upgrade options, trinity or llano don't really make a whole lot of sense if you don't plan on using the integrated graphics (and pairing them with a graphic card which is something like 1/8 as fast as the APU itself just seems odd anyway...). You wouldn't actually need to rely on working hw acceleration for video playback however if using these chips since they'd be fast enough for software decoding even of full hd h.264 streams.
Something like a Ivy Bridge Pentium g2120 with HD2500 would even have much faster graphics than the 8400gs provides. AFAIK it should also be possible to use accelerated video decode with those chips though they lack the post-processing features of the more expensive intel cpus (but I don't think you could use them with linux anyway). Even if it wouldn't work though no biggie.