For the storage, I'm hesitating for an SSD, because of the lifetime... Maybe I'll put my current 500gb 2.5"...
You might have lifetime issue with a random USB Flash drive but if you're willing to pay the money one of the halfway reputable SSDs costs, you won't have lifetime issues unless you put a very heavy load on it (the kind you might see on some multi-user servers and for which they make even more expensive SSDs).
The benefit of an SSD might be small in your application anyway. You didn't say exactly what you want to do so.
Do you know if the latest Ubuntu 12.04 is fully supported? I know that for the GPU, there is some issue with old ubuntu version (kernel < 3.0) but in my case I won't use the GPU.
Ubuntu might never be fully supported. Certainly the current stable version isn't. People just started working on this...
But all it means for a headless server is that you might lose a couple of watts and get higher temps.