My Hard Drive is in a removable cage that has a lot of leeway to shake. What materials would be safe to use to stuff in between the crevice to dampen it?

Are you saying the hard drive itself has loads of clearance between the sides, or that the cage itself doesn't lock securely to the case?

I've built several systems with removable drive cages (Antec, Foxconn, etc) and I've used self-adhesive 1/4 inch thick neoprene rubber weatherstripping to attach to the cage (1/4" thick, 1/2" wide). The foam makes a more solid lock to the rest of the case and greatly muffles the rattling. I've also used this type of foam to isolation mount power supplies and brackets. The 1/4" thick foam compresses to about 1/16" but expands back to close to original thickness when compression is removed.

To take up clearance between the hard drive and the sheet metal, I can't think of a better way than to use grommets.