Personally, I keep all my scrapbook paper scraps and I even keep the scraps from photos that I crop. You just never know where you'll be able to use a 1/2 inch square of water or flowers, but they can come in very handy. There is just no greater thrill for me than to use a button I've gotten off an old shirt or to use a scrap rather than tear up a perfectly good 12" paper for an inch. I keep pockets from jeans. I keep bottle caps from beer (I'm not a big drinker so I get them from friends). I even keep the mirrors from my make-up compacts. They pop right out ladies. It may sound crazy but it really does make me feel good, one that I don't have to go buy something for a craft and that I'm helping the planet at the same time.
Popsicle sticks, ribbons from packages, buckles and snaps, old keys...anything can work. I'd just love to hear what items you keep for your hobbies. I know there's tons of things I'm not even thinking of.