"Evolutionary theory doesn't say that vestigial characteristics have no function. A trait can be vestigial and functional at the same time. It is vestigial not because it's functionless, but because it no longer performs the function for which it evolved. The wings of an ostrich are useful, but that doesn't mean that they tell us nothing about evolution. Wouldn't it be odd if a creator helped an ostrich balance itself by giving it appendages that just happen to look exactly like reduced wings, and which are constructed in exactly the same way as wings used for flying?
"Indeed, we expect that ancestral features will evolve new uses: that's just what happens when evolution builds new traits from old ones."
Another clarification: a vestigial characteristic occurs consistently in most individuals in a species, while an atavism is the rare resurgence of a characteristic that had been absent in previous generations. With that settled, I'll cover some examples of human vestigiality and atavisms.
The best-known vestigial organ in humans is the appendix. The original function of the appendix, which can still be observed in many herbivorous mammals, is "as a fermenting vessel ... containing bacteria that help the animal break down cellulose into usable sugars." Since humans evolved from leaf-eating animals but no longer eat leaves, we have an appendix, but one that has been greatly reduced in size. While it maintains minor immune functions, our appendices do far more harm to us than good due to the prevalence of appendicitis.
Another vestige of our past is goosebumps, which we get when we're cold or feel threatened but are basically useless. But in mammals that actually have fur, raising that fur would be useful to insulate from the cold or to appear larger when threatened – exactly the same two situations. And one more example: ear-wiggling muscles. Most people can't use them at all, and the rest still can't move their ears more than slightly. But other animals use these muscles "to move their ears around, helping them ... detect predators, locate their young, and so on."
Now for atavisms. I've already mentioned human tails: while very rare, they may contain hair, muscles, nerves, blood vessels, and most tellingly, extra vertebrae. But Coyne expands on this point by saying, "Indeed, recent genetic work has shown that we carry exactly the same genes that make tails in animals like mice, but these genes are normally deactivated in human fetuses." (I'll do a separate post that expands on the evidence from genetics.)
I want to add one atavistic feature that Coyne doesn't mention: a tiny, almost imperceptible point on the outer rim of the ear known as Darwin's tubercle. Only 10% of the population has it, but I'm lucky enough to be part of that statistic. Darwin's tubercle demonstrates our common ancestry with other primates, which have significantly more prominent pointed ears, possibly to help funnel sound into the auditory canal. Below is my ear, a macaque's ear and an example illustration from Darwin's The Descent of Man.
It's both startling and fascinating to realize that I carry tangible, visible evidence for evolution with me wherever I go. And by no means is this connection to the past is something to be ashamed of. On the contrary, to bear such tokens of our history just serves as a reminder of how far our species has come.