Most of the obvious anatomical homologies are between anatomical structures which are in active use by the species in question, but some anatomical homologies involve structures which are no longer needed but which also haven't disappeared entirely. A vestigial organ or structure is any organ or structure found in a species which is not being used as it is in other species. Contrary to popular belief, vestigial organs and vestigial structures aren't necessarily useless or functionless.
Vestigial does not mean useless or nonfunctional because it is difficult if not impossible to prove that any particular structure is actually functionless. It is possible that some vestigial organ really is functionless, but scientists and biologists don't assume so dogmatically. All that's necessary for an organ or structure to be labeled "vestigial" is for there to be homologies in other species where the use or function is clear, but that same use or function is not the case for the species in question. The use may be odd or it may simply not be identified yet.
A Whale of a Pelvic Bone
An example of such a structure is the pelvis of whales. All tetrapods (including whales) have pelvic bones. In most animals the pelvic bones are needed in order to be able to move the lower or rear set of limbs for the purpose of locomotion. In some species, such as whales, these limbs don't exist for the most part — although vestiges of them may remain.
Despite this lack of any need for them, whales still have pelvic bones. They are quite small compared to their counterparts in other animals, but they exist. Perhaps they serve some function such as helping to support the whales reproductive anatomy, but there are many different types of structures which would be better suited to such a task.
The question is, why would a whale, which basically lacks lower limbs and doesn't need pelvic bones to move, have pelvic bones that are homologous to creatures that do need pelvic bones to move? Similar homologies exist for snakes and legless lizards. Once again, the only explanation that makes sense is if these creatures evolved from a common ancestor along with all the other tetrapods.
Another common (and frequently misunderstood) example is the appendix. In humans, the appendix has little apparent function, although it now appears that it may store some immune cells. However, the analogous organ in many other species does have obvious function. Moreover, the human appendix can be positively disadvantageous in the sense that it is subject to nasty infections that can be fatal.
The appendix is a vestigial organ because it does not serve a function like the homologous organs in other animals even if it might serve a function in humans. So, the question becomes, why do humans have an appendix? (Or why does the human appendix not function like the homologous organ in other animals?) Evolution, the idea that we all have common ancestors, provides a meaningful answer. Creationism does not.