Most atheists and agnostics were not raised that way, especially in America where secular atheism remains a minority. Instead, most atheists were raised in religious households and among them, most were raised in Christian environments: Protestant, Catholic, Orthodox, Pentecostal -- the entire spectrum of Christianity. Even if the family didn't go to church regularly, the type of Christianity preferred tended to affect the overall religious and familial environment. If you were Christian once, what sort was it?
I've seen it suggested that the type of Christianity a person was raised with can have a strong effect on their atheism later on. A person with a fundamentalist background, for example, might become an atheist who is far more inflexible than a person who had a more liberal Catholic upbringing. Do you suppose this might be true? After all, we can't even always identify the ways in which our upbringing affects our attitudes, methods of thinking, and personal inclinations.
Given this, it's not hard to imagine that some aspects of our religious background -- especially if the religion was indoctrinated into us from a very young age -- will continue to influence our habits of thinking and attitudes even when we have long since abandoned the outward trappings of that religion. Perhaps we shouldn't be so quick to say that we have "given up" religion unless we have spent some time looking how the unconscious ways in which a religion might continue to affect the way we approach questions about politics, philosophy, and of course religion itself.