Many have trouble comprehending that "not believing X" (not believe gods exist) doesn't mean the same as "believing not X" (believe gods do not exist). The placement of the negative is key: the first means not having the mental attitude that proposition X (gods exist) is true, the second means having the mental attitude that proposition X (gods exist) is false. The difference here is between disbelief and denial: the first is disbelief in the broad or narrow sense whereas the second is denial.
The distinction here should be relatively simple and straightforward, but it's difficult to explain when someone doesn't automatically "get it." The stumbling block for many people seems to be the assumption that when faced with any given proposition, the only options are to either believe that it is true or believe that it is false — so when faced with the question of whether any gods exist, a person must believe that either at least one god exists or believe that it is false that any gods exist (in other words, deny that any gods exist).
This is incorrect. It may be that most of the propositions that come immediately to mind are those which we either positively believe are true or deny as false, but there are a myriad of other propositions which don't fall into either category. A little careful thinking about a couple of hypothetical scenarios may help reveal how this is the case.
An Atheist's Yellow Shirt
Do you believe that I am wearing a yellow shirt? To put the question more technically, do you believe the proposition "Austin is wearing a yellow shirt" is true? It's a simple question where the meanings of all the terms are relatively straightforward, so there shouldn't be any issue with comprehension. There are lots of people wearing yellow shirts every day, so there is no issue with logical or physical impossibility.
The biggest problem is ignorance: you really have no way of knowing what I am wearing right now. I might not even be wearing a shirt, never mind a yellow one. There's just no reasonable basis for you to believe that I am wearing a yellow shirt. You should believe that it's possible that I'm wearing a yellow shirt, and you might reasonably believe that I sometimes wear a yellow shirt, but you shouldn't believe that I am wearing one right now.
By the same token, though, you also shouldn't believe that the proposition "Austin is wearing a yellow shirt" is false. Your ignorance of what I am wearing should prevent you from denying this proposition the same way it prevents you from affirming it. As you can see, not believing that this proposition is true isn't the same as believing the proposition is not true: not affirming that I am wearing a yellow shirt doesn't entail denying that I am wearing a yellow shirt.
This describes the most basic level of disbelief: you don’t actively believe my claim, but you don’t deny it either. Many atheists take this position with respect to theistic claims when those claims are too vague or incoherent to adequately evaluate. Obviously such claims don’t merit rational belief, but there simply isn’t enough substance to say anything more about them.
A broader range of responses to a proposition can be seen through a more complicated scenario. If I tell you that I visited Canada last week, would you believe me? Visiting Canada is common and unremarkable, so there is no reason to think that my statement is false, but, you also have no reason to think it is true. You could accept me at my word, but you could just accept my claim as plausible without giving it further thought, much less believing it is true.
We can modify my claim to state that I crawled from my house to Canada. Again, such a feat is logically possible — but it also isn’t very likely. Why would anyone do such a thing? While you might step right up to assert that my claim is false, a more cautious position would be to “reject belief in” my claim pending further evidence and support. You aren’t actively believing it (because it seems implausible) but you aren’t denying it either (because it’s not impossible).
Once again, not believing that a proposition is true doesn't entail believing that the proposition is false. This narrower form of disbelief is also a common atheistic reaction to theistic claims. In such cases the claims are coherent and understandable, but there is a lack of substantive support — for example, actual evidence to back the claims up. Lacking sufficient evidence to warrant rational belief, the atheist does not adopt the belief — but the atheist also does not necessarily deny the claim due to a similar lack of contrary evidence. The reaction, then, is to simply “reject belief in” the claim because the theist offers no good reasons to believe.
Belief vs. Disbelief vs. Denial
It should be fairly clear now that there are more that while there is no middle ground between the presence or absence of belief that some proposition is true, it's not the case that one has to either positively affirm or actively deny that any given proposition is true. Disbelief in a proposition may be nothing more than the absence of belief that the proposition is true and this may be due to nothing more than ignorance of the proposition, though it may be due to other reasons, like a desire to think longer about it or gather more evidence.
Disbelief in a proposition might go a bit further as the active rejection of belief that it is true, but without going so far as to actively deny that it is true. Denying that a proposition is true is the same as affirming that it's contradictory proposition is true but the evidence might not warrant this any more than affirming the truth of the original claim.
Once again, there are any number of possible reasons for why a person might refuse to accept a proposition as true without also asserting the contradictory is true — they don't even have to be good reasons. What matters is the fact that a failure or refusal to believe a proposition, including the proposition that some god is true, doesn't require that one assert that that proposition s false.