One source, perhaps the most common, for a breakdown of this absolute male/female dichotomy comes from considering how to classify a transgendered individual. The usual gender identity theory response to this is to talk in terms of two different kinds of sex: "sex" and "gender" which correspond, roughly, to the body and mind respectively. Most people have their sex and gender the same, but this division allows us some way of understanding those who don't, and of seeing what constitutes our society's common perception of sex and how it affects such individuals. You are unlikely to find this distinction between "sex" and "gender" in the dictionary, which will typically tell you that the two words have identical meanings, except that "gender" is primarily a grammatical term which is only roughly associated with sex, that its use as a verb is archaic, and depending on your dictionary, that using it to mean sex is "loose or jocular" (Chambers) or "Obs. or Colloq." (Webster's). Personally, I'm not desperately fond of this constructed usage of "gender" and would rather the word were kept for its grammatical meaning, but it is a useful concept, and the word has become established.
Rather than "gender" and "sex", I would prefer to use the unwieldy but more obvious terms "psychological sex" and "physical sex". However, if we look more closely at what is meant by our gender, we find that a significant portion of it is what sex we socialize as. Which is to say, what sex we present ourselves as to other people, and what sex they treat us as. There is no need for this to align with what sex we think of ourselves as being. Someone in convincing drag may well be treated as a member of the sex they are presenting as, yet their internal model of their sex will probably remain unchanged -- they know they are putting on an act. So we can further divide gender into two components, a social sex which is how a person is perceived by others, and a psychological sex which is how they perceive themselves.
So having examined gender a little more closely, let's do the same for sex. Humans are good enough at pattern matching that we can classify a person's physical sex as male or female based on their body shape, and its sexual characteristics. However, we have to bear in mind that that body may have been shaped by hormones and reassignment surgery. What we see as a person's sex may well differ from that seen by, to take an extreme example, a pathologist working with bones and DNA. It seems reasonable, then, to divide sex too into two further kinds, a physical sex which is the general bodily form, and a biological sex which is what the biology you can't see would tell you. This gives us a total of four kinds of sex: social, psychological, physical and biological.
Thus far, there has been an implicit assumption that each kind of sex has one of only two possible values: male or female. This is clearly wrong. We are not genetically restricted to having either XX or XY chromosomes; the human body is not always unambiguously male or female. When I was first introduced to the concept of gender, I spent a long time puzzling at the idea of having a sex which was intrinsic to one's thought processes, rather than a reflection of the external body. The obvious assumption is that my psychological sex is happily aligned with the other three, but I think a better explanation is that it isn't really either male or female, and would feel no more out of place in a body of any other sex. So instead of saying that a sex can be male or female, we must allow for an alternative, or better a number of alternatives. A first look at these alternatives is to be found here.
Right at the start I said that society conditions us into thinking of sex as either male or female. Does this mean that our social sex is always going to be shoe-horned into being one or the other? I think that by and large it does. Institutionally, we are required to classify ourselves as male or female. Forms say
public toilets are of two kinds; we have titles and forms of address which apply to men or women. And most people will automatically classify you as male or female, and become terribly distressed if there is any confusion over the matter.Even assuming that social sex is constrained to be either male or female, this model of sexual identity leaves us with a bewildering number of possible sexes to consider. Is it practical for day to day use? I don't think it is. Because most people have all four of their sexes aligned, and either male or female, and because most people do only think of sex as being an either-or thing, it doesn't seem unreasonable to take the pragmatic view to consider by default only social sex. Provided that we are aware that the situation is more complicated, that we are ready to adopt a more complex model to accommodate someone who does not fit our simplistic expectations.