These types of error are typically investigated by asking test persons to form judgments about various ficitious situations. It seems to me that in some cases one may question whether the tests actually reveal what they are taken to reveal. They suffer from what might be called the distance between class-room dialogues and real conversations. (On this compare my earlier blog on experimental philosophy.)
The following example comes from the work of Amos Tversky and Daniel Kahneman, two leading proponents of this branch of research (quoted in the Wikipedia article). Test persons were given the following character sketch of a woman called Linda: "31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations". They were then asked to rank the probability of certain facts about Linda, among others "Linda is a bank teller" and, "Linda is a bank teller and is active in the feminist movement".
Test persons tended to consider the latter description more probable. The answer is wrong, we are told, because a single assertion of the form “L is X” is always more probable than a conjunction of the form “L is X and Y”. The error supposedly shows a cognitive bias (of the kind called the “conjunction fallacy”). What trips people up, apparently, is the fact that the description of Linda is more likely to fit someone who is an active feminist than someone who is a bank teller.
However, it seems to me that that is not the relevant comparison. We are told that Linda is a bank teller, so the likelihood of her being one plays no role in the context. There seems to me to be a more plausible explanation: if we are given the alternatives (1) “Linda is a bank teller FULL STOP” and (2) “Linda is a bank teller and active in the feminist movement”, the natural way to read (1) is as implying that Linda is not active in the feminist movement. So what we will actually be comparing, in all likelihood, are the descriptions “Linda is a bank teller but she is not active in the feminist movement” and “Linda is a bank teller and she is active in the feminist movement”. In other words, her being a bank teller drops out of consideration altogether: we are simply comparing the likelihoods that she might or might not be an active feminist; and, given her background, the latter comes to sound a fairly plausible alternative. (One might even think that, given that she is a bank teller, she will need other outlets for her civic involvement; maybe she even chose her trade in order to to have time for political activities.)
Undoubtedly there are such things as cognitive biases, and some of the tests carried out in this field might accordingly be revealing. But often a kind of smart-alecky attitude shines through in the manner ordinary forms of reasoning are being shown up, as though the only way of judging a claim is in accordance with the principles of propositional logic or probability calculus. This makes for a deafness to the endless subtleties of actual human conversation. That is a serious drawback, for it is, after all, the reasoning of people in everyday contexts that one purports to be studying.