10 March 2004

Testing for Failure

Remember the little quiz I gave 2 weeks ago?

Given:
A. Some holidays are rainy, and
B. All rainy days are boring.

Which of the following statements can be deduced?
1. No clear days are boring.
2. Some holidays are boring.
3. Some holidays are not boring.

Most people I asked got it correct: only statement 2 can be deduced. The problem is, they couldn't quite explain why statement 3 was wrong.

Some of the reasons given for rejecting 3 were more right than others... but they all betray an ignorance of the logical underpinning of the exercise.

"The category of not-boring is undefined, hence you cannot make deduction 3. You can't draw a Venn diagram to solve the problem..."
"Cannot be deduced.. since it is not known if non-rainy days are boring as well"
"clear days is an introduction of an extra term", or a resort to alegebra...

These are all commendable attempts at grasping the logic, but flawed ones.

The key phrase here is "Syllogistic Logic", or "Categorical Syllogisms". All sample attempts failed to recognise the type of logical exercises they were doing.

This is how to solve a syllogism. This is how to reject a conclusion as false or true, within the rules of syllogistic reasoning. There's even a link to a java exercise applet (in the first link) to test more interesting and difficult variants that aren't that easy to solve if you use the sample reasoning I quoted above.

You'd wonder, what's so great about this quiz?

Major corporations and the civil service of many countries require job applicants for managerial posts to take a barrage of tests - personality, IQ, critical reasoning, writing, and so on.

Now, one of the most popular critical reasoning tests, published in 1984 by The Psychological Corp division of Harcourt Brace has a "Deduction" section, and the example question is the one that we've been discussing.

Here's the official answer and reasoning from the test-makers. Look and weep.

"Example one, the conclusion does not follow. You cannot tell from the statements whether or not clear days are boring. Some days may be.

Example two, the conclusion necessarily follows from the statements since, according to them, the rainy holidays must be boring.

Example three, the conclusion does not follow even though you may know that some holidays are very pleasant."

Look, and weep! The reason why statement three is rejected has NOTHING to do with a logical proof or disproof! The real reason why statement three MUST be rejected, is because it commits the logical fallacy of the illicit minor. What the testers have done here is commit the fallacy of the non-sequitur, rejecting a statement on non-logical grounds.

Look, and weep! For anyone who's taken this particular test, you'll remember that the actual questions in the "Deduction" section... don't deal at all with Syllogistic Logic, but with Sentence or Predicate Logic (the usual, and more familiar operators of AND, IF, THEN, NOT, OR. Nove, and everyone else's algebra methods and mathematical approach will be correctly applied for a sentence logic question).

What I want to know... is how can a psychological testing corporation be even trusted to make a competent logic test?

Look, and weep! The critical reasoning test is broken into: inference Deduction, Interpretation, and Evaluation. Now, if you do well in the test... it doesn't mean that you'll make excellent decisions in the real world. Since when are real-world problems so easily bounded into a category?

Since when are we supposed to ignore, like in the Evaluation section, that all the evidence/statements offered MUST BE TRUE? I shudder to think that managers end up not questioning and looking critically at the data given to them, or applying multiple facets of 'critical reasoning' to attack a problem, from all ends.

What I want to know is... Why are these test-making companies even in the business, if the tests they make... make no sense?

In making decisions about important questions, it is desirable to be able to distinguish between arguments that are strong and arguments that are weak, as far as the question at issue is concerned. An argument to be strong must be both important and directly related to the question.


Now, that's rubbish. That's not a Strong statement they just defined; it's a Relevant statement they just defined. Would you trust a test-making company that can't distinguish a "relevant argument" from a "strong argument"?

If the world is so badly screwed up by idiots in managerial positions and complete baboons in the civil service, you now know who to blame.

No comments: