Grab the CAT by the Tail


Many GMAT test-takers are often confused by how scores are generated for the test. Lots of this confusion stems from the seemingly straight-forward, but not easily explained, notion of a Computer Adaptive Test (CAT).

The confusion is pervasive enough, it seems, that GMAC often makes a point to clarify what a CAT is and does at various conferences, on their website, and on their official GMAT Blog. In a recent blog post, Dr. Eileen Talento-Miller provided some information about how CATs are designed and how they function in a way that even us lay-people can understand. Dr. Talento-Miller is a Ph.D.-certified psychometrician, and in many respects she does a sound job laying out the CAT basics in an easy-to-digest fashion.

While the basic function and characteristics of the test are clearly outlined, a couple aspects of the test could stand a bit more inspection to better clarify them for test-takers.

Number of Questions Answered Correctly

One chief question is what role the number questions answered plays in determining your score. In Dr. Talento-Miller’s blog post she states that “how many questions you completed” is a determining factor in generating scores. While this is true, some research of our own indicates a wide range of possible scores for a similar number of questions answered correctly. We studied a number of test results from the GMATPrep software that GMAC provides test-takers (and touts as using “The same technology used by the official GMAT exam”). And well…we noticed some interesting (and disconcerting) trends. Here’s what we found on the correlation between scores and number of questions answered correctly when we analyzed results for the Quantitative portion of the test:

 

GMATPrep Quant Score

# of Questions Correct

<10

12 – 13

10 – 15

12 – 17

16 – 20

12 – 13

21 – 25

15 – 19

26 – 30

15 – 21

31 – 35

16 – 21

36 – 40

15 – 24

41 – 45

19 – 23

46 – 50

21 – 26

The right column contains the ranges of questions answered correctly for the score ranges in the left column. When we first saw this we thought, “What? Really? So someone can get the same number of questions correct and end up with either a 22 or a 38? Really? Someone with 23 questions correct can get a 36 or a 50? Really?”

Apparently it’s true. We looked at more GMAT Prep results and the ranges more or less played out the same way. Which kind of defies expectation. We accept that the estimated ability component does play a role, but that the impact would be so great that the number of questions you get correct is almost irrelevant seems to defy logic.

Importance of Individual Questions

In her blog post, Dr. Talento-Miller says that “all the questions are important.” This may be true to some extent, but how true exactly is anyone’s guess. In reviewing hundreds of GMATPrep results, question by question, what we found is that there is no precise way to ascertain how the scores are generated from one question to the next. To some degree or another we believe the following factors may impact your score:

  • Location – Test-takers who score well at the beginning of the test tend to score better overall on each section than test-takers who don’t if other more quantifiable measures are constant (total number of questions correct, for example). This would indicate that questions near the beginning of the section impact score more than those later on, but we’ve also seen a number of instances where this isn’t the case.
  • Consecutive sets of Right/Wrong – Research also indicates that sets of consecutive right or wrong answers impact scoring beyond what would be expected of the aggregate increase in score over that stretch. The degree to which this circumstance impacts score is, again, indeterminate, and may depend on whether questions in that run are discriminating questions.
  • Discriminating questions – In looking at so many test results, we found that a difference in score between two otherwise similar test results often comes down to different results for a couple specific questions in a section. How and to what degree these distinctions impact score is unclear, but it leads us to believe that at some point test-takers are given discriminating questions that have a greater impact on the score than others.
  • Discriminating results – It also appears that responses to different difficulty level questions impact scores in different ways. Getting a hard question wrong is less detrimental to your score than getting an easy question wrong, as getting the former wrong is more likely to be the “unexpected” result for a given question.

What these circumstances establish is that the GMAT is a highly sophisticated test, one for which the algorithms and mechanisms used to generate scores cannot be easily replicated. Consequently, the GMAT is a very difficult test to “game” from a holistic strategy standpoint. Other standardized tests present the opportunity – whether large or small – to improve one’s score not through learning material but from an effective strategy for managing the test. The GMAT thwarts such strategies. Given this, there are a few things you want to take into account as you’re preparing to take the GMAT:

1. Yes, all questions on the GMAT are “important,” which is to say they all matter or count in some way towards your score. What’s left out here is that they’re not all equally important. How important is each question? We don’t know, but what we do know is that they’re not all equal.

2. Because all the questions aren’t all equal, it’s at a minimum ambiguous to tell test-takers to “just answer every question to the best of your ability within the time allotted.” Don’t let this statement confuse you. You DO NOT need to WORK every question, but you DO need to supply an answer for every question (even if you don’t spend any time working it). For most test-takers it is tough to actually solve every question in the allotted time without a significant drop in accuracy, so while working on every question is the goal you need to develop a solid alternative if you cannot do that.

3. The number of questions you answer correctly appears to have much less impact on your score (assuming you answer a minimum number correct) than would be expected given that this test evaluates and compares candidates. The bottom line is this: you should have a pacing approach developed and honed through practice that allows you the best chance to answer correctly as many questions as you can. There is no set way to go about this (see this post: Aesop’s Fables about GMAT timing), but the numbers show that you shouldn’t be compelled to necessarily rush through the test to get through all the questions.

4. Because the correlation between number of questions answered correctly and score is so loose, many test-takers would benefit from slowing down. While everyone has this notion of the “two-minute average” as the optimal timing for the GMAT, results would indicate otherwise. If question difficulty has a greater impact on scores than does number of questions answered. Slowing down enough to allow you to achieve greater accuracy while answering enough questions may be better than trying to reach the “two-minute average” mark to answer all questions in a section.

5. Be leery of anyone touting a specific test-taking strategy designed to get your score up. The bottom line is this: to do better on the GMAT you need to improve your knowledge base and your ability to answer questions (and harder questions) correctly more often. Your test-taking strategy will need to be determined on the basis of your testing practice and history, and honed as you take more practice tests to better understand yourself as a GMAT test-taker.

Until Next Time…

  • (877) 223-3828
CONNECT WITH US
COPYRIGHT ©2002 - 2014 BELL CURVES LLC. ALL RIGHTS RESERVED. All tests names are registered trademarks of the respective testing companies, which do not endorse and are not affiliated with Bell Curves.
BELL CURVES - 151 West 46th Street, Suite 901 - New York, NY 10036 Bell Curves is an educational services and test preparation company. It delivers high-quality consulting services, test preparation programs, and self-study resources to students throughout the country.
Equal Opportunity Employer - Privacy Policy - Refund Policy