• New methods of testing
• Adaptive testing
• Simulations and case studies
• Likely candidates for simulation testing

Microsoft seems to have put almost as much effort into the new Windows 2000 exams as the product itself, in a determined effort to re-establish credibility for the whole MCSE/MCP scheme. MCSE certification has come to be seen as a joke by some employers, who regard the exams as too easy to pass and bearing insufficient resemblance to real problems that occur in a live network.

New methods of testing

For Windows 2000, the exam content has been drastically revised, but this largely reflects the more complex and wide-ranging nature of the product. The more significant changes are in the method of assessment. The multiple choice questions that do little more than test a candidate’s ability to recall facts are being replaced by two new techniques designed to measure a candidate’s ability to think on the hoof and solve real problems without being able to fall back on basic memory skills. These are adaptive testing and simulation/case study. With adaptive testing, each question depends on the answer to the last so that the level of difficulty can be quickly tailored to the candidate’s ability. Simulation and case study testing attempts to measure on-the-job skills rather than rote recall, by presenting candidates with realistic scenarios where some of the newly acquired skills have to be applied.

To some extent Microsoft is playing catch up with other major IT certification schemes, notably those of Novell and Cisco, but is also stretching the envelope further. Indeed the whole field of IT testing is moving fast, with the development of online methods that exploit the computer’s ability to present simulations and score answers almost instantly, so that the whole examination process can become more flexible and interactive, reflecting the real world more closely and making it harder to cheat.

The latter is also a sore point for Microsoft, whose exams have suffered more than most in the past from over familiarity as candidates report on the type of questions posed and even post them up on the web. This is an unavoidable problem for all exams that are taken on an ongoing basis rather than simultaneously in the traditional way, but nonetheless it has contributed to the loss of credibility for the MCSE certification scheme. The new techniques tackle this problem by making each exam unique with each candidate given, in effect, a randomly selected set of questions, even if there are still some that crop up more than once. So the new exams are designed to provide greater proof against over familiarity as well as a better measure of hands-on ability.

This all has significant implications for how candidates prepare for exams. As well as having to embrace a much broader syllabus encompassing fields such as security and network configuration in greater depth, it is also necessary to gain familiarity with the new style of questioning.

Adaptive testing

Adaptive testing brings a particularly important difference in that it is no longer possible to revise your answers during the course of an exam. Once you have submitted an answer it is gone forever, but there is a saving grace. If you made a mistake the next question is easier, and you still have time to get back on track and reach your true level. By the same token though if you have fluked a correct answer, the next question will be harder, and you will almost certainly end up descending to your true level.

In fact with adaptive testing your entire score depends merely on the level of difficulty of the last question you answered. The advantage of this form of testing is that it is supposedly easier to sift a bunch of candidates into the correct order of merit, without having to ask too many questions. To use the jargon of the trade, it is a more granular process than basic multiple choice testing. With multiple choice questioning, it is possible to differentiate between good and bad candidates to some extent by setting questions of varying difficulty. Then good candidates score more highly than bad ones. But it is difficult to ensure that the scale of increasing difficulty is sufficiently accurate to avoid bunching and obtain a good spread of marks.

These risks are not absent from adaptive testing, where it is important that there is a smooth graduation between easy and hard questions to prevent too many candidates being caught in "ghettos" of narrow mark ranges, but it is easier to cater for a wide spread of ability within a single exam. Indeed the process has great potential for the future. Its success in assessing candidates accurately depends on the statistical likelihood that after a certain number of questions candidates have reached their correct level. This probability can be adjusted by changing the number of questions that are asked before cutting the process off. Some candidates will be given more questions than others, depending on how consistent their answers are. But the number of questions will always be fewer, in the range 15 to 30 for Microsoft exams, than in the case of multiple choice exams, which have 50 to 70 typically. The duration may well be the same, however, so candidates should have longer for each question.

So in all cases adaptive testing makes it possible to assess a candidate’s ability with acceptable accuracy in a smaller number of questions than is possible with multiple choice testing. A significant disadvantage for candidates though is that the score report will provide no feedback on which questions were answered incorrectly. Of course candidates will have some idea of their weaknesses but will be unable to tell for certain from the exam results the subject areas that need most revision. Microsoft’s answer to this criticism is that certification exams are merely measurement tools, and that some practice tests have been provided to allow candidates to assess their strengths and weaknesses. Indeed preparing for an exam based on adaptive testing requires considerable practice. With multiple choice exams, it is a good idea not to agonise too long over any question and proceed reasonably briskly to the end, in the knowledge that you can check and revise answers afterwards. This exploits the ability of the brain to continue working on problems in the background after the candidate has moved on to other questions. Sometimes the solution to a more difficult problem will occur only towards the end of an exam, having eluded the candidate at the first pass.

With adaptive testing there is no longer this scope for revision, and it is more important that the candidate hits the ground running. For this reason, practice tests are recommended. At time of writing, practice tests were available from a number of vendors, but just two, MeasureUp Inc, and Self Test Software Inc, had been approved by Microsoft.

Microsoft has been quite coy about releasing details of the new exam structure, merely affirming that these new methods will be introduced. The objective is to keep candidates guessing, but Microsoft’s UK education Manager, Clare Curtis, did confirm that there would be no more basic multiple choice tests. Of course adaptive tests are still based on the question and answer format which by its very nature bears only limited resemblance to real life. For this reason Microsoft has also introduced case study and simulation testing, bringing additional challenges for exam setters and candidates alike. Making the tests life-like is not so difficult given the increasing power of computers and their ability to simulate complete network environments. The harder part is making the tests truly representative and of the right level of difficulty, so that again bunching is avoided and candidates are relatively evenly spread across a wide spectrum of percentages.

Simulations and case studies

With simulations and case studies, candidates are given a scenario, involving perhaps configuring a server and setting up user permissions within a precisely specified Windows 2000 network. A number of tasks will be identified, and candidates will be scored for each one completed successfully, just as they would if they were given as separate questions. Some of the tasks though depend on others, and so some flexibility is needed in the marking process to give candidates credit where it is due, even for an answer that may not be correct. And because the tasks are inter-related, they will usually all have to be completed before the assessment begins.

Until a particular simulation test has been completed, candidates have the chance to consider all the answers together as part of a single whole task, but there is no going back once the solution has been submitted. Some of the tasks or questions within a case study or simulation will require multiple answers, and in such cases candidates will be credited for those that are correct. In some cases though there is a penalty for getting some of the answers wrong within a single task, over and above missing out on the marks for it. After all in the real world it is not much good only doing part of a configuration or solving just some of a user’s problems. Indeed Microsoft has stated that on some of these multi-part questions, marks will be deducted where the wrong answer has been chosen, although the minimum score for any one group of questions is zero - i.e. negative marks for questions within one set will not affect another set. In effect therefore there is a bonus for getting each task comprising a set of questions totally right.

To familiarise themselves with this new format, a demo case study exam can be downloaded from www.microsoft.com/Train_Cert/mcp/coretaq.htm and then clicking on the words "Case Study-Based Test Demo" within that page. This demo is purely for practice and familiarisation, and does not give a score.

Likely candidates for simulation testing

Some exams are naturally better suited to simulation type testing while for others adaptive testing is more appropriate. An obvious candidate for case study testing is the exam entitled "Analysing Requirements and Defining Solution Architectures", and so not surprisingly Microsoft has chosen to base this on the new simulation format. However, Microsoft says it is considering case study testing for other exams with similar content. This could be quite a few in the case of Windows 2000.

Let’s consider the exams to be taken so that we can identify which are most likely to incorporate simulation testing: