New methods of testing
Simulations and case studies
Likely candidates for simulation testing
Microsoft seems to have put almost as much effort into the new Windows 2000 exams as
the product itself, in a determined effort to re-establish credibility for the whole
MCSE/MCP scheme. MCSE certification has come to be seen as a joke by some employers, who
regard the exams as too easy to pass and bearing insufficient resemblance to real problems
that occur in a live network.
New methods of testing
For Windows 2000, the exam content has been drastically revised, but this largely reflects
the more complex and wide-ranging nature of the product. The more significant changes are
in the method of assessment. The multiple choice questions that do little more than test a
candidates ability to recall facts are being replaced by two new techniques designed
to measure a candidates ability to think on the hoof and solve real problems without
being able to fall back on basic memory skills. These are adaptive testing and
simulation/case study. With adaptive testing, each question depends on the answer to the
last so that the level of difficulty can be quickly tailored to the candidates
ability. Simulation and case study testing attempts to measure on-the-job skills rather
than rote recall, by presenting candidates with realistic scenarios where some of the
newly acquired skills have to be applied.
To some extent Microsoft is playing catch up with other major IT certification schemes,
notably those of Novell and Cisco, but is also stretching the envelope further. Indeed the
whole field of IT testing is moving fast, with the development of online methods that
exploit the computers ability to present simulations and score answers almost
instantly, so that the whole examination process can become more flexible and interactive,
reflecting the real world more closely and making it harder to cheat.
The latter is also a sore point for Microsoft, whose exams have suffered more than most in
the past from over familiarity as candidates report on the type of questions posed and
even post them up on the web. This is an unavoidable problem for all exams that are taken
on an ongoing basis rather than simultaneously in the traditional way, but nonetheless it
has contributed to the loss of credibility for the MCSE certification scheme. The new
techniques tackle this problem by making each exam unique with each candidate given, in
effect, a randomly selected set of questions, even if there are still some that crop up
more than once. So the new exams are designed to provide greater proof against over
familiarity as well as a better measure of hands-on ability.
This all has significant implications for how candidates prepare for exams. As well as
having to embrace a much broader syllabus encompassing fields such as security and network
configuration in greater depth, it is also necessary to gain familiarity with the new
style of questioning.
Adaptive testing brings a particularly important difference in that it is no longer
possible to revise your answers during the course of an exam. Once you have submitted an
answer it is gone forever, but there is a saving grace. If you made a mistake the next
question is easier, and you still have time to get back on track and reach your true
level. By the same token though if you have fluked a correct answer, the next question
will be harder, and you will almost certainly end up descending to your true level.
In fact with adaptive testing your entire score depends merely on the level of difficulty
of the last question you answered. The advantage of this form of testing is that it is
supposedly easier to sift a bunch of candidates into the correct order of merit, without
having to ask too many questions. To use the jargon of the trade, it is a more granular
process than basic multiple choice testing. With multiple choice questioning, it is
possible to differentiate between good and bad candidates to some extent by setting
questions of varying difficulty. Then good candidates score more highly than bad ones. But
it is difficult to ensure that the scale of increasing difficulty is sufficiently accurate
to avoid bunching and obtain a good spread of marks.
These risks are not absent from adaptive testing, where it is important that there is a
smooth graduation between easy and hard questions to prevent too many candidates being
caught in "ghettos" of narrow mark ranges, but it is easier to cater for a wide
spread of ability within a single exam. Indeed the process has great potential for the
future. Its success in assessing candidates accurately depends on the statistical
likelihood that after a certain number of questions candidates have reached their correct
level. This probability can be adjusted by changing the number of questions that are asked
before cutting the process off. Some candidates will be given more questions than others,
depending on how consistent their answers are. But the number of questions will always be
fewer, in the range 15 to 30 for Microsoft exams, than in the case of multiple choice
exams, which have 50 to 70 typically. The duration may well be the same, however, so
candidates should have longer for each question.
So in all cases adaptive testing makes it possible to assess a candidates ability
with acceptable accuracy in a smaller number of questions than is possible with multiple
choice testing. A significant disadvantage for candidates though is that the score report
will provide no feedback on which questions were answered incorrectly. Of course
candidates will have some idea of their weaknesses but will be unable to tell for certain
from the exam results the subject areas that need most revision. Microsofts answer
to this criticism is that certification exams are merely measurement tools, and that some
practice tests have been provided to allow candidates to assess their strengths and
weaknesses. Indeed preparing for an exam based on adaptive testing requires considerable
practice. With multiple choice exams, it is a good idea not to agonise too long over any
question and proceed reasonably briskly to the end, in the knowledge that you can check
and revise answers afterwards. This exploits the ability of the brain to continue working
on problems in the background after the candidate has moved on to other questions.
Sometimes the solution to a more difficult problem will occur only towards the end of an
exam, having eluded the candidate at the first pass.
With adaptive testing there is no longer this scope for revision, and it is more important
that the candidate hits the ground running. For this reason, practice tests are
recommended. At time of writing, practice tests were available from a number of vendors,
but just two, MeasureUp Inc, and Self Test Software Inc, had been approved by Microsoft.
Microsoft has been quite coy about releasing
details of the new exam structure, merely affirming that these new methods will be
introduced. The objective is to keep candidates guessing, but Microsofts UK
education Manager, Clare Curtis, did confirm that there would be no more basic multiple
choice tests. Of course adaptive tests are still based on the question and answer format
which by its very nature bears only limited resemblance to real life. For this reason
Microsoft has also introduced case study and simulation testing, bringing additional
challenges for exam setters and candidates alike. Making the tests life-like is not so
difficult given the increasing power of computers and their ability to simulate complete
network environments. The harder part is making the tests truly representative and of the
right level of difficulty, so that again bunching is avoided and candidates are relatively
evenly spread across a wide spectrum of percentages.
Simulations and case studies
With simulations and case studies, candidates are given a scenario, involving perhaps
configuring a server and setting up user permissions within a precisely specified Windows
2000 network. A number of tasks will be identified, and candidates will be scored for each
one completed successfully, just as they would if they were given as separate questions.
Some of the tasks though depend on others, and so some flexibility is needed in the
marking process to give candidates credit where it is due, even for an answer that may not
be correct. And because the tasks are inter-related, they will usually all have to be
completed before the assessment begins.
Until a particular simulation test has been completed, candidates have the chance to
consider all the answers together as part of a single whole task, but there is no going
back once the solution has been submitted. Some of the tasks or questions within a case
study or simulation will require multiple answers, and in such cases candidates will be
credited for those that are correct. In some cases though there is a penalty for getting
some of the answers wrong within a single task, over and above missing out on the marks
for it. After all in the real world it is not much good only doing part of a configuration
or solving just some of a users problems. Indeed Microsoft has stated that on some
of these multi-part questions, marks will be deducted where the wrong answer has been
chosen, although the minimum score for any one group of questions is zero - i.e. negative
marks for questions within one set will not affect another set. In effect therefore there
is a bonus for getting each task comprising a set of questions totally right.
To familiarise themselves with this new format, a demo case study exam can be downloaded
from www.microsoft.com/Train_Cert/mcp/coretaq.htm and
then clicking on the words "Case Study-Based Test Demo" within that page. This
demo is purely for practice and familiarisation, and does not give a score.
Likely candidates for simulation testing
Some exams are naturally better suited to simulation type testing while for others
adaptive testing is more appropriate. An obvious candidate for case study testing is the
exam entitled "Analysing Requirements and Defining Solution Architectures", and
so not surprisingly Microsoft has chosen to base this on the new simulation format.
However, Microsoft says it is considering case study testing for other exams with similar
content. This could be quite a few in the case of Windows 2000.
Lets consider the exams to be taken so that we can identify which are most likely to
incorporate simulation testing:
|There are two sets of exams, one for
candidates who have already passed three particular Windows NT 4.0 exams (70-067,
70-068, and 70-069), and the other for candidates who have not. Each set has the same
optional exams, but with different compulsory exams to establish the groundwork. For the
second group of those without the relevant NT 4.0 experience, there are four compulsory
exams, which will usually each require a five-day course plus preparation and revision.
Exam 70-210: Installing, Configuring and Administering Microsoft Windows 2000
Professional (the desktop version).
Exam 70-215: Installing, Configuring and Administering Microsoft Windows 2000
Exam 70-216: Implementing and Administering a Microsoft Windows 2000 Network
Exam 70-217: Implementing and Administering a Microsoft Windows 2000 Directory
Then for those with NT 4.0 experience, there are two options. One option is to take a set
of two exams, Exam 70-216, Implementing and Administering a Microsoft Windows 2000
Network Infrastructure, and Exam 70-217, Implementing and Administering a Microsoft
Windows 2000 Directory Services Infrastructure. Alternatively, more experienced candidates
can sit just one accelerated intensive exam that combines the above two, this being Exam
70-240, Microsoft Windows 2000 Accelerated Exam for MCPs certified on Microsoft
Windows NT 4.0. Naturally these exams for existing MCPs concentrate on the parts of
Windows 2000, such as Active Directory and the Kerberos security mechanism, that are new.
Then on top of these compulsory exams, all candidates have to take three more, chosen from
a list of core exams plus other elective options including some third party
certifications. The exact shape of this optional list has not been finalised, and
Microsoft will post up to date details on www.microsoft.com/mcp/certstep/mcse.htm.
Among these options are:
Exam 70-219: Designing a Microsoft Windows 2000 Directory Services Infrastructure, and
Exam 70-220: Designing Security for a Microsoft Windows 2000 Network.
As can be seen, some of these exams are ideally suited to a simulation/case study format,
in particular Exam 20-216 and 20-217. For others, such as 70-219 and 70-220,
a hybrid approach would be preferable, with some questions testing overall knowledge
combined with a case study to put it all into practice. Whether this is how things will
work out we will see, but it looks a good bet.