Chapter 1: How it all began
Last month I was honored to join NetApp and NetApp University in developing a new exam.
Someone at NetApp has probably selected me to be the perfect cast for this job. Because you can only get there on explicit invitation and can not apply for it. The whole thing came about through my membership in the NetApp United program, which I’ve been part of since 2017. The responsible manager Petya Stefanova (@PetyStefanova) had written to me on a Friday morning asking if I would like to attend “a workshop in the U.S.A.” It should take place in Durham North Carolina at NetApp’s Research Technology Park.
After the first shock was digested, I immediately called my boss and asked for permission to attend. What can I say? I have never had such a fast “go go go”.
On Monday, then the big surprise. It’s not going to Durham, but directly to the NetApp headquarters in Silicon Valley, and by the way, “you’re developing a new exam for NetApp University, then you’ll get the title for the exam you developed, Subject Matter Expert and NetApp pays everything. “Wow!
The SME’s are documented experts with at least 2 years experience performing the job role. As a reward for their participation a badge is presented to individuals who have demonstrated NetApp subject matter expertise through their participation in the development of either a technical accreditation, certification or training material.
Chapter 2: The journey to Silicon Valley
Boom, now I had stress. I’ve never been outside Europe and a trip to the US has always been a big dream of mine. Since it should start in two weeks, I had the problem that I have no passport. Average duration for such a passport is 6-8 weeks. But there is also an express option that gives you a full passport within 4 business days. The city can be royally paid with 92 €.
Then quickly fill out the ESTA USA online form (no, I’m not really a terrorist, no, I do not want to sell drugs in the U.S.A., etc.), and after an hour, I got the approval to enter.
Within one day, NetApp booked flights for me, booked the hotel and rented a car. No grunts, no discussion, just do it and you’re done. Impressive!
To my great luck, I was not alone on my journey. NetApp United member Stefan Assler (@Data_Stefan) from Switzerland was also one of the party. So we were able to fine tune which flight we take and share a rental car.
Chapter 3: The first day at Silocon Valley
Since we arrived on Saturday before the workshop in San Francisco we had a bit of time for sightseeing. Which, in the case of IT guys, means posing in front of the Silicon Valley corporations;)
Chapter 4: The item development workshop
To develop an exam at NetApp, so-called Item Development Workshops (IDW) are carried out. In our IDW we were 13 SMEs who were supposed to work on the questions. It was always important to note that there were always at least 6 SMEs in the room when it came to approving a question.
As a result, the whole process is not legally challengable or it is avoided that two SMEs look deeply into each other’s eyes and say, “Jo, that suits you!” and the answers are not technically correct.
Chapter 4.1 The first day
The first day started, as usual, with a round of introductions. Afterwards we were told how to write questions. The procedure was as follows:
In this case, congruent means whether the question is “straightforward”. So it fits the topic of the exam.
Is the question relevant for the exam or is it asking for nonsense?
3-foot-fence means in this case, the examinee can answer the question after a short thought or is it a 1-foot-fence where the examinee looks at the question and immediately recognizes the correct answer without reading the question. A 5-foot fence would then be a question that is very difficult to answer.
Distractors are the wrong answers in this case. These must be designed so that you do not immediately realize that these are the wrong answers. For example, if I ask for a network topic and two of the answers are “Ernie” & “Bert,” then they are NOT plausible distractors.
Before we were shown the right answers, we had to answer the question ourselves.
All SMEs discuss the question and the answers together.
Six simple points for each question. That should be possible. Each of us should develop 12 questions from different topics. Being a NetApp Certified Hybrid Cloud Administrator (NCHCA) exam, the range of topics was large.
For each subject area, there were corresponding specifications such as many questions of the type “reason” and how many questions of the type “recall” must be. For the type “reason” we had to develop scenarios and the examinee has to find the right solution to a question from the scenario.
For the question type “recall” we were able to develop questions according to the scheme “What is correct from the listed points”.
To keep us in mind for which skill level we are developing the exam, this drawing was written on the whiteboard:
The entire workshop was led by a so-called psychometrician. Psychometricians are people who pay special attention to the psychological approach of an exam. How does the student perceive the questions? Can the candidate see the correct answer too fast by the spelling? Is the sentence structure too complex? And much more.
Chapter 4.2: The second day
On the second day, we continuously wrote questions, discussed them with the other colleagues, corrected them and, if necessary, rewritten them again. So not really exciting. But exhausting.
A small feature was there in this workshop.
NetApp is expanding its academic alliances and for that reason had Ann Beheler, Ph.D. from the National Convergence Technology Center invited to watch the development of an exam.
For me she was a great help with spelling questions;) Thanks Ann.
Chapter 4.3: The third day
On the third day we went to the preserves. Each of the 174 questions so far has been thrown onto the screen by the psychometrician, and the six criteria congruence, relevance, 3-foot-fence, plausible distractors answer the question and discuss the question with the entire group.
The challenge: we only had 3 minutes for each question. If no consensus could be reached on the content and accuracy of the question in the 3 minutes, the question was rejected and had to be rewritten. I was a bit unlucky. Had to revise ten of my questions.
At least we had then in the evening 110 of 174 questions checked in this way.
Chapter 4: The fourth day
On the fourth day, we continued with the review of the questions and then we went to the revision of the rejected questions. Some questions were so complex that you had to read about one page of code to find a mistake in the syntax. However, questions of such high complexity were then filtered out and rewritten by the review process.
It was also important to provide an official reference for each question and the correct answers. In most cases this was the official NetApp documentation page.
So you see, the questions do not just come out of the blue, but the answers have to be detectable in a technical report or admin guide or similar.
Chapter 4.5: The last day
On the last day of our workshop, the questions were then finalized and the difficulty level of the question determined in a rating matrix.
Each question was thrown on the screen for 30 seconds, and everyone had to quickly grasp the question and answer and give a “felt” rating.
Can the examinee answer the question on the first try? Is the level of difficulty low, medium or high?
After that, our appraisals were consolidated and “conspicuous” questions were again thrown on the screen. Conspicuous questions were in the case of those who came more to the evaluation that one can not answer the question.
So these questions were rewritten and corrected until all SMEs reached consensus.
Chapter 5: The Hybrid Cloud Admin Exam
Why should you take this exam?
First of all, you should of course take the exam, because I have worked on it;)
Joking aside…
The “cloud” has existed for many years. Everyone has heard something about her, everyone is talking about it. However, over the years that the cloud has existed, it has also become apparent that “the cloud” is not the solution to all the challenges of a fast-paced IT world.
The solution is somewhere in between. So between cloud and on-premise solutions. The hybrid cloud. I have parts of my data and workloads in my own data center, other parts in the cloud.
This holds great potential, many opportunities, but also some stumbling blocks.
To demonstrate that you know how the hybrid cloud works, how to operate and administer it, how connectivity to hyperscales works, and what the technical capabilities are, you should definitely take this exam.
Be one of the first to take this exam. It will certainly not hurt your career. The exam will be released on NetApp Insight 2018 and will be named NetApp Certified Hybrid Cloud Administrator (NCHCA) NS0-300.
As you know, you can take the exams at Insight for free. So just try it.
Here is your way to NetApp Insight 2018 registration.
Thank you for reading
Greetings from San Francisco
Der Schmitz
P.S .: No I have no photographic memory. No I can not tell you what the answers to the questions are. Please refrain from such requests.
Special thanks to...
@Mercedes_Adams for your almost maternal commitment to us,
@Haley_Heter for organizing the workshop,
@Linda Dolen for helping organizing us,
@Scott_Gelb for your experience in writing exam questions,
@Stefan_Assler for being there with me 😉
@Petya_Stefanova for your recommendation, having me as SME,
@NetApp and @NetApp University for the outstanding hospitality.
DISCLAIMER: This post is my personal observation and is not officially authorized by NetApp or others. Misinterpretations or misunderstandings reserved.
This is a great and wonderful story.
I am sure that I will remember you when I take the exam.