The Exams Stink - Lets Make Them Better

[Begin Rant]

I usually avoid the Cert forums on Dynamics websites, but found myself trolling through them tonight.

In the past, I remember thinking that what was frustrating was that these forums fostered ‘cheating’, and I still feel they do to some extent, in so far as they take advantage of the fact that an exam can only ask a few questions, and by seeking specific answers, they enable the test taker to learn as little as possible.

Then I realized that what was really exasperating me was that they were ‘cheating’ to learn something that wasn’t that helpful in the first place. The questions on the exam didn’t really measure ability with NAV (whatever that means) anyway.

[End Rant]

My question to you guys is what sorts of questions would you ask that address the kinds of issues that NAV professionals are really confronted with? Maybe we could put all of the questions together, provide the answers, and have an exam that is actually useful. Then it wouldn’t matter if people memorized the answers or not - because the questions would be relevant.

A couple of excercises comes to mind:

Create a routine that automatically posts Sales/Purchase/Warehouse documents.

How to use the Debugger

… (many more)

I’m curious what you guys think…

I could not agree with you more in what you are trying to say here … but…

Not sure if you read my Blog about this some time back my opinion of cheating has not changed much since. The thing is that when this cheating is actively supported in the community, the cheats really don’t feel so bad and keep doing it. Now its just a part of the certification process.

What it has led to, is a status quo, where every potential NAV client knows that the exam results are totaly worthless, and that they need to completely ignore any persons “NAV Qualifications”. Thus what is even the point of doing the exams?

I was in a situation last year, where I was selecting a NAV team for an NSC, and had narrowed it down to about 45 to try to find 10. Now the interesting thing was that everyone of the 45 had higher NAV qualifications than me. The most senior person in the room had left University just over 2 years ago. So if we were going to pick someone based on qualifications, then I would not even have made the short list.

So even if the exams were improved, what would change? These people learn the tests to pass the test, not to learn Navision, the information vanishes the day after the test anyway. I think we all just need to work with this as it is, i.e. we pay money for the test, we get a bit of paper, and we throw the paper in the trash.

I think it really is not the issue of the wrong questions, its a question of the whole concept of a multiple choice test. We are trying to use Objective exams to test a subjective quality, and that just won’t work.

The two questions you suggest above are perfect questions for a NAV exam, the question is how do you mark them? Its not like you could do a code compare, becasue the code will be differnt in each case. rememebr the days when you did tests inschool and the teacher took them home and marked them subjectively according to some guide lines. But the issue there is that you need examiners that are smarter than the people taking the tests.

My personal belief is that fo the volume of developers needed, that this is an unachivable goal, so whats the point of keeping it. me I would have that an individual gets certified by having three end users sign off that after ONE YEAR of using their solution, it worked great.

PS you are making some great posts lately, some of them woudl be even better as a Blog. If you want me to set you up a blog, just let me know. think you have a lot to contribute to the community that we would love to hear about.

Good point about the concept of a multiple choice test - you’re right, I think that’s inherently flawed.

But my point isn’t exactly to create an exam (posting title notwithstanding), but to get an idea of what our (not Microsofts) expectations are of someone that calls themselves a NAV professional (developer, finance consultant, etc.)

If we could come up with some set of questions that captured that knowledge we could:

  1. Use the questions to help us hire people that meet our expectations

  2. Help us more quickly train newcomers

  3. Expand the ‘minimum’ level of knowledge over time

The end result would be delivering better, quicker NAV solutions and that would make us all look good.

I’m not really concerned with how one would evaluate the answers to the questions (at least at this point). As you said its subjective and very time consuming. I’m interested in what the questions might be.

(by the way, thanks your kind words about my posts, I might just take you up on your offer)

It obviously depends upon who you are hiring or what you want them to do. I learnt my trade by reading manuals, attending courses, running through scenarios on the software etc. My knowledge expanded with the first implementation I was involved in. I was the only consultant at the time, so did finance through distribution. That was probably the greatest way for me to learn. The second place I learnt was here!! People asked questions and I went away to try and find out what the answer is. It should come as no surprise to know I still do this. If you are wondering why then I have a low number of posts they are only the last 9 months, they still have not corrected themselves David [:D] but my points are a better reflection!

Still to the point in hand. Quoting David again he used to set a task of building a database from scratch and processing through transactions. I feel this is an excellent test. You can set up different scenarios depending upon the sort of consultant you are employing, and naturally set traps. 18 months ago a NAV specialist came to the company I worked for the company I then worked for. He had 10 years experience and a lengthy all-encompassing CV for NAV. In his first week he was helping out on the support desk and did not know what the standard posting group error message meant, you know the one where it states there is no account for Gen. Bus. Posting Group and Gen. Prod. Posting group combination. They also knew nothing about currency. I am not saying the CV was wrong, may elaborated upon, and perhaps the experience was in slightly different areas. Actually setting them a task to do and monitoring the results is one way. For developers give them a specification and ask them to produce the work, see how many document it [:D] In the UK I believe teachers generally, as part of the interview process, have to take a class in the new school, they are given an outline brief of what the lesson should be and are assessed in front of a live audience. Gives a feel if they can do the job!

The exams are annoying but I will also add another point not mentioned here. The Microsoft partners attain status by a number of criterion. So if you want to be a gold partner you have to get “x” points. One of the sections is exams, each exam equates to points. So the partner instructs the employee to take them and perhaps puts an incentive on it. Now the employee may already be doing the job, have years of experience in the area but the exams do not measure this, they measure the menu path of “x” and the obscure tick box “y”. So I can understand why people do it. We should see them as meaningless but the authors Microsoft do not. Somewhere something has to change!

Oh and a question how about

“Does Manufacturing have any impact on Costing”?

When I took the manufacturing course the instructor stated that it did not really and chose to ignore the manual. Pre-Microsoft I grant you but clearly the authors response would be No, answer anything else and you are wrong [:D]

That’s along the same line as saying “our university grading system sucks. It doesn’t reflect how well that person does in the real world, it just shows how good a student takes a test”.

I believe your complaint is valid, I think the old Navision testing, where you had to create a solution for a golf course (I think?) with posting routines was great. But it’s time consuming and requires a lot of resource and money to maintain. I don’t think MSFT is going to have this type of testing anytime soon since it’s a strong departure from other MSFT tests.

Well, again, I’m not thinking really of creating a new exam – and I’m certainly not trying to put MSFT out of business :slight_smile:

Adam/David: The idea of being able to setup a database from scratch seems like a very good excercise. What sort of tasks would someone be expected to accomplish to focus that a bit, to say be able to create a vendor, purchase something, and sell it? Or something more complicated?

Alex: the old NAV exam did have its strengths… if you dreamed up a question like that, what would it ask?

Its on the Dynamics NAV WIKI site, here:

I also have posted it at least three times here or on mbsonline, so I will search and see if I can find more.

Clearly the explanation behind you high post count [:D][:D]

Found one of them for you:

It was a topic very dear to me, i.e. teaching Navision. Actually from what I could find I posted once on mbsonline, then revised it when we moved to Dynamics User Net, and then finalized it on the WIKI. The wiki is the most complete version, but I really need to finish it, still not complete.

I agree

Probably the best suggestion of a teaching tool I have seen, probably the reason I quote it so often [:D]

I think the essential question about the exam topic is:
Why are there any exams at all?
Obviously there are more than one possible answers to this question… - Is it a kind of “making a selection” in order to get only the good ones doing the navision programming job? Or is it possible to take an exam and have already a lot of questions answered which one might encounter in one’s programmer life? …

I think a comparision can be made with the driver’s license exam. The main purpose of such an exam should be (in my opinion) to give something like a starting point for a serious beginner. Neither will it be possible to select the good drivers (developers) out from the beginning, nor will anybody, who took the exam successfully, be able to do his job right from the beginning, because the most stuff one needs is learned during the years of experience…

With that in mind, the question, how sophisticated or “stinking” an exam is or should be is more marginal. There remain only some rather crude criteria of wide scope: the exam should not be too easy, it should not be too difficult, it should cover as close as much the learning matter - that’s nearly all.

By the way, if the exams are to be improved, at first some questions should be examined if there are the right answers… - and when I say “right” I don’t mean “leading to a passed exam” but “right” in the sense of objectively right…

Actually no I think you have it backwards. The point here is that to get a university degree (at least at any proper institute), you need to do a lot more than just pass a test. you have to actively work on projects, you have to go out in the field and get work experience, you have to have your piers asses your abilities in the real world, and then all this is combined to give you a degree. But just passing one multiple choice test is not enough.

Currently in the NAV world, passing 2 multiple choice test with stolen exam questions, and you are a certified expert. That is the issue. A significant issue here, is that university exams are unique, so last years exam is of no use for cheating, only in so that it hints you in the direction you should be learning.

In fact one option might be to create a good Navision test, and allow people to sit it once a year, and each year write a new test.

[:$] Ahw shucks, and thanks.

David - I just read the post that Adam linked to. Great job!

Does anyone else have ideas of questions they might ask?

Some very good points very well said. In fact I was thinking of also using the Drivers test analogy, but in a different way.

I wanted to point out that a drivers test generally has three phases, The theoretical test, the practial test and the real world test. And what we are discussing is the Theoretical test.

But there is one major difference; in the Drivers test, there is an absolute number of rules, they are clearly defined, and you must know them all. The test is there to see if you really learnt those rules, because if you get into a car, you can not know just some of them, you just can not start the Practical test until you know them. So in the case where you read the book there is only one book. But in Navision there is not a clear book of laws, there are guide lines. When I am driving and I see a Red light I stop, that’s it, don’t think about it. I don’t think “hmm there are no cars coming the other way, so i am allowed to run the Red.” In NAV its never hard and fast, so the test can not be hard and fast, it needs to allow some inteligence an interpretation on behalf of the testee, and a multi choice question can do not do that. Also in a drivers test, the multi choice test only allows you to then sit for the practical test, but you are not licensed untill you pass the practical test. In NAV as soon as you pass the multi choice test you are certified and licensed to drive, even though red lights.

[Y] absolutely, and very well said. Its crazy that there are questions that indicate that ABCorD is the solution, and its clearly nonsense.

Thanks. Also feel free to go to the WIKI entry and edit or add to it.