Coöperatie Examens MBO was founded almost six years ago by and for Dutch vocational education (mbo), with one clear remit: to develop generic exams for the subjects that every mbo student takes. Today, the cooperative handles hundreds of thousands of exam sittings a year, spread across more than a hundred educational institutions and several hundred test locations. The entire examination process, from construction through to grading, runs on Remindo, the digital assessment and examination platform from Paragin Group. The cooperative also offers examination for the subject citizenship in Skillsly.
Erik Degen, director of Coöperatie Examens MBO, talks about building from scratch, working with Paragin Group, and the national standard that has grown out of it.
The challenge
The cooperative came into being when the mbo numeracy exam changed from a central exam into an institutional exam. Rather than have every institution reinvent the wheel, the schools decided to join forces.
“We were founded almost six years ago by the mbo sector, to develop generic exams for subjects that every mbo student takes, such as numeracy and citizenship. The reason was that the numeracy exam changed from a central exam into an institutional exam, and the numeracy requirements changed as well. That is when the institutions said: let’s set up our own cooperative and make our own exams. I have been director since the very beginning, so we really did start from zero.”
The cooperative serves the whole of mbo, from level 1 (entry level) up to and including level 4. Building a single exam for so many different institutions brings challenges, scalability among them. With hundreds of thousands of sittings at hundreds of locations, scalability was a genuine concern. In practice, it proved to be no bottleneck at all.
“In Remindo we don’t run into scalability problems. Sitting volumes are not an issue at all, and neither are the different test locations.”
The solution
The search for a partner ran through a tender, which Paragin Group won. For Erik, it was about more than software.
“We already knew Paragin, and during the tender Remindo came out on top. We were genuinely looking for a partnership. And right from the start we had a good rapport with Paragin. Twenty years ago, exam software was still relatively straightforward; back then you could translate everything into a PDF. Now it has become genuinely complex software, with a great many options and choices, and strong security.”
In Remindo, the core team sets up the exams in full, from construction through to formal approval. Where the cooperative initially worked with individual items, two years ago it moved to complete exams in the digital environment.
“We started out with item banking, developing assignments in Remindo and then drawing on them as needed. But we noticed that some of the items never got their turn. That is why we switched to test banking two years ago: you set up complete exams in the digital environment. Those exams are then also given approval advice by the National Numeracy Approval Committee (LVR).”
Twice a year, the cooperative delivers the exams to more than a hundred institutions, deliberately fully digital. Paper is out of the question.
“We have an outright ban on printing. Because what happens to a paper version? It ends up in a desk drawer and is used again six months later, exactly as it was, without having been revised. We get no data back, which is very important for quality improvement, and with paper sittings there is of course a risk of information leaking.”
Once students have sat their exams, those exams are graded. This is done by a large pool of assessors, connected to Remindo through the cooperative’s own assessor module.
“At the end of the examination process, every exam has to be graded. Via an API connection, the work to be graded goes to our own assessor module, which distributes it fairly across all 400 assessors. We developed that module ourselves, but it does talk to Remindo. On top of that, we analyse the exams together with CitoLab. That really gives quality a boost.”
Erik continues:
“We will shortly be starting to use AI as an aid in grading. And we genuinely deploy it as an aid: the assessors remain at the helm. Meaningful human intervention matters here, and every exam is assessed at least once by a trained assessor.”
For numeracy, everything runs in Remindo, both formative and summative. For citizenship, the cooperative also uses Skillsly, likewise from Paragin Group; that track was piloted in March and has been developed further for a new release on the basis of the feedback received.
Collaboration and implementation
The implementation was not a one-off project with a clear end point, but the beginning of an ongoing collaboration.
“We completed the implementation in a relatively short period. But because we keep evaluating and improving, it is really a continuous process. Feedback we receive from schools leads to technical improvements to Remindo, but of course we cannot do that without Paragin. It is a genuine partnership.”
Several teams from both the cooperative and Paragin are involved, and they move with the subjects the cooperative examines.
“Different teams are involved, both on our side and at Paragin. Initially we had only a numeracy team; now we have a citizenship team as well, along with a support department that plugs into everything. We have always found the collaboration a pleasure.”
The results
The biggest gain lies in standardisation, and it reaches well beyond the cooperative’s own organisation. What began as an internal wish to get data in order grew, through collaboration within mbo, into a way of working for the sector as a whole. Standardisation is, paradoxically, the gateway to flexibility.
“The impact of working with Remindo is very significant. It is not only about standardising the assessment process itself. We thought it would be smart to standardise student data, student results and planning data as well. We tackled that in a working group, with Paragin involved from the outset. Paragin and Remindo really took a leading role in that. That is impact across the whole of mbo, and it is actually beginning to move towards higher professional education as well.”
That standardisation puts a norm in place across the entire process, which makes deviations visible so they can be improved in a targeted way.
“Because we have standardised the grading, for example, we can also see which assessments deviate from it, and we apply improvements there. You can see that both construction and delivery are becoming steadily more efficient and more consistent. That is useful for us as an exam provider, and for the schools too: they can organise their planning and delivery processes better.”
And because everything is digital and data-driven, exams can be improved intelligently.
“Based on the data you get out of the exam analysis, you carry the improvements through. That is what makes it so interesting.”
Looking ahead
The cooperative is continuing along the path it has taken, digital examination of knowledge and skills, together with Paragin Group. The driver is not that things are not good enough today, but the conviction that they can always be better.
“We improve because things can always be better, not because they are not good now. Because we give each other feedback, we are continuously improving the process. We are simply carrying on down the road we have taken.”
Finally, Erik sums up in a single word what has carried that collaboration all these years.
“Involvement is a good word. The partnership is essentially about thinking together about the way forward. And that is very pleasant indeed.”