Problems with certifications

I've developed a distrust for certifications. They exist for a purpose, to solve a certain set of problems, to fill a certain gaps. But, often, they don't mean a lot in a practical sense. Often, certifications are not a trustworthy measure of skill, and chasing certifications can come at the expense of chasing skills and real impact in people's lives.

They aren't all bad - they exist to meet some very important needs. But there is a lot worth rethinking.

Also, of course not all certifications are the same. I don't want to paint everything with the same brush. This article is about problems, I have also seen people do very good work.But that is not what this article is about.

Let me give you a few examples of how certifications have gone wrong. These all things I have personally experienced. These stories are not unique:

Cramming for tests

I used to work for a Google Cloud partner in South Africa. It was a very certification-heavy organisation. One of my friends and colleagues was interested in moving up the ranks and so he decided to get a Google Cloud Associate certificate.

He proceeded to do the bare minimum in order to pass - He spent a weekend cramming for the exam. He passed on the first try.

He didn't pass because he was already a Google cloud guy. He was an Angular and UI/UX guy. He hardly used Google cloud products. He had experience in hosting PHP applications on VMs.

Later, he admitted that he didn't remember much of what went into the certification, he just had some words on his CV. Cramming is not a great way to retain information in the long term.

I'm not saying that these certifications signify nothing. They do indicate a certain level of aptitude, grit, technical prowess at the very least. My friend was generally a smart and hard working guy and could probably figure out what he needed to do if he needed to work on something related to the certificate. But that was all true before he started cramming.

And if a person studied for such an exam and really put the time and practice in to master the material then the certification can be a useful guide. Some people do take it seriously, dive in deep, and put their learning into practice.

Some people pass easily because they already have the skills. This is also good to know.

So it doesn't signify nothing. But this signal is very noisy.

Cousera certifications

A lot of organisations are putting courses on Cousera and giving people certificates when they complete things. I'm all for making learning material accessible in different ways, I'm convinced that Cousera and similar have done a lot of good things for people, but I have seen some very funny things here as well.

Here is one of my favourite examples: Google Cloud Data Analytics Professional Certificate. Sounds kinda hardcore, doesn't it?

There is a lot you can learn by pursuing this certificate. I know many people who have benefited. There is good information in there. However, the system is very very easy to cheat.

The final course required to get your certificate is based around a capstone project. At face value, that seems like a great idea. Projects are great.

Here is a screenshot of the final assessment based on that capstone project:

Now, if there was more to the assessment perhaps it would be worthwhile, but this is where it sits in the course:

So once you tick the box, you get congratulated.

There are some other assignments and exercises before this, point. But for a capstone project, it's pretty weak.

Quantity versus quality

I was working with an organisation that was receiving quite a lot of funding from international do-gooders such as UNICEF. When working on that level, everything becomes a numbers game. They wanted to touch as many lives as possible and get as many people certified as possible.

The Google Cloud Analytics course above, and others like it, played a big part in that organisation. Lots and lots of people got certified.

That same training organisation ran their own 9 month long web dev course that made big changes in people's lives very reliably. It took noobs (not a derogatory term, I am a noob at a great many things) and turned them into professionals. Practically everyone who finished the course got a good job at the end, and kept that job - it wasn't interview hacking, they had real skills - both hard and "soft".

One life touched - one life changed. It was powerful and inspiring. It was a place to innovate around how education could really work for people.

Of course this is an expensive thing to do. So the scale was too small to get the attention of big funders. One could hope that large funders would provide enough that there would be some money left over for supporting such provably effective programs. Perhaps it would be worth finding ways to take the innovations of such programs and share them so that other training providers could benefit. Perhaps that would be worth attempting, or even considering.

Alas, no. The high touch program just didn't touch as many lives or generate as many certificates, so it was not something that was particularly well-resourced. It was not especially core to the organisation's strategy because the real money came in elsewhere.

Collecting certificates

I was recently talking to a large software dev consultancy about their grad program. I love grad programs so I seek them out. My goal there is to help where I can - make existing grad programs better, and help organisations set up grad programs if they don't yet exist.

The consultancy already had a well-established grad program and they were talking me through their processes. The aggressively over-simplified version is that they find grads they want to invest in and get them started with some training. Once the grads get to a certain point then they are expected to pursue a few different certifications, largely to do with AWS products. After that they do more training.

When they told me about the emphasis on certifications, I had a lot of questions.

Turns out that their general procedure for certifications is:

1. Get the certifications
2. Train the grads properly so they actually have the skills that the certifications are meant to validate

They do not, internally, see their AWS certifications as a valid measure of AWS skill. They do it because it helps with marketing. Their clients like to see lots of certificates, so they collect lots of certificates.

Certify then teach

This approach of hitting certifications and then focusing on learning afterwards is not unique. It's something I've done as well.

I was working for a training organisation that offered nationally recognised certification in South Africa. It was "NQF level 5" certified. This is equivalent to the second year of a Bachelor’s Degree programme or a HND diploma.

It was silly. Some of it meant well - the assessments did at least ask questions about unit testing (this section basically needed a person to be capable of something like the helloworld of testing. A very low bar) and version control (this didn't need to be Git. Google docs has version control too).

It also expected all the data science students to answer a bunch of questions about VOIP... Remember VOIP?

It would have been nice to toss the certification out, but we needed it.

We needed it because it meant that organisations who paid us for training could get some tax breaks for funding "certified" training.

It was also useful for marketing to would-be students - people are more likely to pursue accredited courses than unaccredited courses. That was the perception anyway. Currently, I don't think that is correct.

So we got people certified.

We did the certification first to get it out of the way, and because there is a lot of administrative work involved in getting a certification finalised - it takes time. So we diligently ticked the boxes.

And then we trained people properly.

This is obviously wasteful. It's more wasteful if you consider the process of becoming a certified organisation - the organisation itself needed to jump through some serious hoops to maintain the right to offer certificated courses.

For example, even though the course was completely online and remote, the certifying body needed to do a site visit as a part of their due diligence process even though they accepted that we did remote training. So we needed to show them that we had a well-ventilated area with good lighting and flushing toilets because students need that kind of thing. This meant renting out an office space for a day.

There was a half-time staff member whose entire job was to handle the administrative burden of jumping through certification hoops; and staying up to date with any changes in those hoops. And it was also necessary to pay external assessors to check that the datascience students had grasped important concepts like VOIP.

Currently, the South African government is revamping these certifications in the hope of making them more relevant, useful and meaningful. I've spent a lot of time digging through the new certification requirements. There are some improvements, and many parts that make me laugh one of those "is this the world we live in?" kinds of laughs.

This is the water we swim in. I kinda want to stage a revolution. Who is with me?

Problem summary

The stories above touch on a few problems in certification-land. Here's a little summary.

Many certifications put too much emphasis on numbers games - piping people through a course, aiming to fill pails rather than ignite. Putting people on a production line, getting them to go through the motions, giving them a stamp of approval and a have a nice day.

Many certifications rely on assessments that are easily cheated and hacked.

Many certifications are out of date or badly designed. They don't stay hip with the times by default. And the South African government, at least, needs a better plan.

Many certifications (probably most of them) rely on assessment mechanisms that are not very much like real world application of the skills. A quiz does not resemble a cloud deployment.

Certifications are big business. And sometimes big businesses optimise for things that are not in the best interest of their clients, or the ecosystem at large.

Certifications are not all bad though

There are real problems that certifications aim to solve. So simply doing away with them wont work.

Certifications are an organising force.

Certifications aim to validate specific things - skills, the ability to make use of and reason about different technologies, knowledge and applied knowledge...

If you know that a Google Cloud Practitioner certificate outlines skills needed for "Cloud Engineers", then people who want to be "Cloud Engineers", or train "Cloud Engineers", can have an idea of what to cover. Certifications can guide and focus education efforts.

And even beyond technical certifications, they are hella handy. There is a lot I and many others say to criticise traditional schooling systems, especially anything to do with standardised assessments (yuck!). But look at how many people can read! Obviously, something is going right.

(If you enjoy podcasts, here is a good one about standardised tests)

And of course, certifications add credibility to professional skills. This is true for tech and beyond. I wouldn't ever want to rely on a surgeon who doesn't have certified skills.

Organisations that hire and work with technical professionals need a way to figure out who to work with. It's expensive to make bad decisions, and it's hard to make good ones. Certifications help there.

They provide a signal that opens doors. But it's a noisy signal that keeps doors closed for many.

Can we do better?

In summary, certifications fulfil 2 main roles. They are:

1. A guide or a pathway - a way to decide what expertise to pursue to reach different goals
2. Credibility indicators - a signal to show who has skills, a way for people to derisk decisions about who to work with

They do a lot of good. But they also contribute to a pretty dysfunctional education ecosystem.

What if we were to think about certifications and their roles from first principles? Is there a way to build something that meets the needs that these certifications hit? Can we certify people in a way that is, efficient, effective and trustworthy?

Yes. I think so. But it might mean doing things a bit differently.

Stay tuned

Wow, this turned into a long article! I wasn't expecting to write so much when I started.

There is a lot more to say, I've seen some people doing some very interesting work in certification-land. And I have a few things I'm working towards as well.

Until next time...