17 February, 2014Processes

Why I refuse to use Codility and so should You

  • Tymoteusz Paul
Tymoteusz Paul
Bristol, UK

Let’s start with explaining what Codility is, for those who may be unfamiliar with the concept. It is a website where recruiters can invite a person to solve a programming challenge. Once you follow the invitation, you will be presented with a notepad, a task, and some fields where you can provide test cases to your solution. Everything you type into the cute notepad is recorded on the timeline and can, later on, be reviewed by a recruiter, together with a score – a measure of how well you did on your test. This is where the first issue arises – the notepad. While there still are some whackos out there who use text editors for heavy-duty programming (not just quick script/fix), but they are a minority and a dying breed. The standard today (cheap standard, as you can get modern and robust IDEs for as little as 0$) is to use IDE, which provides you with proper code completion, online code validation, and API suggestions. With Codility notepad, you get none of that as all you see is basic code coloring: no completion, no API suggestions, no PEP-8 validation. I frankly cannot even get the print right (especially since I moved to python 3), and my IDE is notoriously correcting me. Does that make me a bad programmer? Maybe, but I have a 45$ tool called PyCharm (I got it cheap) that completely resolves the problem with remembering the order of arguments for each function. I don’t know a single programmer who doesn’t ridicule the notion of writing non-pseudocode code on paper; this is no different. Of course, I could ignore it, fire up my visual studio and just develop the proper solution, but then if a recruiter peeks into the solution, all he will see is one big copy/paste (and most likely will conclude that I am a cheat).

But that brings me right to the next point – all of their questions are old, well known, and slow in adding new ones. And when they finally do – Codility is infested with people who apparently pay their subscription fee just to do the solutions and post them all around the web. Seriously, you can find answers to the new questions just a day or two after they were added. And they usually are good, better than what most coders can come up within 30 minutes, especially without the proper environment.

But let’s just say that you are an honest person and enjoy the tournament of writing in a notepad. This brings us the right to the tasks, the bread, and the butter. So let’s have a quick look at a sample task that is available at hand:

The prime is a positive integer X that has exactly two distinct divisors: 1 and X. The first few prime integers are 2, 3, 5, 7, 11 and 13. The semiprime is a natural number that is the product of two (not necessarily distinct) prime numbers. The first few semiprimes are 4, 6, 9, 10, 14, 15, 21, 22, 25, 26.

You are given two non-empty zero-indexed arrays P and Q, each consisting of M integers. These arrays represent queries about the number of semiprimes within specified ranges.

Query K requires you to find the number of semiprimes within the range (P[K], Q[K]), where 1 ≤ P[K] ≤ Q[K] ≤ N.

For example, consider an integer N = 26 and arrays P, Q such that:

P[0] = 1 Q[0] = 26 P[1] = 4 Q[1] = 10 P[2] = 16 Q[2] = 20

The number of semiprimes within each of these ranges is as follows:

(1, 26) is 10, (4, 10) is 4, (16, 20) is 0. struct Results {

int * A;

int M;


Write a function:

struct Results solution(int N, int P[], int Q[], int M);

that, given an integer N and two non-empty zero-indexed arrays P and Q consisting of M integers, returns an array consisting of M elements specifying the consecutive answers to all the queries.

For example, given an integer N = 26 and arrays P, Q such that:

P[0] = 1 Q[0] = 26 P[1] = 4 Q[1] = 10 P[2] = 16 Q[2] = 20

the function should return the values [10, 4, 0], as explained above.

Assume that:

N is an integer within the range [1..50,000]; M is an integer within the range [1..30,000]; each element of array P is an integer within the range [1..N]; each element of array Q is an integer within the range [1..N]; P[i] ≤ Q[i]. Complexity:

expected worst-case time complexity is O(N*log(log(N))+M); expected worst-case space complexity is O(N+M), beyond input storage (not counting the storage required for input arguments). Elements of input arrays can be modified.

I must say that in my long career in IT, which included AI development and neural networks, I never had a task like that in trying to beat my PC at rock-paper-scissors. And I do have the almost complete history of all jobs I have ever resolved (I am an e-hoarder). Most of my tasks begin with "fix..," "figure out….," "rework/rewrite for performance….," not with "for collection X process it to Y without loops." That is another issue I have with Codility: it doesn't test your knowledge of a specific language, or even its libraries but their understanding of mathematics and algorithms. While most people like to think that programming is mostly about creating algorithms, it isn't spending hours in debates over one line of code. It's a world where you sit down, write some code, run it through tests, and pray that it works. It always depends on your knowledge of the language, very often of understanding of available libraries, and very rarely on your understanding of mathematics. Quite frankly, most programmers will never deal with a mathematical issue in their careers, and still, they will be delivering excellent code that works and brings money.
On top of that, not only is your time limited and short, by default, you get 30 minutes to solve a challenge; you are also valued on the performance of your task. And you don't need just OK performance; your script is run against solutions that teams of developers optimized over a long period, which also included peer review. If you think that even an exceptional programmer knows all performance chinks for every part of the language, you are misguided and just plain wrong. It is even more valid in the modern programming world. A world in which scaling up and sideways is not only as easy as moving a slider from one side to another but is also much cheaper than putting an extra day or two to squeeze every bit of performance from a routine.

But let's keep going, you feel like sharpening your math-foo, so you work very hard on the solution in their little notepad, you test it against all the cases you can imagine, and it passes, so you submit your work. What happens next? Nothing! As a candidate, you do not receive any feedback, you will never know if/what was wrong with your solution, not even something as simple as "your score is 70%," not to mention details of what use case You've failed. This is an awful practice that does not create a better programmer; it only breaks people down when after beating a recruiter into a pulp, they hear "your score was so low that we will reject you." What would make codility half-useful, at least in the self-improving aspect, would be to present a candidate with his results, list of failed tests, and some articles explaining why your solution may have failed those.

But I guess that if they started doing that, the tests would be compromised even sooner than they are now and in a much wider variety of languages. The effect of that would be that they would need to come up with new challenges all the time, which just may break their model of pair-bottomfeeding with so-called "recruitment agencies." And while I do understand that some jobs do require that level of scrutiny that codility expects from candidates, I also know from experience that the high-ticket companies would never entrust such easy-to-cheat tool with any degree of screening.

Related Posts