Running Check50

Martijn Stegeman, University of Amsterdam

Step 1. Getting Started

If you have a working Python installation, please run pip3 to install check50:

pip3 install check50

You should now be able to run the tool:

$ check50
usage: check50 [-h] [-d] [--offline] [-l] [-o {ansi,json,html} [{ansi,json,html} ...]]
               [--target TARGET [TARGET ...]] [--output-file FILE]
               [--log-level {debug,info,warning,error}] [--ansi-log] [--no-download-checks]
               [--no-install-dependencies] [-V] [--logout]
               slug
check50: error: the following arguments are required: slug

The reason that check50’s complaining is that it needs a slug, a path to the checks that have to be run. These checks are usually hosted in a GitHub repository, which makes them easy to find. Let us now choose an assignment for which we have an existing check.

Step 2. The Assignment

The assignment we’re going to be working on is called “Hello”. It’s supposed to be a Python program that asks for a name, and then outputs “Hello, name!”. Looks like this when run at a command prompt:

$ python3 hello.py
Name: Martijn
Hello, Martijn!

For now, we’ll try to run the check without creating that program. As it happens, the slug for this program is cs50/problems/2020/fall/sentimental/hello. From that slug, you can infer that it lives in this location:

• On github.com
• In the organization cs50 (you can use your personal organization!)
• In the repository problems
• In the branch 2020
• In the folder fall/sentimental/hello

Let’s run it by typing the slug and adding the flag -l (lowercase letter). This ensures that the check is run on your own computer instead of Harvard’s cloud infrastructure.

This also means that check50 is only critically dependent on two things: the pip infrastructure and the availability of a GitHub repository with checks. So anyone can use it at any time.

$ check50 cs50/problems/2020/fall/sentimental/hello -l
Connecting......
You seem to be missing these required files:
hello.py
You are currently in: ~/dev, did you perhaps intend another directory?

Again, check50 is complaining, this time because a file called hello.py is expected to be present. Check50 always runs checks on the files that are in the current directory! So if you, or your students, want to check a program, they have to cd into that directory.

But that’s OK! Lets’ create an empty file called hello.py. Let’s see what check50 says now:

$ touch hello.py
$ check50 cs50/problems/2020/fall/sentimental/hello -l
Checking........
Results for cs50/problems/2020/fall/sentimental/hello generated by check50 v3.2.0
:) hello.py exists.
:( responds to name Emma.
    expected prompt for input, found none
:( responds to name Rodrigo.
    expected prompt for input, found none

Finally it works! Of course, your empty solution in hello.py shouldn’t pass all checks. And indeed, your empty program correctly fails two of those. Only the “hello.py exists” check passes now.

Here are the three potential results that you may encounter for each check:

• :) is reported for any successful check
• :( is reported when a check is run but fails
• :| is reported when a check is not run, always because an earlier check failed

Step 3. What checks look like

Have a look at this example check for hello.py:

import check50

@check50.check()
def exists():
    """hello.py exists."""
    check50.exists("hello.py")

@check50.check(exists)
def veronica():
    """responds to name Emma."""
    check50.run("python3 hello.py").stdin("Emma").stdout("Emma").exit()

@check50.check(exists)
def brian():
    """responds to name Rodrigo."""
    check50.run("python3 hello.py").stdin("Rodrigo").stdout("Rodrigo").exit()

As you can see, there are three checks:

• exists(), which does nothing more than verify the presence of the correct source files used in the remainder of the checks

• veronica(), which checks if the program works correctly when fed the name “Emma”

• brian(), which checks if the program works correctly when fed the name “Rodrigo”

Never mind those function names!

From the example you may notice a few things:

• There are dependencies between checks. The second and third check specify exists as a precondition.

• The """docstrings""" atop each function are used as check descriptions when running check50.

• The check specifies a kind of “chain of events”, where expected input and output are specified, as well as the expectation of a clean exit.

Incidentally, the third check is fundamentally the same as the second one. It is a simple measure to ensure that the program isn’t hardcoded towards the name “Emma”. Of course, student code could be hardcoded in the following way and still pass the check:

name = input("Name: ")
if name == "Emma":
    print("Hello, Emma!")
else:
    print("Hello, Rodrigo!")

So you may want to write better checks to encourage students to write better code :-) But that’s something that can develop over the years!

Step 4. Creating your own checks repo

To get started, create a GitHub repository and add a subfolder for the check that you would like to write. Then you need two files at a minimum:

• A file called __init__.py that contains the actual check code, like you’ve seen in step 3.

• And a file called .cs50.yml that contains some of the basic configuration for your check. Most importantly, it specifies which files (from the current user directory) to include in the check and which to ignore. You can also use it to include distribution modules or other features specific to your course.

  Here is an example that shows the basics:

    submit50:
      files: &submit50_files
        - !exclude "*"
        - !include "*.py"
        - !require hello.py
  
    check50:
      files: *submit50_files
  

  Note that only Python .py files are taken into account for the check, and besides, the file hello.py is required. You might imagine a very eager student who has completely modularized their code into multiple Python files. Using !include to allow *.py ensures that these external modules are available when checking.

Step 5. Creating checks for Jupyter Notebooks