6.3 Checker

Keywords for configuring check::

• :users—specification of users that are acceptable for submission. Can be either a list of user lists, each representing a known team, or procedure which will accept a list of users and throw an exception if they are unacceptable. The default is to accept only single-user submissions. The pairs-or-singles-with-warning procedure is a useful value for pair submission where the pairs are unknown.

• :eval?—whether submissions should be evaluated. Defaults to #t. Note that if it is specified as #f, then the checker body will not be able to run any tests on the code, unless it contains code that performs some evaluation (e.g., using the facilities of handin-server/utils).

• :language—the language that is used for evaluating submissions. If the value is of the form (list 'module spec), then make-module-evaluator is used with spec as its #:language argument to generate an evaluator. Otherwise, the value is passed as the language argument to make-evaluator. There is no default for this, so it must be set or an error is raised. (See call-with-evaluator/submission for further details.)

• :requires—paths for additional libraries to require for evaluating the submission, same as the requires argument for make-evaluator (see handin-server/sandbox). This defaults to null—no teachpacks. Note: if a module language is used (See call-with-evaluator/submission for further details), it is passed as the allow-read argument.

• :teachpacks—an alternative name for :requires, kept for legacy checkers.

• :create-text?—if true, then a textual version of the submission is saved as "text.rkt" in a "grading" subdirectory (or any suffix that is specified by :output below, for example "hw.java" is converted into a textual "grading/text.java"). This is intended for printouts and grading, and is in a subdirectory so students will not see it on the status web server. Defaults to #t.

• :textualize?—if true, then all submissions are converted to text, trying to convert objects like images and comment boxes to some form of text. Defaults to #f, meaning that an exception is raised for submissions that are not all text.
  (Effective only when saving a textual version of the submission files: when :create-text? is on.)
  This flag is effective only when saving a textual version of the submission files — when :create-text? is on. The possible configurations are:

  • :create-text? is on and :textualize? is off (the default) — in this case a text version of submissions is created, and submissions must contain only plain text. The text file has the same semantics of the submission and can be used to run student code.

  • :create-text? is off — allowing submissions that contain non-textual objects, but no text file is created so grading and testing must be done using DrRacket (because the saved submission is always in binary format).

  • Both flags are on — allowing submission with non-textual objects and generating text files, but these files will not be usable as code since objects like images cannot be represented in plain text.

• :untabify?—if true, then tabs are converted to spaces, assuming a standard tab width of 8 places. This is needed for a correct computation of line lengths, but note that DrRacket does not insert tabs in Scheme mode. Defaults to #t.
  (Effective only when saving a textual version of the submission files: when :create-text? is on.)

• :maxwidth—a number that specifies maximum line lengths for submissions (a helpful feature for reading student code). Defaults to 79. This feature can be disabled if set to #f.
  (Effective only when saving a textual version of the submission files: when :create-text? is on.)

• :output—the name of the original handin file (unrelated to the text-converted files). Defaults to "hw.rkt". (The suffix changes the defaults of :markup-prefix and :prefix-re.) Can be #f for removing the original file after processing. The file is always stored in GRacket’s binary format.

• :multi-file—by default, this is set to #f, which means that only DrRacket is used to send submissions as usual. See Multiple-File Submissions for setting up multi-file submissions.

• :names-checker—used for multi-file submissions; see Multiple-File Submissions for details.

• :markup-prefix—used as the prefix for :student-lines and :extra-lines below. The default is ";;> " or "//> ", depending on the suffix of :output above. Note: if you change this, make sure to change :prefix-re too.
  (Effective only when saving a textual version of the submission files: when :create-text? is on.)

• :prefix-re—used to identify lines with markup (";>" or "//>" etc), so students cannot fool the system by writing marked-up code. The default is ";>" or "//>", depending on the suffix of :output above.
  (Effective only when saving a textual version of the submission files: when :create-text? is on.)

• :student-line—when a submission is converted to text, it begins with lines describing the students that have submitted it; this is used to specify the format of these lines. It is a string with holes that user-substs fills out. The default is "Student: {username} ({Full Name} <{Email}>)", which requires "Full Name" and "Email" entries in the server’s extra-fields configuration. These lines are prefixed with ";;> " or the prefix specified by :makup-prefix above.
  (Effective only when saving a textual version of the submission files: when :create-text? is on.)

• :extra-lines—a list of lines to add after the student lines, all with a ";;> " or :markup-prefix too. Defaults to a single line: "Maximum points for this assignment: <+100>". (Can use "{submission}" for the submission directory.) See also add-header-line!.
  (Effective only when saving a textual version of the submission files: when :create-text? is on.)

• :user-error-message—a string that is used to report an error that occurred during evaluation of the submitted code (not during additional tests). It can be a plain string which will be used as the error message, or a string with single a "~a" (or "~s", "~v", "~e", or "~.a" etc) that will be used as a format string with the actual error message. The default is "Error in your code --\n~a". Useful examples of these messages:

  "There is an error in your program, hit \"Run\" to debug"

  "There is an error in your program:\n----\n~a\n----\nHit \"Run\" and debug your code."

  Alternatively, the value can be a procedure that will be invoked with the error message. The procedure can do anything it wants, and if it does not raise an exception, then the checker will proceed as usual. For example:

  (lambda (msg)

  (add-header-line! "Erroneous submission!")

  (add-header-line! (format "  --> ~a" msg))

  (message (string-append

  "You have an error in your program -- please hit"

  " \"Run\" and debug your code.\n"

  "Email the course staff if you think your code is"

  " fine.\n"

  "(The submission has been saved but marked as"

  " erroneous.)")

  '(ok))

  (message "Handin saved as erroneous." 'final))

  (Note that if you do this, then additional tests should be adjusted to not raise an exception too.)

• :value-printer—if specified, this will be used for current-value-printer.

• :coverage?—collect coverage information when evaluating the submission. This will cause an error if some input is not covered. This check happens after checker tests are run, but the information is collected and stored before, so checker tests do not change the result. Also, you can use the !all-covered procedure in the checker before other tests, if you want that feedback earlier.

Within the body of check:, users and submission will be bound to the checker arguments—a (sorted) list of usernames and the submission as a byte string. In addition to the functionality below, you can use (!eval expr) (or ((submission-eval) 'expr)) to evaluate expressions in the submitted code context, and you can use (with-submission-bindings (id ...) body ...) to evaluate the body when id’s are bound to their values from the submission code.}

(require net/sendmail)

(post:

(define info

(format "hw.rkt: ~a ~a"

(file-size "hw.rkt")

(file-or-directory-modify-seconds "hw.rkt")))

(timeout-control 300)

(log-line "Sending a receipt: ~a" info)

(send-mail-message

"course-staff@university.edu"

"Submission Receipt"

(map (lambda (user) (user-substs user "{Full Name} <{Email}>"))

users)

(list (user-substs (car users) "{TA Name} <{TA Email}>"))

null

`("Your submission was received" ,info))

(message (string-append

"Your submission was successfully saved."

"  You will get an email receipt within 30 minutes;"

" if not, please contact the course staff.")

'(ok)))

This is used to process the :student-line value in the checker, but it is provided for additional uses. See the above sample code for post: for using this procedure.

For the latter two !test forms, note that the result and equal? forms are not evaluated in the submission context.

(!all-covered

(lambda (where)

(case (message (string-append

"Incomplete coverage at "where", do you want"

" to save this submission with 10% penalty?")

'(yes-no))

[(yes) (add-header-line! "No full coverage <*90%>")

(message "Handin saved with penalty.")]

[else (error "aborting submission")])))