Terminology
When I was introduced to the grading/marking process, I was told about a difference between grades and marks:
Grades: Are the result of completing a module/course;
Marks: Are the result of completing a summative assessment within a module;
Within Moodle grades are marks and marks are grades, within this musing I will not be using them interchangeably.
Scales
We give value to outcomes by mapping results to a ‘scale’. Although these scales are percentage points in the range $[0,100]$, we often use a more limited scale that maps to discrete percentage ponits.
Within Strathclyde we have two such scales:
Type A: A plain percentage scale for quantitative assessments.
Type B: A qualitative scale that uses 21-points of the Type A scale.
The Type B scale uses letters to represent these points. We also have the idea of descriptive bands (‘you can think letter grade’) in which we band percentage points together. For both Type A and Type B, we have bands within bands depending on various factors such as UG/PG and how many points should be in a band,.
The combined Type A/B for both UG and PG at Strathclyde are presented below:
| Class | Descriptor | Type A | Type B | |||
|---|---|---|---|---|---|---|
| UG | PG | High | Medium | Low | ||
| First | Distinction | Exceptional | 100–90 | 100(A1) | 92(A2) | |
| Outstanding | 89–80 | 84(A3) | ||||
| Excellent | 79–70 | 78(A4) | 75(A5) | 72(A6) | ||
| 2:1 | Merit | Comprehensive | 69-60 | 68(B1) | 65(B2) | 62(B3) |
| 2:2 | Pass | Satisfactory | 59-50 | 58(C1) | 65(C2) | 62(C3) |
| Third | Fail | Unsatisfactory | 49-40 | 48(D1) | 45(D2) | 42(D3) |
| Fail | Inadequate | 39-30 | 38(E1) | 35(E2) | 32(E3) | |
| Weak | 29-20 | 20(F) | ||||
| Minimal | 19-1 | 10(G) | ||||
| None | 0 | 0(H) |
where each descriptor leads the sentence:
<descriptor>demonstration of the learning outcomes
Observe the following:
Both scales represent the absence of a submission with the 0 point.
The failure point is different: PG is below 50%, and UG is below 40%.
The scales are not exactly linear, enabling a somewhat maximisation of points for grading positive things and minimising the points we use for failure.
The bands in Type A are, for the large part, 9 percentage points. The minimal and exceptional bands, however, are 19 and 10.
The Bands in Type B are mainly clustered into three points per band but when you do beyond fail they slowly drop to one.
As an aside, it would be great if the scale was designed to be logarithmic, but alas not…
A hidden 10-point quantitative scale can be found if we just use the descriptors! I shall call this one ‘Type C’.
By-Design Moodle cannot represent our Qualitative Scales
By default, Moodle uses a percentage points scale for representing grades and marks. One can even define a ’letter grade’ mapping in which you can map descriptors to percentage ranges… However, Moodle enables one to present custom scales that map ’things’ to percentage points.
The issue with Moodle is that custom scales are not generic. The mapping of points to percentages is performed by dividing the position of the scale with the number of points, and whether one is using the scale is aggregate or normalised. That is: the mapping is dependent on how values from the scale are used for final calculations. Specifically, a normalised mapping will index the scale by 0, and aggregate will index from 1.
For instance, consider the scale: A,B,C,D.
Normalised Mapping
| A | 3 | 3/3 | 100 | | B | 2 | 2/3 | 66 | | C | 1 | 1/3 | 33 | | D | 0 | 0/3 | 0 |
Aggregate Mapping
| A | 4 | 4/4 | 100 | | B | 3 | 3/4 | 75 | | C | 2 | 2/4 | 50 | | D | 1 | 1/4 | 25 |
Comparing how Moodle represents scales with our Type A/B/C scales, you can see that by-design we cannot use Moodle scales to represent them,
If only Moodle would enable us to provide a precise mapping between scales and percentage points! Alas we should use percentage points in Moodle for reporting grades and marks.
Moodle Grade Books
Not going to touch that with a barge poll.
Advanced Grading
Not talking about the use of MCQs and other electronic assessment methods available within Moodle, we will look at the grading of assessments that students submit to Moodle. The former should be reported as percentage points where possible, the latter is a bit more complicated.
By-default Moodle will enable ‘simple’ grading of assessments, where the final marks awarded to assessment are reported. Luckily, Moodle also enables one to download a grading sheet through which you can bulk upload grades an dinline feedback. Annotated submissions and feedback files can also be bulk uploaded.
My interest is with capturing how the marks themselves are calculated. By default Moodle does not capture mark calculation. To use Moodle, one needs to use ‘advanced grading’, in which weighted criteria can be designed and marked.
There are two current ‘advanced grading’ solutions:
Marking Guide
Where the assessment designer can describe a set of (quantitative) criteria that captures: a description of the criterion; its maximum value; and a marking guide for markers.
The Moodle integration means that students will see, up front, the set of criteria and their marks. When marking the assessment the markers are given a simple form with the set criteria to provide a summative score and formative feedback. The marking guide will auto tally the marks!
The marking guide can be shared too.
What is attractive about marking guides is their ability to:
- capture quantitative and qualitative assessments;
- automate the tallying;
That is, for quantitative assessments you can offer students an assessment full of exercises and use the marking guide to nicely report the results.
Unfortunately, there are two issues with marking guides. For one, we cannot set weights as percentages nor use an existing scale. We must use a quantitative scale of marks (which could be assigned qualitatively) and Moodle will calculate the percentage from dividing the totals by the sum of the maximum marks. This means that the design has to ensure that the set of marks for each criterion are weighted appropriately.
For example, consider the following criteria for a technical report.
| Criteria | Weight |
|---|---|
| Knowledge | 40 |
| Reasoning/Argument | 40 |
| Relevancy | 10 |
| Literacy | 10 |
Then the marking guide has to translate the weights into percentages. The easiest route is a one to one mapping, but then markers will have to decided how to allocated marks appropriately.
With the marking guide I cannot use Type A/B scales from strathclyde directly, I have to side step with external decisions… Furthermore, I cannot use advanced calculations such as, discount the worst two scores or use a weighted mean ex cetera.
A better solution would be to have the marking guide use a given scale for each criterion, and then use a weighted sum to calculate the final score. Funnily enough the Moodle grade book enables this to a certain degree… In fact I use the grade book to report the individual scores for each exam question and have an ‘advanced’ calculation to tally them up… If only we could ‘Yo Dawg’ the functionality from grade books into the marking guide….
The second issue! One cannot bulk upload to moodle! You have to be online to do so.
Marking Rubrics
Where the assessment designer can describe a set of qualitative criteria that captures: a description of the criterion; a custom point scale and a marking guide for markers.
Rubrics differ from guides in that each qualitative criterion will be based on a fixed number of points. As a marker you are offered a table from which to select the criterion’s value, the rest of the interface and UX follows marking guides.
The marking rubric can be shared.
Unfortunately, there are two issues with ‘marking guides’.
By design each criteria must have the same number of points, but the points values can differ. You cannot set weights as percentages nor use an existing scale to map the ‘points’.
For example, consider the following criteria for a technical report.
| Criteria | Weight |
|---|---|
| Knowledge | 25 |
| Reasoning/Argument | 25 |
| Relevancy | 25 |
| Literacy | 25 |
Assuming we have a 6 point rubric (tables must have the same number of columns) then the criteria can be encoded as:
| Criteria | Points |
|---|---|
| Knowledge | 0,1,2,3,4,5 |
| Reasoning/Argument | 0,1,2,3,4,5 |
| Relevancy | 0,1,2,3,4,5 |
| Literacy | 0,1,2,3,4,5 |
where we have assigned the values $0–5$ to each point.
Notice that each criterion’s weighting is the same. If we were to change the weights then we will have to recalculate the point’s values so that their values take into account the desired weightings.
| Criteria | Weight |
|---|---|
| Knowledge | 40 |
| Reasoning/Argument | 40 |
| Relevancy | 10 |
| Literacy | 10 |
The easiest route is to keep the weightings for each criteria the same. That is not consumerate with good assessment design, some criteria are worth more than others…
Like the marking guide, I cannot use Type A/B scales from strathclyde directly. Instead I would have to use the ‘Type C’ scale for a less precise measure of attainment.
The second issue, like that of marking guide, is that one cannot bulk upload to moodle! You have to be online to do so.
If Marking Guides enabled use of qualitative scales, then Marking Rubrics would not necessarily be needed.
Coda
If designed correctly, Moodle could easily incorporate many of my teaching workflows for weighted criteria using qualitative scales. It does not.
If only it could, and if only one could bulk upload with advanced grading…