TFA! Evidence-based Results

Evidence-based Results


This project explored students' perceptions of the potential of technology-enabled feedback to improve their learning. It aimed to evaluate how a range of technical interventions might encourage students to engage with feedback and formulate actions to improve future learning.


Online publication of grades and feedback


Sheffield Hallam University promotes the use of the Blackboard Grade Centre as the primary tool for the publication of grades and feedback to students in each of their modules. The project looked at what students' value most about having their grades and feedback published online and the extent to which this approach encourages them to engage with and use their feedback.


Online publication of grades and feedback through the Grade Centre enables students to access their grades and feedback at a time and place of their choosing. In common with the use of technology to support learning more generally (Parkin & Thorpe, 2009), the students appreciated the flexibility and convenience that this offers, providing support for much of the existing literature in the area (Bloxham & Boyd, 2007; Denton, 2001a, 2001b, 2003; Denton et al, 2008; Gipps, 2005; Price & Petre, 1997; Race, no date). The study found that the online publication of grades and feedback can offer students the flexibility to receive and read their feedback in private surroundings. This provides support for earlier research, for example Price & O'Donovan (2008) suggest that receiving feedback in privacy enables students to engage with and respond to their feedback when they are emotionally ready to do so. Students in the current study appreciated this aspect.


"…you don't have to share it with everyone whereas if you in a seminar and everyone's talking about what they got you kind of have to feel the pressure to join in whereas if you get in on Blackboard you can see it at your own leisure."


Students perceived that the ability to publish grades and feedback online enabled staff to return their feedback more quickly, keeping the feedback and grades in close proximity to the assessment activity. The importance of the timeliness of feedback is often mentioned in the literature but this tends to be anecdotal (HEFCE, 2007; Mutch, 2003). Clearly if students do not receive feedback in time for it to be meaningful, either in relation to the task assessed (a delay reduces the currency and relevance of the feedback) or to facilitate additional learning that can be taken into future assessments i.e. feed forward, then they are less likely to engage with their feedback.


Whilst students responded positively to the quick turnaround possible in receiving grades and feedback online, this did not follow when grades were made available online prior to feedback being made available elsewhere for collection, and in some cases after some considerable time had passed. In these circumstances students were less likely to engage with, or even collect, feedback (Winter & Dye, 2004). When grades are given before feedback, i.e. adaptive release  in reverse, it was found to be counter-productive; that is to say that when grades are given before feedback, the feedback is seen as less valuable than when feedback is given first.


The study found that students value the ability to monitor their own progression and to see how they are achieving on each assignment during, rather than following, the module (Carless, 2006; Maclellan, 2001). The Grade Centre collates grades enabling students to easily track progress and see how their performance on different assessment tasks builds to an overall profile for the module. This has been promoted internally as a key benefit of using the Grade Centre to publish grades and student in the study certainly reinforced the value of this approach. However, some students demonstrated a strategic approach to future assessments by focussing on the number of marks needed and using this to determine the degree of effort (Entwistle & Ramsden, 1983).


"…it keeps them [grades] all in one place; it means you can see how you're progressing throughout the course of the year and how well you need to do maybe in your next piece of coursework."


When delivered through the Grade Centre, feedback is automatically stored online and alongside other learning resources, and students commented on the value of this as they see the virtual learning environment as a learning hub. The study indicated a strong preference for the use of the Grade Centre as it has enabled staff to present grades and feedback alongside learning materials placing it 'in context' and providing a direct connection.


Students valued the perceived permanence of access to their online feedback. The study revealed that they frequently refer back to it to support future learning and assessments. This was different from the way in which students engaged with feedback when it was delivered hard copy. Students did value hardcopy feedback, many stating that they would never throw it away, but few had a logical storage system for such feedback and the majority rarely referred back to it after an initial read through and so its value was transitory.


The study highlighted conflicting views regarding handwritten and typed feedback. There are three key issues here; personalisation, thoughtfulness and legibility. A small number of students perceived handwritten feedback to be more personal as the tutor had taken time to write comments specifically for them. Although this perception suggested that electronic feedback was impersonal, this depended upon the way in which comments were presented. Electronic feedback can easily be made more personal through the use of the student's name and making reference to their previous assignments for example. Electronic feedback was, in some cases, perceived to be more thoughtful than handwritten feedback (Race, no date). Students recognised that tutors could more easily edit and revise their feedback as they read through assignments thus presenting a more cohesive and considered response. A large number of students claimed that they were more likely to engage with feedback when returned in a typed, and therefore legible, format (Bridge & Appleyard, 2005; Denton et al, 2008; Jones & Behrens, 2003; Price & Petre, 1997). While there were conflicting views, overall there was a strong preference for typed feedback.


"It obviously makes it a lot more beneficial to me as a student to receive that in a much more legible form … typed feedback is much better than written feedback, because you can read it, normally.  Lecturers have a tendency to scrawl."


Summary Findings


Overall, students expressed a strong preference for the publishing of feedback and grades online. It provides greater flexibility of access to feedback, enabling students to read and respond to feedback when they are emotionally ready and in relative privacy. The prompt return of feedback and grades also means students will be more inclined act on it because it is current, relevant and meaningful in terms of the original assessment. The storage of feedback alongside their learning offers a sense of permanence and students are more likely to refer back to it when working on future assignments. Access to grades in a single place enables students to monitor their progression and see how their performance on different assessment tasks contributes to the overall assessment profile. The use of technology pushes the feedback to the students removing the burden to seek out feedback from tutors and makes it easier for students to engage with their feedback as they have ultimate control over how, where and when they receive their feedback. Additionally, typed feedback is more legible and readable, which counters the perception that handwritten feedback can be more difficult to read and understand. 


Adaptive release of grades


The adaptive release of grades is facilitated at Sheffield Hallam University through the use of Assignment Handler. This enabled the project to explore students' perceptions of how adaptive release encourages them to engage with their feedback. Adaptive release through Assignment Handler allows tutors to release feedback to their students, but withholds the grade until the student has produced a reflective account on their feedback. Once this reflective account has been submitted, the grade is released automatically into the Blackboard Grade Centre without further intervention from the tutor.


Broadly, the project found that students understand the educational value of separating the grade from the feedback as a means of encouraging them to read and reflect on their feedback.


"It makes you think about your feedback because it's very easy just to read feedback and think 'oh, I did alright' or 'oh, that's not so good' but if it actually makes you think about it and you have to write about it because that's how you're going to get your grade then I think that's good for yourself."


Students acknowledged the benefits of reflecting on their feedback and recognised that this was important to improve future learning. However, the purpose of reflection and action planning needs to be made explicit in order to prevent students from taking an instrumentalist approach.


The project provides evidence to support Nichols (2007) recommendation of putting 'feedback before marks to encourage students to concentrate on the feedback first'. The students involved in the study clearly articulated the benefits of this process and the way in which it facilitates reflection not simply on the grade achieved but on the feedback received.


"If I have to reflect on the feedback before receiving the grade then it sticks in my mind a bit longer, the feedback I receive, the points that I'm going to use and it's a little bit easier to remember when I'm working on my next assignment."


However, a strong theme emerged in that students felt they had fulfilled the assessment task by completing their assignment. Writing a reflection was seen as an additional requirement and in some cases this need to engage with their feedback was negatively perceived as 'enforced' reflection. It emerged that adaptive release changes the boundaries of the assessment process and in order for students to fully engage with this approach, the importance of reflecting on their feedback must be identified as a key step in the process right from the start.


While the findings of the study support the notion that disengaging the grade from the feedback enhances student engagement with their feedback (Potts, 1992; Black & Wiliam, 1998; Carless, 2006), the process can cause frustrations and anxieties when not fully explained. The study found that students were more likely to engage with the process of reflection when they had been told explicitly a) that they would be required to reflect on their feedback before receiving their grade and b) why this would be of value to them.


As a new intervention, many of the students have never encountered the process before and this contributed to the importance of explaining it adequately. The study also highlighted uncertainty around the practice of reflection. Where students were required to reflect on their feedback with little guidance around what to write, who they were writing for and what would happen to their reflections, the intervention was much less effective in terms of encouraging reflection than for those students who fully understood the process.


Interestingly, some students believed that the key purpose of the reflection was to offer a response to the tutor regarding the quality of their feedback or the validity of the grade. This had the effect of inhibiting their engagement with the process. Others in the study correctly believed that the reflection was for their own benefit, and should be used for action planning. 


It is also interesting to note that very few students involved in the study had written formal action plans prior to the introduction of the adaptive release mechanism. During the interviews, many of the responses suggested a tacit, almost sub-conscious, approach to action planning. Ding (1998) suggests that when students read feedback comments, they do little with them. The current study would argue that while students may not demonstrate doing anything 'formal' with their feedback comments, they do in fact digest the comments and seek to remember them for future assignments.


"Yeah it's just stored in my memory.  I don’t tend to write action plans down.  I tend to retain things in my memory and then if I need to look something up I can usually remember where it is that I found it before."


This, however, is very difficult to monitor as the process is largely private and unknown. The introduction of the adaptive release mechanism has given students space to formalise this process and one cohort of students in particular are using the reflective accounts to feed in to Personal Development Portfolios.


Whilst students articulated the benefits of receiving feedback first, the study also highlighted their desire to receive their grade as soon as possible. Many perceived the reflective process as a burden or extra work, as discussed earlier. These students tended to be instrumental in their approach, choosing to ignore or rush the reflection on their feedback by submitting blank or surface level reflections.


The use of technology allows the immediate release of grades following the submission of the reflective account. Though the process of adaptive release may be possible without technology, the logistics of this would most likely result in a delay in receiving the grade. The immediacy allowed by the technology provides a practical solution.


Summary Findings


Overall, the benefits of the adaptive release mechanism were acknowledged by students when they fully understood the process. The use of Assignment Handler enables the adaptive release of grades to occur with large cohorts of students and releases the grades immediately on submission of the reflective account. This would be difficult to achieve without a technical intervention and helps to reduce frustrations experienced as a result of withholding grades. The study also found that students are able to produce action plans from their feedback but this is often a subconscious process and Assignment Handler provides a space for students to formalise this process. However, the study acknowledges that this is a new approach and therefore not many students will have encountered before and therefore emphasises the importance of explaining the process in order for students to fully engage with the reflective process and action planning.


Linking feedback to assessment criteria


One approach to presenting feedback to students is to provide feedback comments that are aligned directly with assessment criteria. This can be done paper-based but the opportunity to use electronically generated feedback from pre-populated comment banks has seen growth in this approach. There are a range of different tools that can be used to facilitate this linking and Sheffield Hallam University has been exploring the use of an internally-developed electronic Feedback Wizard, which allows tutors to generate individual feedback documents for an entire student cohort. Each document includes an assignment-specific feedback template containing a matrix of assessment criteria and feedback comments, and other remarks individually written for that student. This method is designed to offer detailed feedback to students in a consistent and equitable way.


At the time of this study, the Feedback Wizard was only in limited use across the institution and therefore few students involved in the study had experience of this tool, although a large number had received feedback, electronically or hardcopy that linked feedback comments to assessment criteria. Participants were shown examples of feedback grids generated by the Feedback Wizard and, coupled with their experience of receiving other forms of feedback grids, they were able to articulate the potential benefits of this approach.


Students suggested that they could understand feedback better when aligned to the original assessment criteria. The provision of this level of detail in an accessible format with explicit links to the assessment criteria was identified as a valuable approach to providing feedback (Maclellan, 2001). Students could easily identify their strengths and weaknesses against specific areas in a structured way that could lead to the development of action plans.


"You could really clearly see what you had to do for the next one and where you could actually improve."


Interestingly, given some students' earlier concerns that typed feedback was impersonal, none of the participants perceived the output generated by the Feedback Wizard to be so. This is even after the students were informed that the Feedback Wizard automatically populates the feedback grid from a bank of pre-populated comments, although individual comments can be written for each student.


On each feedback document, the Feedback Wizard provides an indicative weighted grade for every assessment criterion. Participants in the study perceived this approach to provide transparency in how tutors calculate the final grade for their work.


"If you just get … a percentage for a mark out of 20 or whatever then it doesn't really give you anything. Whereas if you understand maybe the process that the lecturer has gone through with regards to how he's got to that figure … it gives you a bit more of a basis of understanding as to how or why they've got to that point."


In order for this type of feedback to be effective, the study found that providing details of the assessment criteria with the assignment task was essential. This enables students to make connections between what they were hoping to attain and what they actually attained, and identify personal targets.


Interesting points were raised about how feedback should be presented. Students acknowledged that aligning feedback to assessment criteria and presenting this in a grid form summarised the comments clearly and cohesively. However, there was a competing preference for feedback to be positioned against the specific point in their original work, so they are able to identify easily the context of the feedback.


"It's quite interesting because you see exactly which bits have got their attention, especially if they've crossed something out which usually means a big no-no.  It makes it easy to see how you can improve next time because you know what they're looking for, which is an ideal way of doing it."


Summary findings


Overall, the study found that students liked linking feedback and grades to original assessment criteria as it enables them to identify their strengths and weaknesses at a glance. This can help students to reflect as they are able to use the assessment criteria to identify learning targets. Feedback presented in this manner also offers a level of transparency as students can see how their grades have been calculated. Technology enables this process to occur at scale, facilitating the generation of comment banks which can be used to create consistent but individual feedback. Without tools such as the Feedback Wizard, this is possible, but it would require greater effort and significantly more time on the part of the tutor. It would entail a great deal of repetition, which the technology eliminates. However, this approach does have limitations and there was a competing preference for 'in context' feedback suggesting that a mixed model would provide the most comprehensive feedback.