Kritik is an online peer-to-peer interactive learning platform for helping professors engage students in a twenty-first-century way. Students can submit their assigned activity online and get evaluated based on a rubric and receive written feedback from their peers on whether they did well, and how they could do better. When you assess your peers’ works, you receive a credit for critical thinking based on the fairness of your evaluation. This is called the Kritik Score. The credit is calculated and adjusted automatically by Kritik’s scoring system. Through a gamified experience, Kritik allows students to develop higher order thinking skills through creating assignments, analyzing and evaluating peers’ submissions. In addition, students will develop the skills necessary to deliver feedback to their peers through a Feedback-on-feedback or FOF plot. The FOF plot illustrates how students' evaluation comment are both Critical and Motivational at the same time.
How Kritik Works
The teacher creates a HOTS (Higher Order Thinking Skills) activity with a grading rubric and schedule it to go through three stages: Create >> Evaluate >> Feedback.
In Create stage, you will create your assignment and submit your work.
In Evaluate stage, you will be assigned to grade up to 5 pieces of submissions (might be less or more depending on what your instructor has set) created by your peers. Similarly, your submission will also be evaluated by your peers.
In Feedback stage, you will first be able to see your evaluators’ comments on your submission. You will review the comments from your peers and provide feedback based on your perception to help them develop their feedback delivery. Once you provided feedback to all evaluators, you will then see your overall submission grade for the current activity as well as the class average. Your submission grade is the weighted average of the grades your peer evaluators have given to your submission. The weighted average is calculated by multiplying each evaluator's grade by the evaluator's Kritik Score. Hence, students with higher Kritik Score have a bigger impact in peer grading.
After you have completed the Feedback stage, your activity grade will be approved and finalized. At the same time, your Kritik Score and Feedback Delivery Score will be adjusted based on the quality of peer evaluations and feedback comments. Your progress in these two scores over the last assignments are displayed in your dashboard.
Throughout this cycle, your teacher and TAs have full visibility to all submissions and evaluations at any time. They may choose to interject, send you a note or edit peer evaluations as they see necessary.
The Science behind Kritik
Lower Levels of Cognitive Learning
According to the paper published by Benjamin Bloom in 1956, the complete cognitive functioning in learning must include six core elements. First, Memorization which is remembering and recalling names, and labels. Secondly, Comprehension, understanding how a concept works and being able to explain it beyond the mere vocabulary we initially memorized. Third is Application, which is applying the concept that we understand in a real-life situation. Above are the first three stages of Bloom's Taxonomy. Furthermore, up until Application, we have completed the lowest half of the cognitive levels.
Higher Levels of Cognitive Learning
Next, we are going to deepen our learning by entering the higher order thinking realm following with Analysis. Analysis consists of breaking down ideas or objects into smaller components, drawing connections and finding evidence to support generalizations. Next, Evaluation is rejecting or defending a stand or decision based on Analysis. Finally, at the highest level is Creation, which is an individual being able to produce a new idea or object by compiling components in a creative and innovative way. Kritik focuses on strengthening students' higher order of thinking skills (HOTS), also referred to as critical thinking skills.
How to Evaluate a Peer
In the Evaluate stage, you are tasked to evaluatepeers anonymously.
For rubric evaluation, carefully follow the descriptions provided in the rubric cells. In the comment section, try to bring up one or maximum two important things that your peers could do to improve, but don’t forget to mention any great things about their work to reinforce those actions. Make your points clear and actionable and if possible, provide specific examples.
It is crucially important to keep a positive and constructive tone and avoid using words that may demotivate the readers or offend them. Our brain has evolved to remember negative feedback much longer than positive ones. Therefore, for every critical feedback on areas of improvement, you should mention two or more things that are positive about your peers’ submission, so reading your feedback will leave them with a net neutral or positive feeling and keep their motivation.
Don’t worry at the beginning when you are still uncertain how to evaluate or provide comment. It is your participation that matters. After completing a few HOTS activities and peer evaluations through the semester, and by paying careful attention to the changes to your Kritik Score and Feedback Delivery Score, as well as any potential comments you may have received from peers you have evaluated, you will soon learn how to fairly evaluate others work just like a TA!
How to Give Feedback on Feedback
In the Feedback stage, all the comments from your peers along with their Kritik Score will be revealed to you, while their identity remain anonymous. You have the opportunity to give feedback to the evaluators on the quality of their comment and help them adjust their feedback delivery in their future evaluations. You are required to give your feedback by picking an appropriate point in the 2D Feedback-on-feedback(FOF) plot and you may also submit a comment.
In the FOF graph, the x-axis dimension represents the level to which you find a comment critical while the y-axis is the level of perceived motivation from the tone.
The right side of the plot are for the feedback with clear and actionable tips that help you improve your submission. If you feel the feedback is too soft without challenging you to improve, pick a point on the left side of the spectrum. Similarly, if the feedback, no matter how critical, is written in a constructive and positive tone, pick a point at the upper side of the graph. But, if you feel the tone is harsh and or doesn’t acknowledge any positive aspect of your submission, you may choose a point in the lower side of the graph. Both dimensions considered together, the best feedback should be marked at the top right corner of the graph and the worse one at the bottom left corner.
Pay attention to what your instructor may have defined the x and y axis for you as they may differ from Kritik’s default ones.
How to Dispute and Flag
After you have received the evaluations from your peers, and while the activity is still in the "Feedback" stage, you may dispute your grade only if you believe your overall grade doesn’t fairly reflect your submission based on the rubric. You may also flag your peers’ comments if you think they are offensive or inappropriate.
When you dispute an item, your teacher or TA of the course will be notified. He/she will review your work and resolves the dispute by either increasing, decreasing or keeping your grade. Keep in mind that your action to “dispute” is a reflection of your “evaluation” skills. Therefore, you may lose your Kritik Score if the teacher doesn’t agree with your dispute.
How the KritikScore works
Kritik Score, KS, a number between 1 to 6, is a representation of how fair and accurate you grade your peers according to the set rubric. It also determines how much you can influence your peers final grade. For example, if your KS is 6, your evaluations of peers hold 6 times more weight than someone whose KS is 1.
At the beginning, everyone starts at 2. When an item completes its cycle and the teacher approves final evaluations for an item, the software re-adjusts your KS based on how accurately you peer evaluated the rubrics. You may gain or lose KS or your net KS change may be zero in each round.
Pay attention to how the border around your profile photo changes. It is where your KS for any particular course is displayed.
Don’t worry about losing KS at the beginning. There is no penalty for losing KS at the start of the semester, However, your teacher may assign bonus points to those whose KS has passed a certain threshold by the end of the semester.
How Marks are Calculated
Creation mark: your total rubric grade for the HOTS activities: It is typically the weighted average of the rubric grades given by your peers, unless the instructor has edited it.
Kritik Score: Your most recent Kritik Score, displayed by the border around your profile photo or in your dashboard.
Feedback Delivery Score: Average feedback-on-feedback points you received from students you peer evaluated in the last completed activity.
Participation to peer evaluation: It is the percentage of completed peer evaluations during Evaluate stage
Participation to Feedback on Feedback: it is the the percentage of your completed Feedback-on-Feedback task you have completed during Feedback stage
Your Kritik Score and Feedback Delivery Score (FDS) are always available in your dashboard while the other grades can be accessed by downloading your gradebook excel file.