In Praise of the Imperfect Assessment

This is the first year since 2008 that I haven’t taught a class in winter, and I miss it. When I became dean of the School of Health & Community Services in July 2022, I quickly realized I would need to give up teaching, which is too bad, because teaching is what got me excited about the mysterious dynamics of teaching and learning with technology, why I enjoyed being a director of a teaching and learning centre for 8 years, why I pursued my doctoral degree, and ultimately why I am a dean.

It’s also why I was thrilled to be invited to be the keynote speaker at the Manitoba Academic Integrity Network at Brandon University in May 2023. I had the opportunity to share some thoughts and facilitate a conversation with a group of educators struggling to deal with a rapidly changing environment.  

For about 8 years, I taught Leadership and Management Principles at MacEwan University’s School of Business and then 5 years as part of the University of Alberta’s Graduate School of Library and Information Science.  The most popular reading in every offering of the course has been In Praise of the Imperfect Leader. It strikes a chord with students who often don’t think they have what it takes to be a leader.

As the authors write:  

We’ve come to expect a lot of our leaders, but no single person can possibly live up to those standards. It’s time to end the myth of the complete leader: the flawless person at the top who’s got it all figured out. In fact, the sooner leaders stop trying to be all things to all people, the better off their organizations will be. No one person could possibly stay on top of everything.  

For many of the students I have taught, this frees them to think about leadership differently, to think about themselves differently, and to embrace shared and distributed leadership models.  Being hopelessly incomplete takes the pressure off.  

And I think this pressure for perfection exists for assessments, as well. To paraphrase:  

We have come to expect a lot from our assessments, but no single assessment can live up to our highest pedagogical standards. It’s time to end the myth of the perfect assessment that can fix all problems – the holy grail of assessments that can achieve technical and essential student learning outcomes in meaningful and engaging ways at the same time they prevent academic integrity in a world with increasingly powerful technological tools.  

Then, when I was reading Justin Reich’s Failure to Disrupt, this sentence jumped out at me.

“All assessments are imperfectly designed.”

All assessments are imperfectly designed. Your assessments may be pretty good; they are all probably above average. But they are all imperfect, and they always will be. Breathe a sigh of relief and embrace this imperfection (especially you perfectionists). But because all assessments are imperfectly designed, it is important for us to come to understand their flaws. How are they imperfect, where are they imperfect, and why are they imperfect? Then and only then can we identify ways to improve them. That’s what this workshop was all about.

ACIFA's position on academic integrity

When my friend Rick and I decided to explore the experience of faculty who reported academic integrity violations, we knew it was a great and under-reported research topic. But we had no idea who would actually read it! For that reason, it was great to see our work prominently featured in ACIFA’s Position on Academic Integrity.

ACIFA brings attention to some of the main points of the work:

  • Reporting academic integrity violations involves a range of difficult emotions, including resentment, frustration, anxiety, fear, and some hurt feelings.

  • Students may harass faculty through email, in their offices or classrooms, and that it may disrupt relationships.

  • Even through these difficult experiences, however, many faculty still feel it is worth the time, effort, and agony because it defends honest students, protects the profession, and the reputation of their academic programs (and many times, their personal reputation, as well).

  • Reporting academic integrity violations should be done cautiously, with care and humanity.

ACIFA includes a number of recommendations that should provide opportunities for necessary discussions regarding academic integrity policy and practice in Canadian postsecondary institutions.

Artificial Intelligence & Academic Integrity: A Chrolological Bibliography

Because of my work exploring the intersection of authentic assessment and academic integrity, I am interested in the recent chatter on ChatGPT. What does it change, exactly? How can/should educators respond? What learning new learning opportunities does it present? This is a short selection of pieces from the explosion of articles that have come out about how the development of artificial intelligence will transform postsecondary assessment practice.

Mollick, E. R., & Mollick, L. (2022). New modes of learning enabled by AI chatbots. Three methods and assignments. SSRN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4300783

AI is a cheap way to provide students with many examples, some of which may be inaccurate, or need further explanation, or may simply be made up. For students with foundational knowledge of a topic, you can use AI to help them test their understanding, and explicitly push them to name and explain inaccuracies, gaps, and missing aspects of a topic. The AI can provide an unending series of examples of concepts and applications of those concepts and you can push students to: compare examples across different contexts, explain the core of a concept, and point out inconsistencies and missing information in the way the AI applies concepts to new situations.

Eaton, S. E., & Anselmo, L. (2023, January). Teaching and learning with artificial intelligence apps. Taylor Institute for Teaching and Learning. https://taylorinstitute.ucalgary.ca/teaching-with-AI-apps

Six thoughts on artificial intelligence and academic integrity (Eaton, 2022): 

  1. Using artificial intelligence for school work does not automatically equate to misconduct. 

  2. Artificial intelligence can be used ethically for teaching, learning, and assessment. 

  3. Trying to ban the use of artificial intelligence in school is not only futile, it is irresponsible. 

  4. Human imagination and creativity are not threatened by artificial intelligence. 

  5. Assessments must fit for purpose and should align with learning outcomes. 

  6. Artificial intelligence is not going anywhere. We must learn to work with new technology, not against it. 

Schmidt, H. (2023, January 22). ChatGPT, popular AI programs under watch at Waterloo region universities. Global News. https://kitchener.ctvnews.ca/chatgpt-popular-ai-programs-under-watch-at-waterloo-region-universities-1.6241471

"When I have essays or something, if I need to clarify the instructions, I’ll put the instructions into ChatGPT and then it’ll just give it to me in an easier way,"

Parsons, J. (January 30, 2023). Post-secondary sector must embrace AI technology in education. University of Waterloo. https://uwaterloo.ca/news/post-secondary-sector-must-embrace-emerging-ai-technology

“The first thing you have to say is it’s super disruptive,” says Dr. Marcel O’Gorman, a professor in the Department of English at the University of Waterloo and the founding director of the Critical Media Lab. “The question is, what is it disrupting? Sure, some of the discussion has to be about impacts in education, but I think that might be missing the mark.” (para 3)

There are always straightforward ways to adapt assessments to foster a culture of academic integrity and engagement. One easy way is to have students complete coursework that involves the creation and evaluation of knowledge, rather than more rudimentary assessment of memorization or simple understanding. (para 9)

Rigolino, R. E. (2023, January 31). With ChatGPT, we’re all editors now. Insider Higher Ed. https://www.insidehighered.com/views/2023/01/31/chatgpt-we-must-teach-students-be-editors-opinion

there’s an existential difference between spell-check and AI-generated writing. While computer programs can be leveraged to reduce the drudgery of proofreading for spelling, grammar and citation errors, these programs aren’t like ChatGPT, which produces coherent texts that students can hand in, with no revision, for a passing grade (at least some of the time). The text is being generated on behalf of the student and is being substituted for the student’s self-generated text. This use of AI is inherently dishonest. (para 9)

Kelly, S. M. (2023, February 1). ChatGPT creator rolls out ‘imperfect’ tool to help teachers spot potential cheating. CNN. ChatGPT creator rolls out 'imperfect' AI detection tool to help teachers spot potential cheating - CNN (ampproject.org)

"We really don't recommend taking this tool in isolation because we know that it can be wrong and will be wrong at times -- much like using AI for any kind of assessment purposes," Ahmad said. "We are emphasizing how important it is to keep a human in the loop ... and that it's just one data point among many others."

Ahmad notes that some teachers have referenced past examples of student work and writing style to gauge whether it was written by the student. While the new tool might provide another reference point, Ahmad said "teachers need to be really careful in how they include it in academic dishonesty decisions."

McMurtrie, B. (2023, February 2). Rethinking research papers, and other responses to ChatGPT. Teaching: The Chronicle of Higher Education. https://www.chronicle.com/newsletter/teaching/2023-02-02

Maier tried out an approach called the I-Search Paper, in which the subject becomes the process of searching for information, what the student learned, and what questions arose from that.

“It really is like a dissertation proposal,” he says, “on a much more informal level”: This is what I learned. This is a question that stemmed from that. This is why I think it’s really important and interesting. These are the kinds of sources I plan to use to find answers.

Most important, says Maier, students must explain how their research changed their thinking. While ChatGPT could be used as a research tool, the final product would be an original work.

Griffith, T. L. (2023, February 14). Why using AI tools like ChatGPT in my MBA innovation course is expected and not cheating. The Conversation.

https://theconversation.com/why-using-ai-tools-like-chatgpt-in-my-mba-innovation-course-is-expected-and-not-cheating-198957

In my course, the notion of “individual work” must change.

I’ll be adjusting the assignments and requiring an appendix describing the toolkit and practices students use. Using AI is not cheating in my course, but misrepresenting your sources is.

The AI will get better, and there will be more of them. Guidelines in work and education need to keep pace and be thoughtfully aligned to how knowledge is constructed in different fields.

Nagel, D. (2023, February 16). K16, GPTZero partner on AI writing detection tool. Campus Technology. https://campustechnology.com/articles/2023/02/16/k16-gptzero-partner-on-ai-writing-detection-tool.aspx

"This technology eliminates the manual process of faculty spot-checking student submissions one by one for potential AI-generated content. It also provides academic leaders with a complete and holistic picture of just how much student-submitted content across their institution is potentially AI-generated."

The academic integrity technological arms race is on. Students will pay for a subscription to ChatGPT (which will continue to get better at evading detection), and academic institution will use taxpayer dollars and student-tuition to subscribe to services to detect the AI-generated work. The cycle will likely ratchet up and become both more sophisticated and likely more expensive for institutions and students.

Benson, A. (2023, February 25). AI programs like ChatGPT could change Saskatchewan education, experts say. Global News. https://globalnews.ca/news/9511957/sask-education-chatgpt-ai/

some academic leaders are already turning to the program for help.

“Personally, I’ve been using it for just about everything,” University of Regina technology professor Alec Couros said. “Emails, discussions with students and developing lessons.”

McMurtrie, B. (16, March 2023). What you need to know about ChatGPT. Chronicle of Higher Education Teaching Newsletter. https://www.chronicle.com/newsletter/teaching/2023-03-16

  • Communicate with students

  • Be cautious about detection tools

  • There are better ways to bolster academic integrity (tapping into what is already known about good pedagogy can help, including designing assessments that seem valuable to students).

  • These tools can be an educational aid

  • Digital literacy is more important than ever

  • Start a conversation!

Milian, R. P., & Janzen, R. (2023, March 29). How are Canadian postsecondary students using ChatGPT? Academica Forum. https://forum.academica.ca/forum/canadian-students-and-chatgpt-a-new-learning-tool

Overall, these results suggest that as of February 2023, there were high levels of awareness of ChatGPT among Canadian postsecondary students, but relatively low rates of problematic use as it pertains to blatant cheating….Documented use of ChatGPT as a learning aid potentially reflects a need for additional learning supports, and an opportunity for institutions to intentionally employ AI technologies for these purposes. Technologies like ChatGPT can provide an invaluable resource for students who need technical terms explained in plain language or wish to have assignment questions reworded, for example.

Yang, H. (2023, April 12). How I use GhatGPT responsibly in my teaching. Nature. https://www.nature.com/articles/d41586-023-01026-9

In previous years, I’ve assigned a literature review to my students. This year, to avoid plagiarism and encourage creativity, I asked students to work in small groups to collect air-quality data on campus. However, the students will still use statistical methods to analyse the data themselves and write individual essays.

Of course, many students are not familiar with creating projects. Some struggled to come up with a suitable method to assess carbon dioxide emissions — so I suggested that they use ChatGPT to help them to design their projects. The model can outline several steps: from identifying a location to choosing a CO2 monitoring device, setting up the equipment, collecting and analysing data and presenting and disseminating the results.

The students did all of the work when it came to scientific analysis and writing their essays — but they also learnt how LLMs can generate scientific ideas and help to plan generic experiments.

Prentice, A-E. (2023, April 26). ChatGPT not the cheating wingman you need, Manitoba colleges, universities warn. Global News. https://globalnews.ca/news/9653932/chatgpt-cheating-manitoba-colleges-universities/

Foltynek, T., Bjelobaba, S., Glendinning, I., Khan, Z. R., Santos, R., Pavletic, P., & Kravjar, J. (2023). ENAI recommendations on the ethical use of artificial intelligence in education. International Journal for Educational Integrity, 19(12). https://edintegrity.biomedcentral.com/articles/10.1007/s40979-023-00133-4

Day, T. (2023). A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT. The Professional Geographer. https://www.tandfonline.com/doi/full/10.1080/00330124.2023.2190373

The discovery of fake sources also calls into question the veracity of statements made in ChatGPT answers to questions. The chatbot will likely improve, but initial enthusiasm should be tempered with a more nuanced and cautious approach to the application of AI chatbot technology to teaching and research. It is unlikely the technology will disappear because of the teething problems identified here and there are still many potential applications of AI chat-bots in postsecondary education (Conclusion, p. 3)

Janzen, R. (2023, May 17). Canadian PSE and the machine: Faculty, staff, and leaders share their thoughts on AI. Academica Forum. https://forum.academica.ca/forum/canadian-postsecondary-professionals-share-their-perspective-on-ai

The Three Dimensions of Leadership

I taught leadership for about 15 years, and I had a moment about four years ago when I realized I was teaching something I couldn’t define. This is actually a running joke in leadership studies, so let’s get it over with.

As far back as 1959, Warren Bennis said of leadership, that “of all the hazy and confounding areas in social psychology, leadership theory undoubtedly contends for top nomination.” Since then, things have probably gotten worse, since there is now servant leadership, complexity leadership, shared leadership, emergent leadership, mindful leadership, Indigenous leadership etc. For any number of reasons, some of these theories are quite hazy, indeed. Some, like shared leadership, need further research. Others, like Indigenous approaches to leadership, are finally being recognized and brought into formal study.  

Another example: “leadership is one of the most observed and least understood phenomena on earth.”  

This has gone on so often that by 1991, Joseph Rost chided leadership studies authors for making a couple of ritualistic comments in the course of their articles about the definition of leadership.  The first statement goes like this: Many scholars have studied leaders and leadership over the years, but there is still no clear idea of what leadership is or who leaders are.”

I don’t think that’s true. I don’t think I could give a definition of leadership that someone couldn’t pick apart, but whatever definition of leadership we use, there are generally three main dimensions.

  • Leadership is personal

  • Leadership is relational

  • Leadership is directional

Leadership is personal

The first dimension is that leadership is personal. The American poet and activist Muriel Ruykeser once said, the world is made up of stories, not atoms.

Bennis and Thomas (2011) suggest that personal meaning for leaders often arises from a crucible, "a transformative experience through which an individual comes to a new or altered sense of identity" (p. 101). Such experiences can include a violent, life-threatening event, the experience of prejudice, or a period of deep self-doubt. These crucible experiences build what Bennis and Thomas call 'adaptive capacity,' or "an almost magical ability to transcend adversity, with all its attendant stresses, and to emerge stronger than before" (p. 112). 

As Callahan and Martin write, we each must understand that one thing that holds the secret of our life. Each of us lives with our own story – a narrative that defines who we are and contributes to leadership effectiveness.  How well a leader knows and is defined by their story – whether it is an illness, a near-death experience, the experience of prejudice – these crucibles have a non-spurious relationship to leadership effectiveness, meaning if you know and are leading your story, you have a far greater chance of being a successful leader, no matter what you are doing.

One of my favourite leadership articles is Beware the Busy Manager. In their observation, they found that only 10% of managers are adept at using their time and energy. And one of the important differences is that purposeful managers work from the inside out. Purposeful managers decide first what they must achieve and then work to manage the external environment. A sense of personal volition characterizes the purposeful manager – the refusal to let other people or organizational constraints set the agenda.  

Purposeful managers are also skilled at finding ways to reduce stress and refuel. They commonly draw on a "personal well" - a defined source for positive energy. Some work out at the gym or get involved in sports. Others share their fears, frustrations, and thoughts about work with a partner, friend, or colleague. Still others refuel their inner reserves through hobbies like gardening.

Some of the worthwhile questions at this point are, what’s your leadership story. Why should anyone be led by you? Why do want to lead? Who do you want to lead? What difference do you want to make in people’s lives? In your community? What are you being called to do?

I think these are essential questions, and sometimes, we are facing struggles and difficulties at work because we have lost contact with our soul. If we can clarify our story and remember why we’re doing what we’re doing, perhaps everything will become more focused. So the most important relationship is that relationship with self.

 Leadership is Relational

Once we get that relationship right, then perhaps we can lead and manage the rest of our relationships in our lives. Leadership is relational. At its most basic level, we have the relationships inside our teams and outside our teams. Effective leaders are intentional and deliberate about the internal culture they are creating with their teams, and they cultivate transformative relationships with partners.

We now know the importance of networks. As smart as you are, if you can build a strong network, it will make you better. And sometimes, you will have to advocate to change the world. Some ways of conceiving the relational nature of leadership include:

Tikkun: Tikkun is the the Jewish notion of helping God to repair God's universe (Grigg, 2008, p.60). Including the concept of Tikkun (and any religious reference) may be automatically controversial for any number of reasons, but this religious concept of repairing a damaged world holds an important part of leadership as healing; leadership as connection to both the human community and the community of life.

Erich Fromm suggests in the Art of Loving that "to be concentrated in relation to others means primarily to be able to listen" (p. 105).  This practice of listening is a practice of love, and listening includes listening to Self. Fromm suggests, "the main condition for the achievement of love is the overcoming of one's narcissism" (p. 109), which requires the ability to see the difference between "my picture of a person and his behavior, as it is narcissistically distorted, and the person's reality as it exists regardless of my interests, needs, and fears (pp. 111-112).

Compassion, in facilitative leadership, is a temporary suspension of judgment so that we come to truly understand them and honor them.

Honor: "Honor is the right way for us to treat others (Callahan & Martin, 2007, p. 53). Honor  is to show respect for ourselves and our ideals, and to act within the confines of our ideals. Leaders characterized by the principle of honor are aware of and unthreatened by their own weaknesses. They are also unimpressed by their strengths and their talents. They see every interpersonal interaction as an opportunity to grow, and, more importantly, for others to grow as well. The desire to honor others is one of the points on their personal compass (Callahan & Martin, 2007, p. 54) 

To be constrained by one's ideals is an important consideration. There may be times when intimidation or humiliation might be efficacious tactics for a leader to achieve their vision or goals, but engaging in these tactics would invite dishonor upon them. To be unthreatened by our own weaknesses is also to embrace the forgotten virtue of humility.   

Humility: To be unimpressed with one strengths and talents displays humility, one of the cornerstones of Jim Collins' conception of the Level 5 Leader. Level 5 leaders represent, in Collins' model (2011), the pinnacle of leadership, and Level 5 leaders routinely credit others, external factors, and good luck for their companies' success. But when results are poor, they blame themselves. They also act quietly, calmly, and determinedly - relying on inspired standards, not inspiring charisma, to motivate (p. 118).  

Collins goes on to describe the interviews with the transformative executives that led to the construction of the Level 5 leader. And what he observed was that  

throughout our interviews with such executives, we were struck by the way that they talked about themselves - or rather, didn't talk about themselves. They go on and on about the company and contributions of other executives, but they would instinctively deflect discussion about their own role. When pressed to talk about themselves, they'd say things like, 'I hope I'm not sounding like a big shot,' or 'I don't think I can take much credit for what happened. We were blessed with marvelous people.' One Level 5 leader even asserted, 'There are a lot of people in this company who could do my job better than I do'” (Collins, 2011, p. 126).  

And true humility is NOT to just say it, but sincerely mean it, and be grateful for the opportunity to lead. 

Nowhere is this relationality captured more clearly than in Siemens, Dawson, and Eshleman’s article on complexity leadership.  Siemens and Downes invented the concept of connectivism, and we see here the connections between people, networks, processes, and tools. These concepts of exchange, self-organization, feedback and economic impact are deeply relational. If you introduce a new tool, what new processes need to be built, and how will this impact people’s identity.

In the relational lens, some of the reflection questions to use to focus on the what is your most important relationship at work or at school? What is your most problematic relationship with your work? Is it with a process? A person? A tool? Who do you most need to forgive? What leadership education do you need to do?

Leadership is directional

Finally, leadership is directional. Leadership implies direction. How do we adapt our organizations to the dynamic changes taking place (a reactive position), and how do we sculpt and shape the world through persuasion, advocacy, and recognition of how the trends can be utilized to help us achieve our mission (a proactive position)? The directionality of leadership  

Change, or adaptation, adaptive positioning, is extremely hard because there are many trends happening all at once. Each of these many political, economic, social, and technological changes impact our programs and the conduct of our business in some form or fashion.

This unalterable changefulness and irredeemable instability can alter patterns of dominance, and these external pressures are likely to create fissures and cracks that can be exploited through bargaining, negotiation, and jockeying for position. Within any postsecondary institution, power is structurally entrenched and unequally distributed. Executives hold greater influence over organizational vision, resource allocation, and advocacy with policymakers. But most postsecondary institutions also have managers with their vision of the future, along with faculty associations and other unionized employees that can and do exercise influence.

Slowly shifting patterns of dominance, such as Indigenization and the call for greater equity, diversity and inclusion (EDI) in postsecondary education after tumultuous racial events in the United States and Canada stand in contrast to totalizing views of power. There are occasions when we lead the culture.  

Some of the important questions to ask at this point are:

What trend do you think holds the most promise for your function, your department the college?
What internal/external advocacy needs to be done?
What leadership education needs to be done? How does this align with your leader’s goals?
What trend provides you more than few midnight anxieties about the future?What skills and equipment do the people who are going to be part of your expedition need?

My eyes are closed!

https://www.opcw.org/media-centre/news/2022/07/opcw-advisory-board-education-and-outreach-advance-strategies-promote

I have just returned from The Hague, where I had the privilege to attend the 13th meeting of the Advisory Board for Education and Outreach of the #OPCW. Before the meeting, I had the great honour to work with several of the leaders of the OPCW to discuss a model for high-end e-learning to inform the OPCW's global educational initiatives. I began by asking them to describe a powerful learning experience. They shared incredible stories and identified these common elements of peak learning experiences:

* Feeling overwhelmed
* Exposure to content that deepened their understanding of the world
* Diversity of opinions within the learning environment
* Interactivity with the instructor and other students
* The growth of self-efficacy
* The opportunity to express what they were learning with family, colleagues, and friends
* Relevance to their real-world environments
* Reinforcing and strengthening their internal motivation

This describes what the best faculty and instructional designers try to achieve in any learning experience in any modality. High end e-learning requires particular attention to these elements, and this can only be achieved if the organization has strategy, infrastructure, and well-prepared instructors who can build community and connection. For an institution like the OPCW, these essentials can be supplemented by Open Education Resources and the issuance of micro-credentials that learners value.

It was a great and exciting meeting, and I am thrilled to be part of such an important mission. (Of course, my eyes are closed in the picture!).

Authentic Assessment for Online Learning MOOC - Final Report

Executive Summary

Authentic Assessment for Online Learning (AAOL) is a massive open online course (MOOC) first offered through the Commonwealth of Learning May 9 – June 3, 2022 using the mooKIT platform developed by the Indian Institute of Technology, Kanpur. The course curriculum was initially drafted by Dr. Dianne Conrad and revised by Dr. Jason Openo in the Winter of 2022. AAOL introduced the theory and practice of authentic assessments as the heart of the learning experience, with an emphasis on how authentic assessments intersect with academic integrity and various forms of human diversity.

The final curriculum for the four-week MOOC was provided by content expert Dr. Jason Openo, co-author of Assessment Strategies in Online Learning: Engagement and Authenticity (2018). AAOL explored the concept of learner-centred design for online assessment in higher education using short videos, open access readings, interactive forum discussions, and a learning portfolio assignment designed to model the characteristics and qualities of authentic assessments. Aligned with the theory of constructivism and authentic assessment, AAOL explored the changing nature of work in a digital age and the competencies and skills needed in the contemporary workplace, focusing on assessment strategies that engage and motivate learners in the e-learning environment and promote both academic integrity and deep learning. The course provided an overview of the fundamentals of creating learner-centered digital assessment through 21st century examples using the 5-dimensional framework for authentic assessment as both a tool for diagnosing existing assessments for authenticity and as a design template for new assessments.

The course was divided into 4 units. Unit 1 explored how the COVID-19 pandemic disrupted traditional forms of assessment and provided an opportunity to rethink assessment practice in postsecondary education. Unit 2 explored the intersection of authentic assessment and academic integrity. Unit 3 highlighted how authentic assessment is a values-based, context-dependent approach that can be applied in different cultural contexts, and how the values of authentic assessments overlap with and support Universal Design for Learning (UDL) principles. Unit 4 reviewed the challenges of moving towards more authentic assessments using practical examples from a wide variety of disciplines. A full outline of the curriculum is included in the final report.

2,541 individuals registered for AAOL, demonstrating the timely appeal of this learning topic. 1,522 active participants were drawn from all over the world, with the greatest concentrations in India (409), Nigeria (281), Kenya (243), Ghana (193), South Africa (132) and Malaysia (125).

An overwhelming majority of students were either postgraduates (1,192) or doctoral level educators (528). An overwhelming majority came from academic institutions (1,467) or governmental institutions (385).

There was a strong response to AAOL, and AAOL was well-received by participants. The following report documents the strong response to AAOL, suggesting that authentic assessment in online learning is a timely educational development topic, and that the instructional team provided a high-quality learning experience modeling best practices in online learning design. The report concludes with several learnings that may be used to refine this course’s development and the development of future of MOOCs, including comments on the mooKIT platform, and learner behaviour in a MOOC within the context of academic integrity.  

Participants submitted 358 pre-course surveys and 220 course evaluations. Respondents to an end-of-course survey (220) expressed a high level of satisfaction with the course, with 95.1% agreeing or strongly agreeing to a statement that learners got what they wanted out of the MOOC, and 99% agreeing or strongly agreeing to a statement that they had gained a greater understanding of authentic assessments.[JFA1]  As the raw responses to the survey indicate, the diverse array of learners valued different learning elements of the course, including the strong teaching presence of the instructional team, the curricular focus on the 9 Principles of Good Practice in Assessing Student Learning, the unit on academic integrity, and the practical steps of designing authentic assessments with UDL principles in mind. Participants specifically cited the recorded lectures, the reading materials, and the community as important elements that contributed to their learning.  

AAOL was designed for digital learning enthusiasts, senior managers, instructors, and others who are interested in building knowledge, competencies and skills that are usable in or transferable to the postsecondary workplace. The forums and the assessments offered a choice to participate in the course as a faculty member or as a leader within an academic institution (e.g., chair, dean, administrator). Strong participation in the varied pathways of the AAOL MOOC suggests learner choice held value, and that multiple levels of institutional influence are required to support the move to more authentic assessments.

Two types of certificates were available for participants of AAOL, a Certificate of Participation and a Certificate of Completion, which were granted according to participants’ level of participation and completion of assessment activities. The Certificate of Completion was awarded to participants completing a minimum of 60% on all 4 unit quizzes, the completion of the learning journal to required specifications, and meaningful participation in a minimum of 4 forum discussions. Certificates of Participation were awarded to those who received a minimum of 60% on all 4 quizzes and meaningful participation in at least 3 forum discussions.

There were a total of 376 certificates issued: 235 Certificates of Competency and 141 Certificates of Participation, for a total certification rate of 14.2%.

The following report elaborates upon the curriculum, the participants’ experience, and reflections from the instructional team for growth and improvement of the CoL’s future MOOC offerings.

Authentic assessment and academic integrity

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

This question appeared in the MOOC forum today: “What is the best practice in curbing academic dishonesty among undergraduate students in higher institutions?”

There are at least 3 different answers to this. The first answer is we don’t know. Students commit academic integrity violations for lots of different reasons. For some, they worry they will not be able to achieve the reward they desire on their own merits; for some, cheating provides a thrill. To know what would curb academic dishonesty, we would need to know that a student was going to cheat or be dishonest but didn’t because we intervened somehow and our intervention was successful in changing their mind and behaviour. Establishing this causal chain of evidence is very difficult research. If we actually knew what worked, postsecondary institutions around the world would be in a much better place.

The second and most prominent approach involves telling students what academic integrity is, providing examples, and then telling them that if they commit an academic integrity violation and get caught, they will be punished severely. The most dominant method is punitive and threatening. This is the path of increased surveillance and the technological arms race where students get more creative in their approaches, including hacking institutional IT systems to change a grade or steal the final exam. Do a Google search for "how to cheat Respondus Monitor" or "how to cheat Proctorio." If we build a jail, there will always be someone thinking about how to escape.

The best practice, and the one this course advocates for, is the third way to answer this question. The best way to curb academic dishonesty is to create relevant and engaging learning tasks that connect with a learner’s motivation and goals. Different cultural conceptions of textual ownership, copyright, language, and educational systems all come into play when discussing academic integrity, but all students have ethical beliefs and learning goals, and if the assessments we design ask students to engage in meaningful tasks they care about, they are less likely to act in a dishonest fashion. Students are more likely to cheat in less personalized, less task-oriented learning environments. By creating more personalize and more task-oriented learning environments, authentic assessments can appeal to students’ higher natures.

Technology cannot stop academic dishonesty. Authentic assessments will not put an end to cheating. But they can help by changing the nature of the game.

Exams and authenticity

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

Are exams authentic? Can they be authentic?

The short answer is yes. The longer answer is in certain circumstances.

There are disciplines where exams are a concrete reality. In my context, for example, if you wish to be a plumber, welder, engineer, nurse, or information technology professional, there are certain exams students must successfully complete to participate in professional practice. In those instances, educators have a professional (perhaps moral) obligation to do what they can to best prepare their learners to take those exams. Using the 5-dimensional framework:

  • Task – Pass the discipline specific exam

  • Physical/digital context: Online proctored exam

  • Social context: Individual, private assessment

  • Polished product: Test score

  • Criteria and standards: Right or Wrong? (1st time pass rate)

In these certain circumstances, exams are authentic because they match what a learner must be able to achieve in a real-world context. In my context, if a nursing student doesn’t pass the NCLEX, they don’t get to be a nurse. Our Nursing program is also judged on the percentage of nursing students who pass the NCLEX their first time. In those certain circumstances, online or in-person proctoring will be required to ensure that students are honest. But during their 4-years of study, our nursing program also makes heavy use of simulation and clinical, which come as close as possible to the ideal of authentic assessment. Students are recorded doing activities, and they can’t fake it.

Remember the major point from the last post; all assessment strategies are imperfect. Because we cannot achieve perfectly authentic assessments in this imperfect world, we need to be clear about how our assessments are imperfect. There will be rare times and occasions when exams are authentic, but exams are often accompanied by other assessment strategies that provide a holistic understanding of the learning process.  

Dr. J

Authentic assessment is a political act

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

Authentic assessment is a political act. As the 9 Principles of Good Practice in Assessing Student Learning puts it, “There is a compelling public stake in education. As educators, we have a responsibility to the publics that support or depend on us to provide information about the ways in which our students meet goals and expectations” (Astin, et al.) We have an obligation to ourselves as educators, to the disciplines we care about, and to our students. But we also have an obligation to society to improve our assessment strategies.

I just finished reading Failure to disrupt: Why technology alone can’t transform education (Reich, 2020). Reich argues that “all assessments are imperfectly designed” (p. 177), which means you won’t and can’t develop a perfectly authentic assessment. But it also means we need to pay close attention to how our assessments are imperfect. In online education, more and more people are turning towards online proctoring and/or auto-grading. There are several problems with this. First, surveillance is not good pedagogy. Secondly, autograders are good at routine tasks, but “these are exactly the kinds of task we no longer need humans to do” (p. 171). Third, as Reich states, “we need students to develop complex communication skills and take on unstructured problems,” such as problem finding and problem framing. If we want to assess people on the kinds of performance that are most worthwhile for people to learn, we will have to depend heavily on assessments evaluated individually, and this represents a challenge to large-scale learning environments. 

I’ll talk more about exams and essays tomorrow. There are times when they are necessary and occasions when they are authentic, but they are imperfect in very different ways than authentic assessment, and one of their imperfections (as noted by the International Centre for Academic Integrity and the European Network for Academic Integrity) is that they are the assessments students are most motivated to cheat on or can most easily outsource through contract cheating.

Dr. J

Astin, A. W., Banta, T. W., Cross, P., El-Khawas, E., Ewell, P. T., Hutchings, P., Marchese, T. J., McClenney, K. M., Mentkowski, M., Miller, M.A., Morgan, E. T., & Wright, B. D. Nine principles of good practice for assessing student learning. https://www.ncat.edu/_files/pdfs/campus-life/nine-principles.pdf

Reich, J. (2020). Failure to disrupt: Why technology alone can’t transform education. Harvard University Press.

Assessment & Teaching Presence

Moore’s 3 forms of interaction matured and evolved in the Community of Inquiry’s 3 presences – cognitive presence, teaching presence, and social presence. Within the Community of Inquiry framework, assessment is part of teaching presence, defined as “the unifying force” that “brings together the social and cognitive processes directed to personally meaningful and educationally worthwhile outcomes” (Vaughn, et al., 2013, p. 12).  Teaching presence consists of design, facilitation, organization, delivery, and assessment.

“Assessment very much shapes the quality of learning and the quality of teaching. In short, students do what is rewarded. For this reason one must be sure to reward activities that encourage deep and meaningful approaches to learning” (Vaughn, et al., 2013, p. 42). 

In its simplest terms, deep learning is good and surface learning is bad. When adopting a surface learning approach, students consume content as a commodity to be acquired through a form of mechanistic transfer.

Students who take a deep learning approach “recognize the dynamic and interrelated structure of content to be learned, and learning is less a process of knowledge transfer than one of exploration, discovery, and ultimately, growth” (Platow, et al., 2010).  Deep learning is an approach to learning consciously or unconsciously selected by the student, but the design of the learning opportunity encourages students to adopt a particular approach.  

In short, assessment is the heart of the student experience, and assessment design will determine how students behave and engage with the learning environment. Authentic assessments change the nature of the relationship to content and can encourage students towards deep learning. Deep learning principles recommend active and interactive learning, and “graded activities that require collaboration and constructivist thought will encourage students to work toward this end” (Vaughn, et al., 2013, p. 33).

Dr. J

Platow, M. J., Mavor, K. I., & Grace, D. M. (2013). On the role of discipline-related self-concept in deep and surface approaches to learning among university students.  Instructional Science: An International Journal of the Learning Sciences, 41(2), 271-285. doi: 10.1007/s1125101292274

Vaughan, N. D., Garrison, D. R., & Cleveland-Innes, M.  (2014). Teaching in blended learning environments: Creating and sustaining communities of inquiry. Athabasca University Press. http://site.ebrary.com/id/10840609