Authentic assessment is a political act

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

Authentic assessment is a political act. As the 9 Principles of Good Practice in Assessing Student Learning puts it, “There is a compelling public stake in education. As educators, we have a responsibility to the publics that support or depend on us to provide information about the ways in which our students meet goals and expectations” (Astin, et al.) We have an obligation to ourselves as educators, to the disciplines we care about, and to our students. But we also have an obligation to society to improve our assessment strategies.

I just finished reading Failure to disrupt: Why technology alone can’t transform education (Reich, 2020). Reich argues that “all assessments are imperfectly designed” (p. 177), which means you won’t and can’t develop a perfectly authentic assessment. But it also means we need to pay close attention to how our assessments are imperfect. In online education, more and more people are turning towards online proctoring and/or auto-grading. There are several problems with this. First, surveillance is not good pedagogy. Secondly, autograders are good at routine tasks, but “these are exactly the kinds of task we no longer need humans to do” (p. 171). Third, as Reich states, “we need students to develop complex communication skills and take on unstructured problems,” such as problem finding and problem framing. If we want to assess people on the kinds of performance that are most worthwhile for people to learn, we will have to depend heavily on assessments evaluated individually, and this represents a challenge to large-scale learning environments. 

I’ll talk more about exams and essays tomorrow. There are times when they are necessary and occasions when they are authentic, but they are imperfect in very different ways than authentic assessment, and one of their imperfections (as noted by the International Centre for Academic Integrity and the European Network for Academic Integrity) is that they are the assessments students are most motivated to cheat on or can most easily outsource through contract cheating.

Dr. J

Astin, A. W., Banta, T. W., Cross, P., El-Khawas, E., Ewell, P. T., Hutchings, P., Marchese, T. J., McClenney, K. M., Mentkowski, M., Miller, M.A., Morgan, E. T., & Wright, B. D. Nine principles of good practice for assessing student learning. https://www.ncat.edu/_files/pdfs/campus-life/nine-principles.pdf

Reich, J. (2020). Failure to disrupt: Why technology alone can’t transform education. Harvard University Press.

Assessment & Teaching Presence

Moore’s 3 forms of interaction matured and evolved in the Community of Inquiry’s 3 presences – cognitive presence, teaching presence, and social presence. Within the Community of Inquiry framework, assessment is part of teaching presence, defined as “the unifying force” that “brings together the social and cognitive processes directed to personally meaningful and educationally worthwhile outcomes” (Vaughn, et al., 2013, p. 12).  Teaching presence consists of design, facilitation, organization, delivery, and assessment.

“Assessment very much shapes the quality of learning and the quality of teaching. In short, students do what is rewarded. For this reason one must be sure to reward activities that encourage deep and meaningful approaches to learning” (Vaughn, et al., 2013, p. 42). 

In its simplest terms, deep learning is good and surface learning is bad. When adopting a surface learning approach, students consume content as a commodity to be acquired through a form of mechanistic transfer.

Students who take a deep learning approach “recognize the dynamic and interrelated structure of content to be learned, and learning is less a process of knowledge transfer than one of exploration, discovery, and ultimately, growth” (Platow, et al., 2010).  Deep learning is an approach to learning consciously or unconsciously selected by the student, but the design of the learning opportunity encourages students to adopt a particular approach.  

In short, assessment is the heart of the student experience, and assessment design will determine how students behave and engage with the learning environment. Authentic assessments change the nature of the relationship to content and can encourage students towards deep learning. Deep learning principles recommend active and interactive learning, and “graded activities that require collaboration and constructivist thought will encourage students to work toward this end” (Vaughn, et al., 2013, p. 33).

Dr. J

Platow, M. J., Mavor, K. I., & Grace, D. M. (2013). On the role of discipline-related self-concept in deep and surface approaches to learning among university students.  Instructional Science: An International Journal of the Learning Sciences, 41(2), 271-285. doi: 10.1007/s1125101292274

Vaughan, N. D., Garrison, D. R., & Cleveland-Innes, M.  (2014). Teaching in blended learning environments: Creating and sustaining communities of inquiry. Athabasca University Press. http://site.ebrary.com/id/10840609

Authentic Assessment for Online Learning - The 3 Interactions

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

As efforts to expand online education proceed, it is critical to design more interactive educational experiences that integrate regular, direct, and meaningful contact and communication (Protopsaltis & Baum, 2019).

Moore (1989) realized this 30 years ago, and he identified three types of learner interaction.

  1. Learner-content

  2. Learner-instructor

  3. Learner-learner

Learner-content interaction “is a defining characteristic of education. Without it, there cannot be education, since it is the process of intellectually interacting with content that results in changes in the learner’s understanding” (p. 1).  Learner-content in this course takes place with the lectures and the readings, the quizzes, and the assignments. The “tyranny of content” (Petersen, et al., 2020) can squeeze out other forms of interaction, however.

Learner-instructor interaction takes place through the curriculum , where the instructors seeks to stimulate the learner’s interest and motivation. Learner-instructor interaction takes place directly in the lectures, and indirectly in every aspect of the course.

Learner-learner interaction, as Moore writes, “is sometimes an extremely valuable resource for learning.” I would go even further; my favorite quote about teaching is probably this one: "The best answer to the question, 'What is the most effective method of teaching?' is that it depends on the goal, the student, the content and the teacher. But the next best answer is, 'students teaching other students.'" (McKeachie, et al., 1987, p. 63). Learner-learner interaction takes place in the forums, where the learners have responded to, expanded upon, and enlightened others with their knowledge and experience. It also takes place in collaborative work.

Authentic assessments have the potential to weave these three interactions into a single learning event. Learners interact with the content in relevant, real-world ways. The instructor designs the curriculum and the authentic task, and can provide coaching to the students. If the task is a collaborative one, learners interact with each other in the process of building the polished product. This is partly why authentic assessments can be such powerful learning experiences.

Dr. J

McKeachie, W. J., Pintrich, P. R., Lin, Y., and Smith, D. A. F. National Center for Research to Improve Postsecondary Teaching and Learning, A. M., & And, O. (1987). Teaching and Learning in the College Classroom. A Review of the Research Literature (1986) and November 1987 Supplement. 

Moore, Michael. (1989). Three Types of Interaction. American Journal of Distance Education. 3. 1-7. 10.1080/08923648909526659. 

Petersen, C. I., Baepler, P., Beitz, A., Ching, P., Gorman, K. S., Neudauer, C. L., Rozaitis, W., Walker, J.D., & Winger, D. (2020). The tyranny of content: "Content coverage" as a barrier to evidence-based teaching approaches and ways to overcome it. Life Sciences Education, 19(2). https://www.lifescied.org/doi/10.1187/cbe.19-04-0079

Protopsaltis, S., & Baum, S. (2019). Does online education live up to its promise: A look at the evidence and implications for federal policy. George Mason University. https://distance-educator.com/does-online-education-live-up-to-its-promise-a-look-at-the-evidence-and-implications-for-federal-policy/

Authentic Assessment for Online Learning - MOOC

https://www.col.org/news/registration-open-for-on-authentic-assessment-for-online-learning/

May 9, 2022 was one of the most exciting days of my professional life because Authentic Assessments for Online Learning launched with 2,440 dedicated educators from around the world! Thank you all for joining this course! 

One of the participants asked, "What is online learning?" This is a seemingly simple question with NO clear answer? 

Duus (2009) observes it is difficult to refer to online education as a singular entity because it is so big (like referring to an ocean as a big body of salt water). Duus makes a critical distinction between “low-end e-learning” and “high-end e-learning." Low-end e-learning is characterized by content transmission and is volume-based. Technology provides the innovation to these popular, mainstream strategies that are often erroneously made synonymous with all forms of e-learning (Duus, 2009, Figure 1). Duus argues that most of the discourse and academic research has focused on low-end e-learning, and while it is simplistic to categorize it this way, low-end e-learning is best conceived as content-heavy education with low interactivity (Protopsaltis & Baum, 2019, p. 30).

Moving towards a learning-centred instructional approach means moving towards interactivity, moving towards student engagement and student knowledge-construction. Moving towards authentic assessments is a move from low-end to high-end e-learning. 

Online education is a form of distance education where the primary delivery mechanism is via internet-based technologies, but this definition lacks nuance because the terminology struggles to keep up with the invention of new approaches and applications. Online education may include synchronous “face-to-face” technologies such as Blackboard Collaborate Ultra, Zoom, or Google Meet, asynchronous or multi-synchronous platforms such as the learning management system (LMS) and Google Docs, and/or participatory flow technologies such as Twitter, Facebook, Mentimeter, and Padlet.

Dron (2014) suggests that online instruction will grow to include “emerging systems and their capabilities for assembly and integration” that allow for a “depth of sophistication that we have not seen before.”

I love that phrase - a depth of sophistication we have not seen before - this depth of sophistication is what we are after as we explore authentic assessments.

Dr. J

Dron, J. (2014).  Innovation and change: Changing how we change. In Zawacki-Richter, O. & Anderson, T. (Eds.), Online Distance Education (pp. 237-265). Athabasca University Press.

Duus, H. J. (2009). A socioeconomic approach to the development of e-learning. E-Learning & Education (eleed) Journal, 5. https://eleed.campussource.de/archive/5/1985

Protopsaltis, S., & Baum, S. (2019). Does online education live up to its promise: A look at the evidence and implications for federal policy. George Mason University. https://distance-educator.com/does-online-education-live-up-to-its-promise-a-look-at-the-evidence-and-implications-for-federal-policy/

CIDER Session

I delivered this Centre for Innovation in Distance Education Rsearch (CIDER) session on April 6, 2022. It’s me reading the highlights of my dissertation in 45 minutes.

The Multiple Realities of Professional Development for Online Contingent Faculty in Canadian Strategy and Practice

The growth of contingent faculty and the growth of online education over the first two decades of the 21st century have generated an emergent but overlooked subgroup of faculty – online contingent faculty. These twin dynamics have placed the professional development of online faculty in a strategically important position for Canadian postsecondary institutions to enhance online instructional effectiveness and mature online educational quality. This presentation outlines how a two-phase multimethod research study employed Ursula Franklin’s technology as practice to explore the following research questions: How are online faculty and their professional development represented in current Canadian postsecondary academic plans? How are the professional development needs of contingent online faculty being served by Canadian teaching and learning centres?

Phase one consists of a document analysis of 17 academic plans from Canadian colleges and institutes covering the current period and immediate future to reveal how faculty development is described and prioritized in academic strategy (the projected reality of the future). The document analysis highlights important strategic purposes of professional development, such as Indigenization and internationalization, but also shows that part-time and online faculty are marginally represented. Email interviews with 12 directors of Canadian teaching and learning centres comprise phase two (the extended reality of experience) and illuminate the contested space of providing educational development services to online contingent faculty. The findings reveal formidable barriers to providing professional development opportunities to part-time faculty who teach online, but also innovative solutions to meet the needs of part-time online educators in Canada.

The expanding possibilities of professional development

The growth of contingent faculty and the growth of online education over the first two decades of the 21st century have generated an emergent but overlooked subgroup of faculty – online contingent faculty. Exposure to part-time instructors and participation in online education can both negatively impact student success, and these twin dynamics place the professional development of online contingent faculty in a strategically important position for Canadian postsecondary institutions to enhance online instructional effectiveness and mature online educational quality. This presentation details the analysis of email interviews with 12 directors of Canadian teaching and learning centres to illuminate the enduring barriers of providing educational development to online contingent faculty, including unequal participation in online education, a lack of programs targeted at part-time instructors, especially in smaller institutions, and that part-time faculty are usually uncompensated when participating in professional development. Despite these significant barriers, the interviews also show that the COVID-19 pandemic spurred innovations to meet the needs of part-time online educators in Canada. Teaching and learning centres are striving to meet the unique needs of part-time faculty with emerging programs that build a comprehensive teaching identity through faculty secondments and mentoring programs. These expanding possibilities for professional development must account for the unique needs of online contingent faculty, the frequent conflict between technology and pedagogy, and the highly political nature of quality assurance in online education. The presentation concludes by sketching a potentially darker future where professional development is minimized by unbundling, outsourcing, and the rise of microprograms.

https://youtu.be/2QGquduDigA

Openo, J. (2021, October). The expanding possibilities of professional development for online contingent faculty in Canada. [Conference session]. Athabasca University 2021 Graduate Student Research Conference, Edmonton, AB.

My small part in the big gamble

Corporate logo for the Organization for the Prohibition of Chemical Weapons

Growing up in Bay City, Michigan, I never dreamed that at some point in my life, I would be an e-learning expert given the chance to work with a bunch of super smart people from around the world who were dedicating part of their lives to the mission of destroying some of the most inhumane weapons ever invented by man. This isn’t as much a failure of my imagination as it is recognition that, as a child, e-learning hadn’t been invented, I had no idea such organizations existed, and I had no conception of just how blessed I would be in my life.

Today, to be a small part of a Nobel Peace Prize-winning organization is more than I could have hoped for. In the picture below, you can see my excitement (I had to wake up at 4:45am to attend the meeting taking place at 1pm Central European Time.

The role of the Advisory Board on Education and Outreach (ABEO) is to provide specialized advice in terms of education and outreach, the development of e-learning strategies, and ensuring these activities are carried out in a coordinated and cost-effective manner.

The United Nations Study on Disarmament and Non-proliferation (2002) education begins by stating:

There has never been a greater need for education in the areas of disarmament and non-proliferation, especially with regard to weapons of mass destruction, but also in the field of small arms and international terrorism…The overall objective of disarmament and non-proliferation education and training is to impart knowledge and skills to individuals to empower them to make their contribution, as national and world citizens, to the achievement of concrete disarmament and non-proliferation measures and the ultimate goal of general and complete disarmament under effective international control.

As noted in the report,

Science and technology transformed the world in the twentieth century. Living standards improved but warfare was rendered more deadly. Weapons of mass destruction -biological, chemical and nuclear - and their means of delivery were developed, as ever more sophisticated conventional armaments were produced and disseminated. The horrors and destruction of armed conflict persist.

And this is Jacques Ellul’s point, exactly, in The Technological Society - we cannot have the good without the bad. As I write this, The Associate Press has reported, “Russia plans to give its nuclear weapons apparatus a practice run this weekend.”

The cynic in my says that education stands little chance in the face of evil, but I have long embraced Camus’s point that if he who bases his hopes on human nature is a fool, he who gives up in the face of circumstances in a coward. To be part of the OPCW’s ABEO is part of that formidable gamble that words are more powerful than munitions. I’ve never been much of a gambler, and when I have bet, I have always lost, but this where I have laid my chips:

High- and low-technology teaching tools and techniques for conveying content, stimulating interest and evoking emotion such as videos, animation, electronic games, theatre, dance, films and the graphic arts such as photography are useful and effective means for presenting the subject of disarmament and non-proliferation as well as concepts such as tolerance, democracy and conflict resolution. (section 27)

Dr. Richard Guthrie, another member of OPCW’s Advisory Board for Education & Outreach, assembled this FAQ regarding chemical weapons and the Ukraine: https://www.cbw-events.org.uk/faqs-ukraine.html

Thanks also to Richard for pointing out the important publications below:

Two publications released last week in the UK may be of interest:

One is a brief report from the House of Commons Library on: "Ukraine: Fears Russia could use chemical weapons" (available at https://commonslibrary.parliament.uk/ukraine-fears-russia-could-use-chemical-weapons/).

The Library is a very useful source of non-party political advice in the British Parliament.  The report has a little bit of a mix up between chemical and biological issues but is useful, nonetheless.

The other is a report from the Royal Society of Chemistry on racial and ethnic inequalities in the chemical sciences (available via https://www.rsc.org/news-events/articles/2022/mar/new-1.5m-unit-to-tackle-racial-inequalities-in-science/).

Pages 39-40 of the report discuss generating a global chemistry community and so may be of particular note to Board members.  This report follows on from the sense of belonging report published last year (https://www.rsc.org/new-perspectives/talent/belonging-in-the-chemical-sciences/).

 

I hope these links are of use.

Is it a micro-credential?

https://www.mhc.ab.ca/NewsandEvents/Stories/2021/December/MHPS-PatrolSupervisorMicro-credential

In early December, the team and I had the great fortune to work with the Medicine Hat Police Service on MHC’s first micro-credential, a Patrol Supervisor.

The micro-credential emerged because an important element of Medicine Hat Police Service’s (MHPS) Strategic Business Plan 2019-2022 is developing a Patrol Supervisor’s Course that will provide the opportunity for constables interested in acting as a sergeants. The 35-hour learning opportunity was offered in a blended learning format consisting of 14-hours of online training that delivered core course content (such as policies and procedures), complemented by 21-hours of in-person training that maximized interactivity and application of content to real-life police situations (such as high-speed pursuits).

MHPS employs a competency-based management (CBM) approach that focuses on the competencies – the skills, knowledge, or attributes – of successful performance. The focus on CBM matches both the essence of microcredentials and the goal of the Patrol Supervisor Course. The Alberta Microcredential Framework defines a micro-credential as a credential awarded upon the successful completion of an assessment of a particular skill or competency, or cluster of skills and competencies, associated with specific and focused training, and are designed to be beneficial in obtaining employment or meeting on-the-job requirements. But is it really a micro-credential?

The story linked to above details more about the course, but in this short piece, I want to reflect on the growing confusion of what a micro-credential is (and isn’t), and this growing confusion is in my own head. I used to think I knew what a micro-credential is, but I’m not so sure anymore.

Recently eCampus Ontario released Micro-credential Principles and Framework, and the MHPS Patrol Supervisor checks a lot of the boxes. It’s issued by MHC, in collaboration with MHPS, and the competency outcomes are aligned to the competencies of patrol supervisors. It’s transcriptable and endorsed by a partner. But if it falls down anywhere, it is in the Summative Assessment of the micro-credential, which is one of the defining characteristics of the micro-credential - assessment and evidence of learning. Micro-credentials require evidence of achievement of outcomes.

One module within the Patrol Supervisor course is a review of the Moncton shootings. This haunting documentary shows numerous occasions when a patrol supervisor could have made a different decision (such as setting up a containment grid or getting wounded officers to the hospital). By debriefing episodes such as Moncton and asking “What if?” or “What other options were available?” there is hope that if these officers found themselves in similarly chaotic situations, they would make better decisions. There is no way to acquire evidence of the impact of this training. The ultimate hoped-for outcome of the training is that, in the heat of the moment, patrol leaders will make good or better decisions.

In short, the decision-making competency of the patrol supervisor will only be verified in an incident we pray never happens.

Digital Toolcrib - Season 2

After two Digital Toolcrib episodes, Chad and I let this fall off the side of our desks and then people actually asked us when we were going to do more episodes! It’s a smash hit. In response to these 2-3 viewers, Chad and I have recorded three additional episodes of the Digital Toolcrib and two of them feature special guests which makes them a lot more engaging, exciting, and practical.

Barriers Facing Trades & Educational Development

https://youtu.be/dpvWzUkDkwE 

Chad and Jason talk candidly about how the difficulties facing trades education and some of the barriers involved in providing educational development to trades instructors, including how the word pedagogy frightens the horses and various other forms of misunderstanding and intimidation. The dialogue hopefully identifies some ways to break down these barriers so trades faculty and educational developers can talk the same language of student success.

Reflective Journaling in Carpentry?

https://youtu.be/6WW-wk7gwz4

Medicine Hat College’s carpentry instructor Paul Schaan discusses why and how he implemented reflected journaling in Carpentry, including how it teaches the whole person and increases connection between faculty and students.

Backwards Design in the Trades

https://youtu.be/k72vCpclCoM

In this episode, Chad Flinn and Jason Openo talk with Dr. Sally Vinden of Vancouver Island University. Sally is lead author of Strengthening Digital Teaching & Learning for Trades, Vocational, Education and Training Practitioners, and she received BCcampus’s Award for Excellence in Open Education (2020). The episode discusses how backwards design can be used to ensure competencies are taught and assessed within trades contexts.

Multiple realities: Professional development for online contingent faculty in Canadian strategy and practice.

Openo, J. (2021). Multiple realities: Professional development for online contingent faculty in Canadian strategy and practice [Doctoral dissertation, Athabasca University]. http://hdl.handle.net/10791/360

The growth of contingent faculty and the growth of online education over the first two decades of the 21st century have generated an emergent but overlooked subgroup of faculty – online contingent faculty. These twin dynamics have placed the professional development of online faculty in a strategically important position for Canadian postsecondary institutions to mature online education and enhance instructional effectiveness. This two-phase multimethod research study employs Ursula Franklin’s technology as practice (1990) as its theoretical orientation to explore the following research questions: How are online faculty and their professional development represented in current Canadian postsecondary academic plans? How are the professional development needs of contingent online faculty being served by Canadian teaching and learning centres? What gaps, if any, exist between the projected reality of academic plans and the extended reality of teaching and learning centres in Canada? Phase one consists of a document analysis of 17 academic plans from Canadian colleges and institutes covering the current period and immediate future to reveal how faculty development is described and prioritized in academic strategy (the projected reality of the future). The document analysis highlights important strategic purposes of professional development, such as Indigenization and internationalization, but also shows that part-time and online faculty are marginally represented. Email interviews with 12 directors of Canadian teaching and learning centres comprise phase two (the extended reality of experience), and they illuminate the contested space of providing educational development services to online contingent faculty. The findings reveal formidable barriers to providing professional development opportunities to part-time faculty who teach online, but also innovative solutions to meet the needs of part-time online educators in Canada.

The document analysis of academic plans shows that professional development for online instruction was a neglected topic pre-pandemic, and the email interviews demonstrate that professional development for online instruction became the central, all-consuming task for educational developers, spurring unprecedented creativity and innovation. But it also shows that part-time faculty and their unique needs were again lost in the mix. Part-time faculty have been called indispensable but invisible, and part-time online instructors have been dubbed the doubly invisible. If it is actually possible for a group of people to be triply invisible, the pandemic added this layer of invisibility because it was difficult to determine how much attention was paid specifically to contingent faculty who teach online and their unique conditions.  

This work attempts to dissipate this fog by grounding its orientation in Ursula Franklin’s The Real World of Technology (1990), where Franklin defines reality as “the experience of ordinary people in everyday life” (p. 36). The professional development for part-time online instructors is not something I explore from a distance. I know this challenge intimately; this has been the nitty gritty of my day-to-day life for the past several years. My efforts to make sense of this tricky terrain have been guided by Franklin’s concerns about how technology affects the quality of our lives, and I hope this work embodies her spirit to solve problems and make the world a better place by employing her concept of redemptive technologies that can arise during a convoluted and tumultuous time such as this one.