Moving toward student-faculty partnership in systems-level assessment: A qualitative analysis

Partnership models have been effective across many areas of higher education such as involving students as teaching and learning consultants, in course design and redesign, and as co-instructors. However, there are few systems-level (i.e., entire programs or institutions) examples of partnership work and virtually none in systems-level assessment. Systems-level assessment models, such as program-level assessment in the United States, are used to inform broad changes to academic programs. Thus, student input may be crucial. This study sought to explore the broad factors that underlie potential student-faculty partnership efforts in systems-level assessment. Participants were faculty and staff members based in the United States and the United Kingdom who engaged in student-faculty partnerships at the program and/or classroom level. Qualitative coding and analyses of interviews with participants resulted in seven primary themes. This study examines patterns evident in student-faculty partnership work across several areas of higher education and begins to lay the foundation for a theory of student-faculty partnership in systems-level assessment.

1. Provided needed time and resource support to the assessment coordinator; 2. Built student skill sets in research, data-analysis, and assessment; and 3. Increased the value students see in the assessment process.
This work is perhaps the most direct attempt at student-faculty partnership in the program-assessment process to date. While these examples provide compelling and noteworthy anecdotes, the major issue is that few, if any, rely on a common theory of why and how partnerships in this arena are supposed to work.

Focus of this research
Without existing evidence of its effectiveness, it has been difficult to convince both assessment professionals and teaching faculty of the need to involve students. Even with a handful of compelling examples, it is often difficult to convince necessary stakeholders to engage in such a different way of thinking about the role of students. Thus, a strong theory of how and why student-faculty partnerships could work is necessary. Without theories of student-faculty partnership in higher education program-level assessment, it is not possible to accumulate empirical support for student-faculty partnership work. Thus, this study explored patterns evident in student-faculty partnership work in other areas of higher education in order to begin the process of developing such a theory. Considering what information was likely necessary to convince relevant stakeholders, the researchers chose initially to seek answers to the following: 1. Who are the people in higher education engaging in student-faculty partnership work? 2. How do people become interested in student-faculty partnership work? 3. What do experts identify as the benefits to partnering with students? 4. What do experts identify as the challenges to partnering with students? 5. Is there a common developmental pattern to student-faculty partnership work? 6. What work is being done at the classroom level that might be scaled up? 7. What work is being done at the program-level that might be adapted to assessment work? 8. Do experts in student-faculty partnership think that student-faculty partnership in program-level assessment is possible?
While these questions helped guide the creation of the initial interview materials, the qualitative nature of this work allowed exploration of additional lines of inquiry and the flexibility to account for unexpected information.

Participants
Participants were faculty and staff members based in the United States and the United Kingdom who engaged in student-faculty partnerships at the program and/or classroom level. . Moving toward student-faculty partnership in systems-level assessment: A qualitative analysis. International Journal for Students as Partners, 5(1). https://doi.org/10.15173/ijsap.v5i1.4204 61 In order to ensure confidentiality and the anonymity of participants in this study, no institutions or organizations will be named. Participants were identified using purposeful sampling (Creswell & Clark, 2007), a process of identifying participants based on their specific area of expertise. Literature was collected through various educational and psychological research databases, online searches using various search engines, and discussions with researchers in the realm of student partnership. Prominent authors in the U.S. and U.K. were contacted by the lead author and asked to participate in the study.
Once the initial interviews were concluded, the researchers did not yet have evidence of saturation. Saturation in qualitative research occurs when new data, themes, or coding are unlikely given a new participant (Fusch & Ness, 2015). Saturation also indicates that it is possible to replicate these data, themes, or coding with new participants. Saturation, not sample size, is the most relevant component to a strong claim in qualitative research.
To achieve saturation (Fusch & Ness, 2015), additional participants were identified through theoretical sampling. Using guidance from our initial interviews and additional literature, we reached out to specific additional experts who were likely to provide perspectives only on the topics needed for saturation. Thus, initial participants were published authors of systems-level assessment and student-faculty partnership work, and subsequent participants were those with specific applied experiences. The final work included in-depth interviews with 15 participants, nine based in the U.K. and six in the U.S. All 15 participants had teaching experience in higher education, had engaged in educational development work, and had engaged in student partnership work at some level. Twelve participants were currently engaged in educational development for other teachers and nine participants had occupied leadership roles in higher education (e.g., pro-vice chancellor, department head etc.). All of the U.S. participants were at least somewhat familiar with program-level assessment while none of the U.K. participants were familiar with the formal process. On average, the participants had spent about 25 years working in higher education.
In addition to seeking saturation of the data, and consistent with best practices in qualitative research, we sought to enhance the trustworthiness of these data through six triangulation strategies (Williams & Morrow, 2009) including transparency, inclusion of a diverse set of viewpoints, multiple analytic checks, bracketing biases and assumptions of the authors, writing memos, and the use of direct participant quotes.

Measures
A semi-structured interview protocol (see Appendix A) was employed to align with the research questions while remaining somewhat informal to allow participants the license to discuss topics in addition to those listed in the protocol. Questions for participants were developed following the applied methods outlined by Kruger and Casey (2014) and Creswell and Clark (2007). Data analysis focused on a combination of field notes and interview transcripts.

Data analysis
All interview transcripts and field notes were transcribed into digital text files. The digital files were then coded and analyzed by the researcher in accordance with grounded . Moving toward student-faculty partnership in systems-level assessment: A qualitative analysis. International Journal for Students as Partners, 5(1). https://doi.org/10.15173/ijsap.v5i1.4204 62 theory methods (Corbin & Strauss, 2008) using NVivo qualitative analysis software (2017, version 11.4.2). A second researcher repeated the coding process for a random sample of the full data set. Data were coded using open, line-by-line coding to identify concepts, themes, and possible meanings of these data (Strauss & Corbin, 1994;Charmaz, 2006). In the second stage, focused coding was applied to the concepts identified in the open coding stage. Each process (i.e., data collection, open coding, and focused coding) informed the other.

RESULTS AND DISCUSSION
Fifteen experienced higher education professionals provided more than 20 hours of interview data. Evidence of saturation was achieved as all participants addressed all themes and participants began to both support each other's comments and provide repeated information. Using focused coding, the 6,258 lines of open codes were consolidated into 191 secondary-level themes and then into 11 primary themes through further focused coding: 1. Examples of partnership 2. The benefits of partnership 3. The challenges of partnership 4. Interest in the benefits and challenges of program-level work in general 5. How to move partnership work forward 6. Similarities and differences between the US and UK systems of higher education 7. Roles and jobs of those who practice partnership 8. How other higher education practices and policies affect partnership 9. Program-level partnership ideas 10. Partnership has many meanings 11. Inspirations for engaging in partnership Next, researchers explored how these themes related to the original research questions.
Who are the people in higher education engaging in student-faculty partnership work? The primary theme, "roles and jobs," is most directly related to this question. The majority of participants worked, at least in part, as educational developers. That those who engage in partnership also engage in educational development makes sense as the role of educational development is to help improve student learning. It is logical that these individuals would seek student perspectives on such learning. In fact, student-faculty partnership work may encourage faculty to move toward educational development roles. As one respondent noted: [As I was doing more partnership] I was moving into an educational development role; I was always integrating it [student-faculty partnership] into educational development, and as I was doing that, I was stepping in some ways away from the classroom and my own teaching and more into the educational development.
All of the experts in the sample were or had been teachers in higher education. Not surprisingly, many of the codes described specific work as a higher education teacher, and the . Moving toward student-faculty partnership in systems-level assessment: A qualitative analysis. International Journal for Students as Partners, 5(1). https://doi.org/10.15173/ijsap.v5i1.4204 63 disciplinary focus of teaching varied (e.g., chemistry, education, communication). Studentfaculty partnerships have the potential to fill in some of the pedagogical training gaps for faculty members. One respondent noted about a first teaching opportunity: I was getting my PhD . . . and I think because of that I got invited by a local university to teach a graduate course on college teaching. But, I didn't know anything about teaching and I certainly didn't know about all of these different disciplines that were different than my own; so, I worked directly with the students to figure out what was best to do.
Most respondents also mentioned having an active research agenda in student-faculty partnership. Participants largely agreed that though the practice of student-faculty partnership can be a driving force toward more positive outcomes, engaging in partnerships without a solid foundation of theory and research risks the long-term success and dissemination of such work. Respondents agreed that theory-building was necessary for the continued evolution of studentfaculty partnership efforts. One participant explained: "I'm also supposed to remain research active, which is great because most of my research is around the co-creation of learning and teaching; that's the term I use, but I talk also about student partners." Additional roles held by study participants included leaders in higher education, developers of student-faculty partnership efforts, consultants to other higher education institutions, researchers, and professionals in assessment, educational technology, and student support. Student-faculty partnership work in program-level assessment should seek to involve as many stakeholders as possible as each brings a unique perspective and set of knowledge and skills to such efforts.

How do people become interested in student-faculty partnership work?
The most prevalent common catalysts for respondents in this study were their own formative educational experiences. Across all respondents, there seemed to be a common underlying theme of falling into student-faculty partnership work by happenstance. The happenstance, though, was always focused on a good educational experience where student perspectives and agency were valued, as demonstrated by a participant: I think I also had, as an undergraduate and then as a graduate student. . . . I had teachers who really thought a lot about teaching, so they were really good models for how to be in a classroom and a university. I didn't have many lecturers who used the banking models or who stood up and talked at us.
Respondents also noted that their willingness to engage in student-faculty partnership efforts was enhanced by working with like-minded people. Support from those at one's own institution and from outside one's own institution can be instrumental in the success of a new project. A participant gave an example: at that meeting, I met [name redacted], who was also getting really interested in partnership and student engagement work. So, I think it was probably the early days Curtis with those enthusiasts together who were thinking what might be seen as sort of quite radically about some fairly interesting topics.
Additional inspirations for partnership work include trying to improve one's own teaching and learning, reading research literature on student-faculty partnership, a personal desire to engage students more meaningfully, and initially engaging in partnership due to the expectations of others. Less prominent themes in this area and each subsequent area are listed in Appendix B.
These findings point to the need to deliberately develop future educators into studentfaculty partnership practitioners by purposefully providing educational experiences, including program-level assessment, that are grounded in partnership. Respondents also tended to agree with the old adage, "success breeds success." Respondents noted that successful first efforts in partnership led to additional, often more complicated and broad efforts.

What do experts identify as the benefits to partnering with students?
The emergent theme, "the benefits of partnership," is most directly related to this question. Building on the benefits of partnership work in the literature, respondents were adamant that student-faculty partnerships improve the teaching, learning, and assessment process generally, no matter the level at which such efforts are applied. Several respondents stated this sentiment directly. For example, one participant stated, "I don't think we can afford not to engage them for the quality [of teaching, learning, and assessment]. If you want quality, they've [students] got to be in there." Drawing on their own experiences partnering with students, respondents also consistently highlighted students' ability to uncover new, previously inaccessible information. One participant noted: In addition to students and faculty members, student-faculty partnerships have the potential to reveal new information to other stakeholders, such as administrators. But I think what really shocked them [upper administration] was [that] the findings were quite different from what they thought.
Respondents shared various examples of a common idea that students and faculty members irreversibly change their approach to higher education teaching, learning, and assessment once they have engaged in student-faculty partnership efforts. This idea is captured by one participant: "It's a challenge but it's sort of a result of the benefits, which is that after faculty and students have participated in partnership it's really hard for them to go back to not being in those kinds of relationships." What respondents are describing in these conversations is student-faculty partnership as a threshold concept. A threshold concept is a term that describes a concept or perspective that, once learned, reveals novel things not previously available to the learner (Meyer et al., 2010). Cook-Sather (2014) described the same belief that student-faculty partnership work is a threshold concept in teaching, learning, assessment, and educational development work. If student-faculty partnership in in the classroom is indeed a threshold concept and the status of partnership efforts as a threshold concept holds at the program-level, then once stakeholders engage in student-faculty partnership in program-level assessment they would not be able to return to their previous ways of thinking and acting. Once this shift occurs for a critical number of stakeholders, partnership in program-level assessment could become the normative practice.
Student-faculty partnership is promising for more than its potential to serve as a recruiting tool. If student-faculty partnership can improve teaching, learning, and assessment practices, widespread implementation of such practices in systems-level assessment practice will result in the betterment of student learning throughout higher education. Additional benefits include developing important knowledge, skills, and attitudes (e.g., sense of belonging, confidence, research skills, metacognitive awareness, motivation, agency in learning) among students and allowing different perspectives to converge to provide a better understanding of higher education.

What do experts identify as the challenges to partnering with students?
The emergent theme, "the challenges of partnership," is most directly related to this question. The most commonly referenced challenges to student-faculty partnership work in this study were perceived conflicts with the current systems of higher education, as one respondent explained: "I mean, it really threatens the whole system that keeps everybody in their place and keeps everything functioning as it has-not that that's been the best idea." Respondents also discussed the system challenges at the program development and redesign level. For example, one participant shared: You know, if you've got a blank canvas, you're fine, but you never-even when it looks like you've got a blank canvas because you're going through a reevaluation so the program can be completely refreshed, there's never really that much time to do it.
In general, respondents believed that the current system of higher education (e.g., class structures, rewards, program structures, etc.) is not set up to support partnership efforts. Instead, respondents felt that these systems sometimes actively work against partnership efforts. In order to successfully build a widespread system of student-faculty partnership, we must consider how to work with or change the systemic factors working against partnership.
Another commonly identified challenge to partnership work was navigating the time, resources, and logistic support necessary for success. An example given by one participant: There is also a challenge in the current climate which is about funding costs and institutions saying they have less and less resources to fund the sorts of things while they are being recognized as . . . more and more important. So it's hard to know how to run these programs without it having resource implications in terms of staffing.
By definition, those faculty members working in true partnership with students recognize the power, knowledge, and experience differential in partnership work. Respondents in this study were sensitive to this challenge as exemplified by a participant: "The fact is they Curtis are students, they have insights but they don't . . . they haven't engaged with the literature on learning and teaching, their life experience is reasonably short, most of them." However, even given their candid view on the power dynamic, respondents went on to state that knowing the state of affairs and accepting them without question are separate things. A participant posited: "[We have to] try to find authentic ways of sharing power. Knowing that we are never going to have a completely equal sharing of power in the classroom because that's impossible." Additional challenges identified include difficulty of achieving representativeness in a partnership effort, convincing others that partnership is worth doing and making faculty uncomfortable and vulnerable to criticism. One respondent described their experience trying to convince a department to engage in research using student-faculty partnership after a previous person had implemented such an effort poorly: Because I remember-doing some of the follow-up and trying to persuade the health centre to take part in research that was going to work in a more valid way again-as partners, when they are really defensive, . . . they wouldn't go for it.
Finally, respondents advised caution in exploring student-faculty partnerships in the realm of assessment. As an example, one participant stated: "[Assessment in the classroom is one of the] most highly guarded and protected aspects of higher education and one of the last holdouts of sole faculty ownership." In order to develop meaningful student-faculty partnerships, we will need to acknowledge and manage the challenges faced by all stakeholders in balancing and redistributing the power around teaching, learning, and assessment practices. We will also need to identify what time, resources, and logistic support is necessary to engage in meaningful student-faculty partnership work. If we want to shift large-scale assessment practices to a partnership model, we will need to tackle the problem of convincing others to engage. We will need to build both an evidence-and logic-based argument alongside an emotional one. Even after initially convincing others, we will need to recognize that we may still need to actively keep stakeholders engaged. Thus, we need to ensure that our initial efforts succeed, thereby avoiding negative associations with failed efforts and ensuring that partnership efforts are set up for success through careful planning and strategy.

Is there a common developmental pattern to student-faculty partnership work?
Within the emergent theme, "the benefits of partnership," there is a subtheme, "partnership leader changes," that is most directly related to this research question. Respondents described much of their work in partnership as moving from focusing on their individual practices to focusing on influencing the practices of others. It is possible that, for the majority of respondents, the move to have a greater influence on others would have happened regardless of partnership efforts. However, if supporters of partnership continue to move into leadership positions, they may be able to leverage such influence to support partnership efforts more widely. Curtis Respondents also believed that most people start out in partnership efforts without much knowledge in how to successfully engage in such efforts. One participant shared: "I didn't know what I was doing when I started, so I've figured out how to do this through experience and working with them [students]." These observations by respondents are both positive and negative. One the positive side, these comments indicate that faculty and students with little to no evidence or support will attempt partnership in their own practice. On the negative side, if partnership efforts are not set up based on what we suspect works based on prior research, theory, and practice, they are more likely to fail.
Additional codes referenced the improvements over time of the training and preparation of others to engage in partnership work and that participant's reasons for engaging in partnership changed over time.

What work is being done at the classroom level that might be scaled up?
The emergent theme, "examples of partnership," is most directly related to this question. Many of the subcategories in this theme are related to specific examples of partnership efforts. To build a system of student-faculty partnerships in program-level student outcomes assessment, we can model practices on those that have been successful at the classroom level.
A number of respondents discussed student-as-consultant programs as an adaption of partnership efforts. These programs involved training students as pedagogical consultants to work directly with faculty members on their classroom-specific practices. When creating a system of student-faculty partnership at the program level, we will need an organizing structure for work across different programs. The structure of the student-as-consultant models, adapted to ensure that true partnership occurs between students and faculty, could serve this purpose well.
Respondents also shared examples of classroom-level partnerships related and unrelated to assessment work, including the co-creation of educational experiences. As one example, a respondent noted that a common in-class feedback tool is start, stop, continue, which they described in the following way: "What would you like me as a tutor to stop doing, what would you like me to start doing that I'm not doing already, and what would you like me to continue doing that's working for you?" Another respondent shared their experience with partnering with students to make classroom-level outcomes meaningful: "I got them to spend 10 minutes just writing a note to themselves about what really motivated them being in the classroom? What really is it they want to get out of it?" Another respondent shared an experience working to redesign an education course in partnership with students: Faculty considered the course to be important, but students didn't like it at all. So instead of guessing why students didn't like it, we brought them into the discussion. Students and faculty worked in partnership to redesign the course description, syllabus, course materials, assignments, and tasks. In these examples, we can begin to develop ideas that might inform practice in the future. It is possible that, for some programs, program-level learning agreements could be developed. Self-and peer-assessment practices could be adapted to the program level. Students can undoubtedly help us develop assessment items that match our intended outcomes and use language that is clear to students. Program-level research on novel assessment methods for outcomes can be conducted in partnership with students and we could find ways of incorporating meaningful formative feedback for students from program-level assessments.
What work is being done at the program-level that might be adapted to assessment work? None of the themes generated during the course of this study relate directly to this question. This does not necessarily mean that program-level partnership is not occurring. Rather, it suggests that program-level partnership is at least rare enough that the expert participants in the current study, both those familiar with partnership and those familiar with program-level assessment, were unaware of the details of any examples of intentional student partnerships in program-level assessment.

Do experts in student-faculty partnership think that student-faculty partnership in program-level assessment is viable?
The emergent theme, "program-level partnership ideas," is most directly related to this question. Broadly, all participants agreed that partnership at the program-level and in programlevel assessment was a good, but challenging idea. After talking through a typical program-level assessment cycle (see James Madison University, 2020, for an example), respondents believed that students could become partners in each of the established processes. One respondent offered a concrete suggestion for how partnership in the objectives might work: Respondents also highlighted the potential for student-faculty partnership efforts in the interpretation and reporting of results. Drawing from their own experience of partnering with students to support classroom assessment results, one respondent shared, I think it had much more of an impact coming directly from students about their actual experiences at the institution than it would if I'd written a report . . . and [it] also helped to get leeway when working with [individuals] who kind of said, "Oh, no. That doesn't happen here," and we could say, "Oh, yes it does, and here's some quotes from students on how they experience it." All participants shared that program-level partnership work interested them and made them think more about program-level teaching, learning, and assessment. Though not surprising, it was unfortunate that participants were unaware of any partnership work in program-level assessment processes. A few participants referenced the idea of a student-asconsultant program that could function at the program-level. These ideas and examples highlight ways that we might embed student-faculty partnership within our current systems of assessment. Yet, the best ideas generated from only our own perspectives are not likely to match the impact of those we develop in partnership with our students. We should consider these ideas as seeds for dialogue and potential prototype projects with our student partners as we work to develop even more meaningful and impactful ideas in partnership with students.

LIMITATIONS
We accept that there are several limitations to the inferences made in this study. First, the participants based in the U.K., while unquestionably experts in student-faculty partnership, were unfamiliar with the U.S. version of program-level assessment. Future research should seek to include U.K.-based partnership experts that are familiar with program-level assessment. Second, all participants in this study were current or former faculty members; no students served specifically as partners on this research. Third, while we made attempts to balance participants across countries, the resulting information was ultimately balanced slightly in favor of U.K.-based experts. Finally, the original intent of this study was to develop an empirical theory of student-faculty partnership in program-level assessment. In other words, we aimed to explore why partnership practices should work better than current practices. Yet, as is often the case in qualitative research (Charmaz, 2006), the final scope of this study is different than intended. We believed that partnerships in program-level assessment were rare; however, we were unprepared for the true scarcity of such efforts. We were also unprepared for the U.K. system of heavily modularized assessment practice. These practices also did not align with our own training and experiences in U.S.-based program-level assessment. These two unexpected developments led us to synthesize the collected information to inform future theory development. Curtis

SUMMARY
Analyses of these discussions provide a set of tentative assumptions from which to start building a working theory of how to engage our colleagues in student partnership in systemslevel assessment: 1. Faculty who engage in partnership tend to be those who engage willingly in educational development, have classroom teaching experiences, and are interested in the scholarship of teaching. Faculty who gravitate to leadership roles may be more willing to engage in partnership. 2. Support from others appears to be crucial as does experiencing early success in partnership efforts. 3. Partnership opportunities may be best received in the context of helping to improve the teaching, learning, and assessment processes rather than individual benefits to faculty. It may also be helpful to highlight the ability to obtain novel information. 4. There are challenges to partnership that should be acknowledged and, when possible, mitigated. 5. There tends to be a trial-and-error feel to partnership efforts. With the growing scholarship of partnership, it will be beneficial to ground future efforts in the literature. 6. If teachers and administrators can understand both systems-level assessment and student partnership separately, the benefits of combining the two appear to be straightforward.
In conclusion, all participants agreed that integrating students as partners in program or institutional assessment processes was both interesting and spurred them to think more about program-level teaching, learning, and assessment. However, even participants who were longaware of program-level assessment practices in general were largely unaware of current partnership work in assessment at the program-level. Thus, this study examined patterns evident in student-faculty partnership work across several areas of higher education and considers how those patterns might inform our future work. We hope that this article provides a solid foundation for both building a strong theory of partnership in systems-level assessment and also in informing future practice.

Semi-structured interview questions
A) Brief introduction and purpose of interviews a. Summary of program-level assessment work in the United States b. Faculty-engagement has been successful in improving assessment process and thus we are wondering if student-engagement could help with improving the learning process? c. We are interviewing experts in partnering with students generally to explore the possibilities of partnering with students at the program level. We are also hoping to draw on partnerships with students in other realms such as at the classroom level. B) Permission to Record and Signing Consent Form C) Warm-up 1) Where do you work and what is your job title? 2) What are your responsibilities as __________?
3) How did you become interested in partnering with students? D) Student Partners Questions 4) What work have you done to partner with students? 5) How has your work partnering with students changed over time? 6) What are the benefits of partnering with students? (Students, faculty, institutions, public) 7) What are the challenges in partnering with students? E) Student Partners in Program Assessment 8) Have you done any work partnering with students at a program level? 9) Do you know anyone else who has done work partnering with students at the program level? 10) How (or how else) might we partner with students to improve program assessment and student learning? (Are there opportunities to scale up classroom-level ideas?) F) Conclusion 11) Is there anything else that you would like to talk about around students as partners?

APPENDIX B
Less prominent themes RESEARCH QUESTIONS LESS PROMINENT THEMES Who are the people in higher education engaging in student-faculty partnership work? N/A How do people become interested in student-faculty partnership work?
• wanting to engage faculty members • desire to develop additional skills • logic • to do research • trying to be clear in setting expectations for students