Research, evaluation, assessment and strategy in student affairs

Fundamental to any profession are the capacity and willingness to assess and evaluate programmes, learning experiences and services. Student affairs practitioners must demonstrate effectiveness to a variety of constituents including internal stakeholders such as students themselves, faculty and administrators, and external constituents including graduates and those who provide oversight for higher education.

We define assessment as the collection of evidence to determine programme or unit effectiveness, evaluation as the use of assessment data to improve organisational effectiveness, research as inquiry related to the testing of theories and constructs so as to determine the alignment of intended and actual outcomes, and strategic planning as a purposeful design to achieve goals and objectives for the unit.

This paper has been developed to provide an overview of the purposes of assessment and evaluation studies and introduce questions that can frame such studies. It also defines student affairs research and describes how assessment, evaluation and research can be incorporated in strategic planning efforts.

Assessment and evaluation

Student affairs staff members design assessment and evaluation studies that address the following questions: Did participation in an experience enhance student learning? If so, how? If there was student change, was that what was desired? Guiding questions for an assessment strategy include:

• What are the issues at hand?

• What is the purpose of the assessment?

• What information is needed?

• Who should be studied and invited to serve as participants in the project?

• What is the most appropriate methodological approach, ie, quantitative, qualitative or both?

• What instrument(s) will be needed?

• How will the data be analysed?

• How will the results be reported?

• How will the project contribute to organisational improvement?

Some of the data techniques that are commonly used in student affairs and services include: mail, telephone and web-based surveys; focus groups; individual interviews; writing samples including personal journaling, portfolios; benchmarking; cost-benefit analyses and other comparative studies; and more traditional testing.

United Kingdom assessment methodologies have tended to centre on the measurement of student satisfaction and emphasise students as customers‚ rather than attempting to gauge the influence of programmes and services on students as co-creators of their own educational development.

Some of the external quality standards used in the UK – for example Customer Service Excellence – arguably have the effect of strengthening this customer/supplier model. The UK National Student Survey, which is a government requirement for higher education institutions in England and Wales, has used a question set with a narrower scope than instruments such as the US National Survey of Student Engagement.

The core focus in Britain’s National Student Survey has been almost exclusively on a student’s classroom experience, with only limited concern with broader life development within a few questions about confidence in skills such as communication.

An important development internationally has been the focus on assessing student learning outcomes. Of particular interest has been the effort of the OECD to develop comparable assessment standards for measuring specific learning outcomes in higher education.

Assessment of UK student affairs and services has also taken the direction of professional market research. Higher Education institutions typically have well-resourced professional market research departments embedded in their structure, often staffed by professionals whose prior career experience is in the commercial sector.

An ever-growing number of external consultancies offer specialised marketing support to universities and colleges and a range of commercial service providers – in particular in the area of student housing, a growth area in the UK private sector – are active in commissioning and disseminating their own market research findings to inform and influence future service provision.

By way of example, the regular release of student ‘insight’ research published by housing provider UNITE is now a feature of the student affairs landscape in the UK. The Council for Advancement and Support of Education is active in professional development and the provision of related support services in the UK.

In addition, over recent years AMOSSHE – The Student Services Organisation, a key professional body in the field of student affairs in the UK – has developed considerable material to support assessment of student services in terms of its value and impact within the broader student experience.

The South African Survey of Student Engagement, which was based on the NSSE, has been standardised for deployment in South Africa and is used by a variety of higher education institutions to measure student engagement.

By way of a case study, The Listening, Living and Learning programme at Stellenbosch University is focused on senior students. While providing student accommodation, it also offers experiential learning opportunities.

A co-curricular transcript indicating a student’s co-curricular journey for specific accredited programmes at Stellenbosch has been approved and implemented, adding not only value to co-curricular programming but also a refocus on assessment practices.

Research on students

Research, though time-consuming and complex, can be very helpful in providing information that can help frame strategic planning efforts and usually is informed by theory.

Among the most comprehensive books are the three volumes of How College Affects Students written by Ernest T Pascarella, Patrick E Terenzini and their colleagues Matthew J Mayhew, Alyssa N Rockenbach, Nicholas A Bowman, Tricia A Seifert and Gregory C Wolniak, which summarises research on US college students.

European student services agencies are noted for regular social surveys of students. In other parts of the world, these studies are called environmental or student needs studies that should be carried out on a regular basis – at least once every three to four years. The following areas are among the focus of research projects in many countries:

• Trends in student numbers, access, participation rates, demographics.

• Student funding (financial aid), student income, living expenses-spending.

• Educational assistance and support system.

• Choice of academic discipline, student interest patterns, impact of advising.

• Study conditions, time spent on studying, use of leisure time.

• Commuter/married student needs.

• Internationalisation of the student experience, global attitudes of students.

• Impacts of career counselling, health services, social issues on college life.

• Impact of employment on success in higher education.

• Accommodation (housing), eating habits and food services.

Strategic planning

Strategic planning is the process of determining what a student affairs organisation intends to be and how it will get there. It has several elements: a mission statement, a vision statement, a statement of values, goals for student affairs, and objectives that need to be accomplished. At times a SWOT analysis – strengths, weaknesses, opportunities and threats – is conducted to provide a context for the planning process.

Normally, a strategic planning group will analyse the institution’s mission statement and align the student affairs and services mission statement with it. The next step is to examine the vision statement for student affairs that will identify, in general terms, the desired nature of the division of student affairs in five to 10 years. Values inform the work of student affairs an are the next sep in the planning process.

Developing goals are the next activity; goals are informed by the mission, vision and values for the division of student affairs. After goals have been established, the next step is to develop measurable objectives related to the goals.

Assessment provides data to determine what happened as a consequence of student affairs programmes, services and initiatives. Evaluation is undertaken to determine if interventions have the desired effects in the context of the mission, goals and objectives of student affairs and services.

In the UK, for example, the increasingly competitive higher education landscape challenges leaders in student affairs to be ever more focused and strategic in their planning. The key UK professional associations such as AMOSSHE and the Association of University Administrators are contributing significantly to this agenda, with an emphasis on professional behaviours, leadership skills and continued management development.

Conclusion

Research, evaluation, assessment and strategic planning have evolved as central elements in the operations of student affairs and services. They frame thinking about how student affairs ought to be positioned to address challenges of the future, and they provide information that is crucial in meeting the need for transparency.

These activities will continue to be important and we urge student affairs practitioners to continue to engage in research, evaluation, assessment and strategic planning in their professional endeavours.

John H Schuh is Distinguished Professor Emeritus at Iowa State University in the United States. Andrew West is Strategic Advisor at the Association of University Administrators or AUA in the United Kingdom. Munita Dunn-Coetzee is Director of the Centre for Student Counselling and Development at Stellenbosch University in South Africa.