Outcome Measurement in Academic Libraries: Adapting the Project Outcome Model

Eric Ackermann

Radford University, USA

Sara Goek

Association of College & Research Libraries, USA

Emily Plagman

Public Library Association, ALA, USA

I. Introduction

In 2015, the Public Library Association (PLA, a division of the American Library Association) launched Project Outcome, a free online toolkit designed to help public libraries understand and share the impact of their programs and services. It equips libraries with short, easy-to-use, patron-focused surveys designed to measure the impact of library services such as childhood literacy, computer training, and workforce development. The project focuses on measuring the outcomes of the programs that many public libraries have in common, thereby minimizing the work for each library in developing measures while also maximizing impact by giving libraries a shared language and the ability to aggregate data. It also gives libraries the resources and training support needed to apply their results and confidently advocate for their future. This session will begin with an overview of the Project Outcome model and the results of four years of activity and engagement in the public library field.

Expanding upon this successful model, PLA partnered with the Association of College & Research Libraries (ACRL, a division of the American Library Association) to develop a version of Project Outcome for academic libraries. While outcome assessment may already be common in the academic field, the Project Outcome model will offer academic libraries access to a standardized set of outcome measures and a free, easy-to-use toolkit. It includes patron-focused surveys across seven service areas, an online dashboard of interactive tools for collecting and analyzing the data, and practical guidance on using the results. This toolkit will provide academic libraries of any size the means to easily measure the learning outcomes of their programs and services and to use that data as the basis for improvements and advocacy. By collecting similar data nationally across a range of academic and research libraries, ACRL will begin to develop a national picture of the impact of academic libraries, thus better supporting its members and furthering its mission.

ACRL appointed a task force for this work in early 2018 and will launch the new tool in April 2019. The task force field-tested seven new surveys in the areas: undergraduate instruction, events/programs, research support, teaching support, digital collections, library spaces, and library technology. This paper describes the task force’s work to establish standard learning outcome measures for academic libraries, initial field-testing results, and how Project Outcome can create opportunities for growth or change.

With over 1,300 libraries collecting more than 200,000 patron surveys in the system, PLA’s Project Outcome has helped the public library field collectively move towards the use of standardized outcome measures. In adapting this model for academic libraries, ACRL intends to support its members and the academic library field to create momentum towards outcome measurement as common practice.

II. Background: PLA’s Project Outcome

Project Outcome builds upon a 2013 initiative led by then-PLA president Carolyn Anthony (former director, Skokie Public Library), who established a Performance Measurement Task Force to develop standardized measures of effectiveness for library services and promote training and implementation tools for using the data collected. The task force comprised a diverse group of public and state library leaders, consultants, data researchers, and analysts. After collecting outcome data from 2013 Public Library Data Service (PLDS) survey responses and conducting pilot tests in 27 libraries in late 2014, the task force identified the seven essential library service areas for Project Outcome to assess that could be easily and directly linked to improving or changing a patron's knowledge, confidence, behavior change and awareness. These key outcomes are based on the social theory that performance is more adequately measured when capturing the outcomes of knowledge, attitude, and behavior change.

Figure II-1. PLA Project Outcome survey areas

The Project Outcome immediate surveys are six questions long and include both Likert-scale and open-ended questions. The immediate surveys are designed to be distributed immediately after completion of a program or service and they aim to help libraries better understand the impact that a program or service has on patrons and their intention to change behavior as a result. The immediate surveys are ideal for assessing the immediate impact of a program or service, informing program or service changes, and providing a “snapshot” for advocacy and reporting.

The Project Outcome follow-up surveys vary in length and follow a yes/no/please explain format. The follow-up surveys are designed to be used four to eight weeks after completion of a program or service and they aim to help libraries better understand whether patrons have changed their behavior or continued to benefit as a result of the program or service. The follow-up surveys are ideal for assessing the impact of a program or service after a period of time, informing internal planning, measuring progress toward strategic goals, and providing evidence for advocacy.

The task force also developed the outcome measurement guidelines, which are designed to help libraries conduct four key alternative outcome measurement activities and demonstrate long-term, collaborative impact on the community: writing your own outcomes; measuring with alternative methods; measuring outcomes over time; and measuring outcomes collaboratively with partners. Libraries can use the guidance, worksheets, references to other experts, articles on the topic, and case studies to develop more complex outcome measurement models. They can be used in conjunction with or separately from the surveys.

PLA launched the Project Outcome toolkit in June 2015, with financial support from the Bill & Melinda Gates Foundation, and, as of January 2019, it has collected over 200,000 patron responses from over 1,300 public libraries in the US and Canada. The standardized metrics allow for data to be stored and aggregated at the local, state, and national level and displayed on a series of interactive data dashboards.

In order to drive awareness and use of the toolkit, staff, with support from the task force, implemented an outreach campaign that included in-person conference presentations with participating libraries as co-presenters and free monthly webinars. The latter have focused on the ease of implementation and published case studies from library staff highlighting how they were able to measure and use outcome data.

Most importantly, public libraries have successfully used the data collected for a variety of activities, such as: making changes to programs; communicating the value of the library to funders, decision makers or the public; informing or measuring progress on strategic plans; and supporting or engaging partners. Several case studies published on the Project Outcome website illustrate the ways public libraries have used outcome data to make improvements to programs or services (see Appendix 1).

The graph below (Figure II-2) represents responses from 760 libraries and further illustrates that libraries participating in Project Outcome see an overall increase in outcome data use.

Figure II-2. Use of outcome data among participating public libraries

III. ACRL Project Outcome for Academic Libraries

PLA noticed a strong interest in the Project Outcome assessment tools from academic libraries. While the survey management tool and data dashboards are available only to public libraries, the resources and webinars are freely available to anyone who registers on the website. They recorded 101 users who signed up from academic libraries in the United States (N=92), Canada (N=7), and Australia (N=2). Of those that joined from the United States, a cross-section of colleges and universities was represented (see Table III-1 below).

Table III-1. Basic Carnegie Classification of academic librarians who registered on the PLA Project Outcome site, as of October 17, 2017

Basic Carnegie Classification	N	Percentage
Associate’s Colleges	12	13.0%
Baccalaureate Colleges	4	4.3%
Master's Colleges & Universities	25	27.2%
Doctoral Universities	49	53.3%
Tribal Colleges	2	2.2%
Total	92	100.0%

PLA alerted ACRL to the trend and planning began soon thereafter to adapt Project Outcome to the needs of academic libraries. In fall 2017, the ACRL Board of Directors approved a project plan and significant investment to build the tool to fit the needs of academic librarians, resulting in the establishment of the ACRL Project Outcome for Academic Libraries Task Force on November 15, 2017. Its charge is: “to adapt the Project Outcome measures, developed by the Public Library Association, to an academic library context. Collecting consistent outcomes data will allow academic libraries to benchmark at the national and state level and will help ACRL better support its members and mission.” A timeline was established, with the product roll-out scheduled for the ACRL Conference in April 2019. Like the PLA Project Outcome tool, ACRL’s Project Outcome for Academic Libraries will provide tools, resources, and webinars at no cost to the academic library community.

The ACRL Project Outcome Task Force consists of eleven members and three staff members. The members were drawn from a cross-section of academic institutions. Two members are from associate’s colleges, two from baccalaureate colleges, one from a master's college, and five from doctoral universities. One additional member serves as a liaison to the ACRL board, and the three staff members represent ACRL and PLA. The task force met in person in Chicago in March 2018 and has held regular virtual meetings since then as a whole and in smaller working groups.

The goal of their work is to create a series of standardized outcome measures that allow for the evaluation of impact over time. These measures must be relevant across a range of programs and types of academic libraries. Each participating library’s results are aggregated into a dataset that in turn allows comparisons of impact at the national level and by Carnegie classification.

III.1 Standardized Outcome Measures

Project Outcome has defined an outcome as: “a specific benefit that results from a library program or service.” Whether quantitative or qualitative, “outcomes are often expressed as changes that individuals perceive in themselves.”

Based on the theoretical work of Schrader and Lawless, PLA developed four outcome measures: knowledge (learned something new), confidence (feel more self-assured or self-reliant), behavior change or application (will apply or use what you have learned), and awareness (increased cognizance or recognition of resources/services/spaces available). These were operationalized in the Project Outcome immediate surveys as four quantitative (Likert-scale) questions.

In addition, two open-ended qualitative questions were added: “What did the users like the most?” and “What can the library do to improve?” These provide more insight into ways to help users, as well as qualitative data that can be used to demonstrate value and impact to the associated stakeholders.

III.2 Survey Adaptation: Scope and Limits

In adapting PLA’s toolkit for academic libraries, the ACRL task force needed to keep as much of the original structure as possible in order to preserve the theoretical basis of the outcomes and project model, which had been verified through field-testing done by the PLA.

This includes keeping unchanged the:

Maximum number of survey areas (seven),

Maximum number of standard questions in each survey (six),

Order of survey questions, and

Response types: Likert-scale and open-ended comments.

No additional validity or reliability testing was planned or conducted. This reflects the focus of the Project Outcome tool on generating useful actionable new knowledge (practical, local, applied) for improvement, which does not always fully align with the standards of social science research (theoretical, causal, prove/disprove ideas, perfect/ideal).

ACRL purposefully chose to maximize PLA’s investment by using existing technology and software, and to keep survey elements unchanged (as described above). This delimited the scope of the task force’s work. In addition, the data collection design relies on self-reported data. While this does provide valuable insight into the perceptions and values of our users, George Kuh points out that its two main limitations are the respondents’ ability to provide accurate information, especially in potentially awkward or embarrassing positions, and “the potential for students to provide intentionally false or non-representative information.”

Given these parameters, the ACRL task force developed surveys for seven library activity/service areas (see Table III-2 below). Along with the text of the immediate surveys, the task force provided some guidance on the definition and intended use of each survey. (Development of the follow-up surveys is still in progress at the time of writing.)

Table III-2. Project Outcome for Academic Libraries immediate surveys for field testing

Activity/Service Area	Definition & Suggested Uses	Survey Questions
Undergraduate Instruction	Services or programs to assist undergraduate students in their coursework and enhance their learning. Examples: one-shot instruction sessions, library orientation programs.	I learned something new that will help me succeed in my classes. I feel more confident about completing my assignment(s). I intend to apply what I just learned. I am more aware of the library's resources and services. What did you like most about this program/service? What else could the library do to help you succeed in your classes?
Events/Programs	Services to inform, enrich, and promote community learning. Examples: exhibits, lecture series, civic engagement services, stress relief activities.	I learned something new from this event or program. I feel more confident about the topic. I intend to discuss or share with others what I just learned. I am more aware of the library's resources and services. What did you like most about this program/service? What could the library do to help you learn more?
Research Support	Services to improve research skills and help researchers succeed. Examples: reference services, research consultation services, workshops for graduate students or faculty.	I learned something new that will help with my research goals. I feel more confident about my ability to complete my research. I intend to apply what I learned to meet my research needs. I am more aware of the resources and research support the library provides. What did you like most about this program/service? What could the library do to help you with your research?
Teaching Support	Services to help faculty and instructors develop curriculum and/or course materials that will better support student learning. Examples: OER services, online learning modules, faculty teaching workshops.	I learned something new that will be helpful in developing my course(s). I feel more confident about incorporating library resources into my course(s). I will apply the techniques and/or information I learned to future courses. I am more aware of resources and services the library provides. What did you like most about this program/service? What could the library do to help you with your teaching?
Digital Collections	Online collections or resources provided by the library that support classroom instruction or research. Examples: digital exhibits, image databases, online archival collections.	I feel more knowledgeable about the topic supported by the digital resource (collection) I used. I feel more confident about my understanding of the topic supported by the digital resource (collection) I used. I intend to apply what I just learned from the digital resource (collection) I used. I am more aware of digital resources (collections) the library provides. What did you like most about this program/service? What could the library do to help you better use digital resources?
Library Spaces	Specific spaces that are intended to be used to enhance academic performance and support student learning. Examples: group study rooms, individual study spaces, testing labs, writing centers, tutoring centers.	1. This space contributed to my ability to learn something new. 2. Using this space makes me feel more confident about my ability to achieve my goals. 3. I am likely to use this space again in the future. 4. After using this space, I am more aware of the library resources and services available to me. 5. What did you like most about this library space? 6. What could the library do to improve this space?
Library Technology	Services to access technology, build technology-related skills and confidence, and help students incorporate technology into their coursework. Examples: technology loan programs, maker spaces, technology related user “how to” workshops.	I gained experience that contributed to my knowledge of educational technologies. I feel more confident about integrating technology into my assignments or projects. I intend to continue to use these and other technologies for future assignments or projects. I am more aware of technology resources and services the library provides. What did you like most about this program/service? What could the library do to help you better use technology?

IV. Field-Testing Results

Following approval of the provisional survey text (Table III-2) by the task force and the ACRL Board of Directors, a call for volunteers to field-test the new surveys was announced on June 26, 2018. The field-testing and data collection process ran through October 31st. In that time, 108 individuals from 100 institutions signed up to field-test the surveys. When the process completed, 54 colleges and universities had actively participated in data collection, submitting a total of 11,449 responses.

Figure IV-1 below shows all volunteer institutions mapped by location, number of responses, and Carnegie classification. Overall, the responses were well-distributed between types of institutions, with community colleges contributing 27% of the total responses, baccalaureate institutions contributing 27%, master’s granting institutions contributing 21%, doctorate-granting institutions contributing 24%, and special focus institutions contributed the remaining 1% of the total responses.

Figure IV-1. Volunteer institutions and responses

Table IV-1. Response count for each survey and number of institutions submitting responses

Survey	Number of valid responses	Number of institutions
Survey 1: Undergraduate Instruction	9148	40
Survey 2: Events/Programs	307	14
Survey 3: Research Support	204	11
Survey 4: Teaching Support	52	7
Survey 5: Digital Collections	5	1
Survey 6: Library Spaces	1303	15
Survey 7: Library Technology	430	5

For this preliminary study, we analyze in detail only the results from the two surveys with the highest response rates. (See Appendix 2 for links to interactive data visualizations with aggregate results of all the surveys.)

The undergraduate instruction survey proved by far the most popular. The breakdown of institutions using this survey by Carnegie classification closely mirrors that of the active participants as a whole. Feedback from survey administrators showed they used it to assess a variety of instruction types including: instruction sessions in specific classes (in anthropology, art, biology, communication, education, English, ESL, foreign languages, health sciences, music, nursing, political science, psychology, religious studies, and social work); adult education sessions; orientation programs; research-related instruction sessions; citation and reference instruction; and general life skills sessions.

Table IV-2. Outcomes: Undergraduate Instruction

Question	Average score*	N=	Responses on scale/ not blank
Knowledge: I learned something new that will help me succeed in my classes.	4.42	9112	99.6%
Confidence: I feel more confident about completing my assignment(s).	4.24	8743**	95.5%
Application: I intend to apply what I just learned.	4.44	9096	99.4%
Awareness: I am more aware of the library's resources and services.	4.46	9093	99.4%
What did you like most about this program/service?	8569	93.6%
What else could the library do to help you succeed in your classes?	6977	76.2%

* Averages exclude responses given as N/A

** The lower number for this question is due to the fact that one survey administrator (using Qualtrics) did not include this question. Those responses have been assumed as N/A.

The open-ended questions, though not requiring a response, nonetheless had very high response rates. Asked what they liked most about the program/service, respondents most frequently used the words: resources, use, library, learning, research, information, liked, and help. Asked what else the library could do to help them, respondents most frequently used the words: more, nothing, help, library, resources, research, books, and study. Overall, this suggests that users feel positively about the instruction sessions libraries provide, and what they would like most is more programs and services. This aligns with PLA’s findings from public libraries, which suggest that patrons most commonly ask for more: more hours, more programs, and more specific classes.

The second most popular survey was on library spaces. In designing this survey, the task force grappled with whether or how it was possible to use learning outcomes to assess spaces. Rather than asking whether library users like the space, outcomes need to address how the space affects their learning. Field-testers used the survey to assess study spaces, recent renovations, and the reactions of incoming students to library spaces.

Table IV-3. Library Spaces: Outcomes

Question	Average score*	N=	Responses on scale / not blank
Knowledge: This space contributed to my ability to learn something new.	4.32	1272	97.6%
Confidence: Using this space makes me feel more confident about my ability to achieve my goals.	4.27	1291	99.1%
Application: I am likely to use this space again in the future.	4.61	1292	99.2%
Awareness: After using this space, I am more aware of the library resources and services available to me.	4.08	1280	98.2%
What did you like most about this library space?	1102	84.6%
What could the library do to improve this space?	989	75.9%

* Averages exclude responses given as N/A

Among the most frequently used words in responses to the question “What did you like most about this library space?” were: quiet, study, space, work, group, open, room, great, and comfortable. The most frequently used words in response to the question about improvement were: more, nothing, rooms, study, chairs, quiet, tables, better, and outlets. The “more” and “better” they wanted included chairs, outlets, food/coffee, quiet spaces, and larger spaces.

As is apparent from Table IV-1 above, not all the surveys experienced such high levels of engagement. In some cases, such as the teaching support survey, this may stem from the fact that fewer programs or services are offered in that area and they engage fewer users (faculty and graduate students are, relatively speaking, a numerically smaller user base than undergraduates). In other cases, such as the digital collections survey, it may be that the topic area is not as widely relevant. For the final toolkit, the task force has changed that survey from digital collections to “digital and special collections” to give it broader applicability.

We also asked those who volunteered but chose not to administer surveys to also provide feedback on why. Responses, sent via email and an anonymous survey, suggested several reasons. Those included: survey fatigue among library users (making librarians reluctant to administer more surveys); time commitments that cropped up between when they volunteered and when they were going to administer the surveys; the administration did not approve the use of the surveys; they needed more time to figure out how to integrate surveys in their workflow; or they decided that the survey instruments did not match their specific needs.

All field-testing volunteers who did collect results were asked to fill out a feedback form about the process for each unique program or service that they surveyed. That form had 274 responses as of November 6, 2018. 77.3% of respondents said they had no known problems administering the surveys. Those who did experience problems reported that they ran out of time or faced unwilling respondents. Asked, in an open-ended question, how they would use this type of data, the words assess/assessment appear in 20.2% of completed responses, improve/improvement appears in 23.2% of responses, feedback appears in 12.6% of responses, and report/reporting appears in 5.1% of responses. In terms of reactions to the survey questions themselves, feedback was mixed, with some volunteers suggesting the questions fit their needs, while others said there were too few or too many questions, or that they were too general.

Overall, the field-testing results support three key findings:

As is common with Likert-scale surveys, a ceiling effect was expected and observed for the quantitative outcome questions, with average results converging close to 4 (“agree”).

Interest or need for assessment does not always translate into practice. Both survey fatigue and lack of time (users’ or librarians’) can make librarians reluctant to administer surveys, even if the results may be useful.

Feedback from survey administrators suggests that while overall a six-question, one-page survey may be less rigorous than some research-oriented survey tools, even asking users to complete this short survey is a time burden. Therefore, keeping the surveys short and simple is an advantage and will enable wider use of this toolkit.

The task force weighed all the field-testing results and feedback in deciding on the final survey text on December 10, 2018. The seven surveys in the final toolkit are: Instruction, Events/Programs, Research, Teaching Support, Digital & Special Collections, Space, and Library Technology. Project Outcome for Academic Libraries will launch in April 2019.

V. Conclusion: Activating Project Outcome for Library Assessment and Advocacy

Beyond the operational goals of the Project Outcome for Academic Libraries Task Force, the broader purposes are threefold:

First, to help academic libraries of all types measure learning outcomes to capture one expression of their value. This aim is facilitated by the creation of simple, easy to use surveys. Along with the toolkit, ACRL will also develop resources that facilitate greater understanding of outcome measurement and its application.

Second, a goal is to make outcome measurement a common practice among the academic library field. In a higher education landscape increasingly dominated by demands to prove return on investment in order to obtain funding, outcome data can help libraries communicate the value of the programs and services they offer. Libraries can use the results from Project Outcome for advocacy on the local level and to secure additional resources, as PLA has shown. The ability to benchmark results nationally and at the level of institution type will help academic libraries understand how they are doing relative to their peers and provide an incentive to do better. From ACRL’s perspective, the aggregate data can also bolster advocacy efforts that benefit academic libraries overall.

Finally, and perhaps most significantly, libraries can activate their results for improvement. When they do so, it increases the value of engaging in outcome measurement, shifting it from an assessment practice they have to do, to something they want to do because it can result in meaningful positive change.

—Copyright 2019 Eric Ackermann, Sara Goek, and Emily Plagman

Author Biographies

Eric Ackermann

Head, Research Services and Library Assessment

Radford University

Brief author biography text.

Sara Goek

Mellon/ACLS Public Fellow, ACRL Program Manager

Association of College & Research Libraries (ACRL), ALA

Brief author biography text.

Emily Plagman

Project Manager

Public Library Association (PLA), ALA

Brief author biography text.

Bibliography

Association of College and Research Libraries (ACRL). “ACRL Project Outcome for Academic Libraries Task Force: Charge.” 2018.

http://www.ala.org/acrl/aboutacrl/directoryofleadership/taskforces/acr-tfpi.

Kuh, George D. (2002). “The National Survey of Student Engagement: Conceptual Framework and Overview of Psychometric Properties.” NSSE, 2002.

http://nsse.indiana.edu/pdf/psychometric_framework_2002.pdf.

Association of College and Research Libraries. Value of Academic Libraries: A Comprehensive Research Review and Report, prepared by Megan Oakleaf. Chicago: Association of College and Research Libraries, 2010. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf.

Project Outcome. “Year in Review: Annual Report.” 2017. https://www.projectoutcome.org/ckeditor_assets/attachments/359/pla-project-outcome-2-year-annual-report-final.pdf.

Project Outcome. “Final Evaluation Findings.” 2017. https://www.projectoutcome.org/ckeditor_assets/attachments/362/pla_project_outcome_executive_summary_final.pdf.

Schrader, P.G. & Lawless, K. A. “The Knowledge, Attitudes, & Behaviors Approach: How to Evaluate Performance and Learning in Complex Environments.” Performance Improvement 43, no.9 (Sept. 2004): 8–15. https://bit.ly/2OBhiFI.

Upcraft, M. L., & Schuh, J.H. “Assessment vs. Research: Why We Should Care about the Difference.” About Campus 7, no.1 (March–April 2002): 16–20.

https://www.stthomas.edu/media/assessmentaccreditation/Assessment_vs_research_why_we_should_care_about_the_difference.pdf.

Appendix 1: PLA Case Studies

Using Project Outcome with Summer Library Program to Track Impact across Time and Strengthen Championship

Appleton Public Library; Appleton, WI: Appleton Public Library (APL) offers a range of activities for children, teens, and adults as part of their Summer Library Program. In an online component of the program, patrons can earn rewards by completing missions, such as visiting a local nature preserve or the police department. These missions help patrons build connections with community resources and other community members through informal interactions. Project Outcome surveys conducted in 2016 showed that patrons benefited from the program, but found some of the missions confusing and hard to follow. APL has since improved the descriptions and layout of the missions, and will review survey results to assess whether these changes resulted in more patrons taking part in missions.

APL reports on the impacts of library programs and services to its board based on the results of Project Outcome surveys and Impact Surveys. Library leaders have found that outcomes resonate with board members, strengthening how they voice their support for the library. One leader shared, “I think [reporting on outcomes] really transformed conversations with our Board, so that they're much more interested… It lets us tell a more complete story about the library, and what we're offering… so they can understand it more fully… When you talk strictly in numbers, or outputs, it's not something that people can hold on to as clearly. And so when you start talking about outcomes and the impact that a library is having, that's where the true heart of what we're doing is. And our Board feels that, as well… It allows them to… have something more concrete to hold on to, and to talk to people about what the library does in a more complete way than ‘X number of people walk in the door every year’ or ‘X number of books walk out.’” A board member shared, “The [outcome] data provides an objective story, backing up much more engaging stories from staff about serving the community with objective numbers… [and] includes things the Board may not think to ask for. This adds dimensions to how the Board considers the library's success in serving the community.”

Using Project Outcome with Summer Reading and Digital Literacy Programs to Support Partnership Development and Expand Services at a Small Library

Burnsville Public Library; Burnsville, WV: Project Outcome surveys showed that caregivers of young participants in Burnsville Public Library’s (BPL) Summer Reading Program wanted tutoring and extra help for their children. Equipped with this information and evidence of program impact, BPL worked with the local school district to have two teachers offer tutoring at the library the following summer for three days each week. A library staff person whose two children participated in the program shared, “[The children] work on reading, math, they get on the computer… It's really benefiting them. And some of the kids are here because they need the extra help, and some of them are here just to try to beat the summer slide. It's working very well.”

BPL also started a new after-school program because surveys showed an appetite for additional programming for children. Children shared that they wanted to learn more about science and technology, so one of the after-school classes will include hands-on Science, Technology, Engineering, Arts, and Mathematics (STEAM) activities.

Using Project Outcome with Story Time and Teen Programs to Improve Programming and Better Meet Community Needs

Plano Public Library; Plano, TX: After a month of participating in a library story time program (including Babes in Arms, Toddler Time, Rhyme Time, and Preschool), PPL administered Project Outcome’s Early Childhood Literacy survey to gather basic feedback from caregivers and learn if they were experiencing the intended gains in support of children’s basic literacy skills. In addition to providing evidence of positive outcomes, some caregivers reported their children had challenges being in such large groups due to sensory issues. As a result, the library decided to offer “sensory story times” at two of their branches with a smaller group size of 10 caregivers and children. A library leader explained, “We have gotten some incredible feedback from our parents about how we are helping their children by adding this program, and how their children are better able to interact with other children.”

PPL also learned from the surveys that caregivers enjoy programming that includes books in different languages. In response, PPL added a new series called Storytime Around the World, in which library staff read books in Arabic, Chinese, French, and Spanish in three-week cycles. The added programming has attracted new families as well as prior story time participants, and staff observed that families often stay to use other library resources. Excitement about the new series has activated patrons to spread the word. A participant in a Chinese story time shared the event information on a Chinese community Facebook page, which brought in many new people who had not previously known about the library’s Chinese language materials. Consistent with these observations, PPL saw an uptick in circulation of materials in the languages featured in the new series. A staff member explained, “Whether or not… they're new library users, we've definitely seen them connecting with the library and library resources in a way that they haven't before. I think at the Maribelle M. Davis Library where they had the Story Time Around the World series, the circulation of [the books in the Junior World Language collection] doubled within about a year.”

Appendix 2: Data Visualizations

Interactive visualization summarizing the field-testing results: https://public.tableau.com/views/ACRLProjectOutcomeField-Testing/Story?:embed=y&:display_count=yes&publish=yes

Interactive visualization of the field-testing results for the individual surveys, with a breakdown by Carnegie Class for benchmarking: https://public.tableau.com/views/Field-TestingBenchmarks/Benchmarks?:embed=y&:display_count=yes

Endnotes

.	Schrader and Lawless, “The Knowledge, Attitudes, & Behaviors Approach,” 8–15.

.	Project Outcome, “Final Evaluation Findings,” 2.

.	ACRL, “ACRL Project Outcome for Academic Libraries Task Force: Charge.”

.	Project Outcome, “Final Evaluation Findings,” 1.

.	Schrader and Lawless, “The Knowledge, Attitudes, & Behaviors Approach.”

.	Upcraft and Schuh, “Assessment vs. research,” 16–20; ACRL, Value of Academic Libraries, 30–32.

.	Schrader and Lawless, “The Knowledge, Attitudes, & Behaviors Approach,” 9; Kuh, “The National Survey of Student Engagement, 3.

.	Project Outcome, “Year in Review: Annual Report,” 12.

.	Project Outcome, “Final Evaluation Findings,” 3.