Here is a somewhat dated evaluation done at the San Jose Children's Discovery Museum, but I thought their qualitative methods were interested.
http://www.hfrp.org/out-of-school-time/ost-database-bibliography/database/discovery-youth
In the first evaluation, they did focus groups with youth were they divided the youth into teams of 2-3 and had them fill the in the answers to 4 open-ended questions. Then they brought all the teams together (about 4 total) and had them discuss.
In the second evaluation, they did a group activity with program participants. Participants broke out into groups of 4-5. Then they circulated around the room to large pieces of paper with a different question written on each. Each group wrote answers on the large sheet of paper for each question, so they were able to react to a previous group's comments.
These techniques hit on 2 principles that I think are key to focus groups and interviews. First, make it fun. Vary the questions, get people up and moving, use the 5 senses, present extremes with humor.
Second, give them something to react to. I find I get much more detailed and colorful responses from people if I present something to them first and get their feedback on it. A list, a board full of written-on sticky notes, a photo, a written description. In both of the examples above, participants were responding to what other participants said. A key to that, I believe, is to have people commit to an opinion on paper first. If you jump into reactions right away, then less vocal opinions get lost.
Showing posts with label evaluation report. Show all posts
Showing posts with label evaluation report. Show all posts
Tuesday, July 26, 2011
Tuesday, June 14, 2011
US Impact Study - National Research on the Benefits from Internet Access at Public Libraries
The University of Washington Information School recently released findings from the US IMPACT Public Library Study. This was a large-scale national study, funded by IMLS and the Bill & Melinda Gates Foundation, looking at internet use at public libraries.
The data was gathered from an impressive scale of telephone surveys, online surveys via public library computers and case studies/interviews at a few libraries.
The report lays out broad use statistics and demographics, but combines this with insightful and detailed analysis. It's a thought-provoking read (though time-consuming, unless you opt for the executive summary).
I'm a little wary of some of their findings since part of their data comes from people who are already library computer users (the online survey). Evaluation-speak, this is selecting on the dependent variable, which could introduce bias into the results. They did have a substantial sample from the random phone surveys and it looks like they did some creative mathematical weighting to combine the phone and web samples to reduce bias as much as possible. But it's something to keep in mind.
A few findings I found particularly interesting.
Almost 1/3 of Americans used their public library for internet access.
While there is higher use of library internet connections among people in households living below the poverty line (44%, higher for young adults and seniors), "people of all ages, incomes, races, and levels of education go to the library for Internet access, whether they have a connection at home or not". Libraries are still great agents of democracy - computer access is for everyone, used by everyone.
Internet access = young adult access. Young adults (14-18 year olds) are high library computer users. What a great initial step for libraries to involve this traditionally hard-to-reach group.
Patrons rely on library computers to take care of the everyday routine tasks as well as to take life-changes steps.
Library computer access differs from other options for computers and wireless (cafes, etc) because it is truly free (no feeling obligated to buy a drink first), it offers a quiet space for work, and it comes with staff to help navigate all ranges of computer and technology issues.
Library internet users can be segmented into 3 groups. Power users, who use the library as their sole access point and come almost daily. Supplemental users, who use the library internet routinely but have other internet options. Occasional users who use the library interest in an emergency, during a time of transition, or to do the quick occasional task.
Low income patrons are less likely to use library internet access overall, but if they do use it, they are more likely to be very frequent users. Same for 19-24 year olds. Youth 14-18 are the most likely user group of library internet and are also very frequent users.
The data was gathered from an impressive scale of telephone surveys, online surveys via public library computers and case studies/interviews at a few libraries.
The report lays out broad use statistics and demographics, but combines this with insightful and detailed analysis. It's a thought-provoking read (though time-consuming, unless you opt for the executive summary).
I'm a little wary of some of their findings since part of their data comes from people who are already library computer users (the online survey). Evaluation-speak, this is selecting on the dependent variable, which could introduce bias into the results. They did have a substantial sample from the random phone surveys and it looks like they did some creative mathematical weighting to combine the phone and web samples to reduce bias as much as possible. But it's something to keep in mind.
A few findings I found particularly interesting.
Almost 1/3 of Americans used their public library for internet access.
While there is higher use of library internet connections among people in households living below the poverty line (44%, higher for young adults and seniors), "people of all ages, incomes, races, and levels of education go to the library for Internet access, whether they have a connection at home or not". Libraries are still great agents of democracy - computer access is for everyone, used by everyone.
Internet access = young adult access. Young adults (14-18 year olds) are high library computer users. What a great initial step for libraries to involve this traditionally hard-to-reach group.
Patrons rely on library computers to take care of the everyday routine tasks as well as to take life-changes steps.
Library computer access differs from other options for computers and wireless (cafes, etc) because it is truly free (no feeling obligated to buy a drink first), it offers a quiet space for work, and it comes with staff to help navigate all ranges of computer and technology issues.
Library internet users can be segmented into 3 groups. Power users, who use the library as their sole access point and come almost daily. Supplemental users, who use the library internet routinely but have other internet options. Occasional users who use the library interest in an emergency, during a time of transition, or to do the quick occasional task.
Low income patrons are less likely to use library internet access overall, but if they do use it, they are more likely to be very frequent users. Same for 19-24 year olds. Youth 14-18 are the most likely user group of library internet and are also very frequent users.
Wednesday, June 1, 2011
Report of Impacts of Library Summer Reading Program
Libraries around the country are poised to start their summer reading programs. I just ran into this evaluation report done June 2010 on impacts of library summer reading programs:
The Dominican Study: Public Library Summer Reading Programs Close the Reading Gap (http://www.dom.edu/academics/gslis/downloads/DOM_IMLS_book_2010_FINAL_web.pdf)
The report was done by Susan Roman, Deborah Carran, and Carole Fiore through Dominican University Graduate School of Library & Information Science. It was funded by an IMLS National Leadership Grant.
The study was a much-needed follow up to 30-yr old seminal research on library reading programs and took on the ambitious scope of being a national study, looking at effects of reading programs in several states across the country. The study sample was all 3rd grade students (going into 4th grade) from 11 schools in large and small communities in urban, rural and suburban areas. Schools had to have 50% or more students receiving free/reduced lunch (standard measure for children living poverty). Researchers collected data using surveys of students, teachers, public librarians and school librarians as well as results from the Scholastic Reading Inventory administered before and after summer reading programs.
Because researchers were trying to have a broad, national scale for the study, they were unable to create a carefully controlled environment for the research. The design was causal comparative - there was no control group, students opted into the summer reading programs as they chose, and summer reading programs (as well as the experiences of non-participants) varied.
Results of the research show some great data and insight into the value of summer reading programs as identified by students, parents, teachers and librarians. These groups strongly believe summer reading programs make a difference. Unfortunately, researchers were unable the demonstrate a correlation between participation in summer reading program and increased scores on the Scholastic Reading Inventory (SRI). While students who participated in the program universally scored higher on the pre- and post-tests than students who didn't participate, there was no evidence that the program caused further increase in these already-high scores.
This would have been a real Holy Grail for demonstrated impacts in ways that funders so often want to see. In fact, results actually showed an increase in scores on the SRI for students who did not participate in the summer reading program. This is a puzzling result, since we usually take it for granted that reading levels drop over the summer months. As the researchers point out, they don't know what was going on for non-participants - maybe they were involved in alternative reading programs. Without the ability to control the context more, it's difficult to interpret these results.
I wonder if the researchers dug into the effects of the summer reading program, controlling for socio-economic status. For example, if they looked at SRI scores with participation in the program and socio-economic status as independent variables, along with an interaction term. I'm imagining a regression that looks like:
SRI score = B1(Program Participation) + B2(Socio-economic status) + (Program Participation * Socio-economic status)
My hypothesis is that perhaps there is a differential effect of summer reading programs - they are nice but not necessary for students from high-income families but they are invaluable resources for students from low-income families. This would certainly support the idea of libraries as democratizing agents in communities.
In the end, the researchers call for a more focused, controlled study of summer reading programs to drill down to quantifiable impacts. I am intrigued by their works so far and hope that it goes further.
The Dominican Study: Public Library Summer Reading Programs Close the Reading Gap (http://www.dom.edu/academics/gslis/downloads/DOM_IMLS_book_2010_FINAL_web.pdf)
The report was done by Susan Roman, Deborah Carran, and Carole Fiore through Dominican University Graduate School of Library & Information Science. It was funded by an IMLS National Leadership Grant.
The study was a much-needed follow up to 30-yr old seminal research on library reading programs and took on the ambitious scope of being a national study, looking at effects of reading programs in several states across the country. The study sample was all 3rd grade students (going into 4th grade) from 11 schools in large and small communities in urban, rural and suburban areas. Schools had to have 50% or more students receiving free/reduced lunch (standard measure for children living poverty). Researchers collected data using surveys of students, teachers, public librarians and school librarians as well as results from the Scholastic Reading Inventory administered before and after summer reading programs.
Because researchers were trying to have a broad, national scale for the study, they were unable to create a carefully controlled environment for the research. The design was causal comparative - there was no control group, students opted into the summer reading programs as they chose, and summer reading programs (as well as the experiences of non-participants) varied.
Results of the research show some great data and insight into the value of summer reading programs as identified by students, parents, teachers and librarians. These groups strongly believe summer reading programs make a difference. Unfortunately, researchers were unable the demonstrate a correlation between participation in summer reading program and increased scores on the Scholastic Reading Inventory (SRI). While students who participated in the program universally scored higher on the pre- and post-tests than students who didn't participate, there was no evidence that the program caused further increase in these already-high scores.
This would have been a real Holy Grail for demonstrated impacts in ways that funders so often want to see. In fact, results actually showed an increase in scores on the SRI for students who did not participate in the summer reading program. This is a puzzling result, since we usually take it for granted that reading levels drop over the summer months. As the researchers point out, they don't know what was going on for non-participants - maybe they were involved in alternative reading programs. Without the ability to control the context more, it's difficult to interpret these results.
I wonder if the researchers dug into the effects of the summer reading program, controlling for socio-economic status. For example, if they looked at SRI scores with participation in the program and socio-economic status as independent variables, along with an interaction term. I'm imagining a regression that looks like:
SRI score = B1(Program Participation) + B2(Socio-economic status) + (Program Participation * Socio-economic status)
My hypothesis is that perhaps there is a differential effect of summer reading programs - they are nice but not necessary for students from high-income families but they are invaluable resources for students from low-income families. This would certainly support the idea of libraries as democratizing agents in communities.
In the end, the researchers call for a more focused, controlled study of summer reading programs to drill down to quantifiable impacts. I am intrigued by their works so far and hope that it goes further.
Subscribe to:
Posts (Atom)