Simple evaluation tool - the card sort

One of my favorite evaluation tools in the Card Sort. To understand visitors' perspective on something, hand them a stack of cards, each with a 1-2 word description of the topic you have in mind. Ask visitors to pull out the words that best and least describe the topic (limit them to 3-4; many people will want to pull out 8 or 9 cards). Then follow up and ask them why they chose the cards they did.

When you analyze the data, it's interested to add them the numbers for what cards were chosen the most, and which cards were ignored the most. Add to that some really great qualitative information about why people made their selections. Often, you find that people have different interpretations of the words than you, as the researcher, had.

Here's an example. Let's say you want to know how people perceive the library's current collection of fiction. Put together a list of 10-15 words that could possibly describe the collection - and don't be afraid to include some negative words (good selection, new materials, lots of options, worn out, not relevant to me, etc). The words you choose are important. Keep them simple, so people can process them quickly. But be specific and even a bit daring - that will bring out interesting comments and discussion. Above all, make sure they are relevant to what you want to know about. Test your words with a few people from the organization. Then try the list out on a few patrons before going live with the study.

As a rule of thumb, when you go live, ask as many people to do the card sort as it takes until you feel like answers are getting redundant. If you must have a number, I would recommend 30 people as a minimum.

To analyze results, tally up the total of "best" and "worst" hits each word got. Then compare what people said as their reasons why they chose those words. What patterns or themes do you see? One note: the more accurately you transcribe people's responses to why they chose words, the better your qualitative analysis will be. Resist the temptation to summarize people's statements when collecting the data. Try to write down what they say as close to word-for-word as possible. It's tough, but worth it. You don't want to add your layer of interpretation until all the data is collected.

Data collection is fun - and not just for you

Here is a somewhat dated evaluation done at the San Jose Children's Discovery Museum, but I thought their qualitative methods were interested.

http://www.hfrp.org/out-of-school-time/ost-database-bibliography/database/discovery-youth

In the first evaluation, they did focus groups with youth were they divided the youth into teams of 2-3 and had them fill the in the answers to 4 open-ended questions. Then they brought all the teams together (about 4 total) and had them discuss.

In the second evaluation, they did a group activity with program participants. Participants broke out into groups of 4-5. Then they circulated around the room to large pieces of paper with a different question written on each. Each group wrote answers on the large sheet of paper for each question, so they were able to react to a previous group's comments.

These techniques hit on 2 principles that I think are key to focus groups and interviews. First, make it fun. Vary the questions, get people up and moving, use the 5 senses, present extremes with humor.

Second, give them something to react to. I find I get much more detailed and colorful responses from people if I present something to them first and get their feedback on it. A list, a board full of written-on sticky notes, a photo, a written description. In both of the examples above, participants were responding to what other participants said. A key to that, I believe, is to have people commit to an opinion on paper first. If you jump into reactions right away, then less vocal opinions get lost.

New survey resource! All Our Ideas

Check out this nifty new tool for doing a quick survey to rank items: http://www.allourideas.org/



This is a great resource for institutions like museums and libraries that want to poll visitors on potential new services or programs. Imagine one of those staff meetings where you create a wish list of things you'd like to do for your visitors. This list may look something like:

- Validate parking
- Provide more food options in vending/snack area
- Have more staff in exhibit areas to give visitors more information about exhibits
- Install more benches, chairs in gallery areas
- Section off part of the library as a "No Shhhh Zone" that people can use for group work
- Offer story time programs on Saturday mornings

It's great for staff to generate ideas about how to improve visitor services. With All Our Ideas you can quickly take it to the next step and ask visitors to respond to the wish list. The site takes your list and gives visitors 2 of the options from the list and they have to vote for one item over the other (or select "I don't know"). They repeat this process over and over in a matter of seconds or minutes. The data is aggregated to a ranking of options, along with fun visualizations of the data.

A great feature of the tool is that respondents can also add their own idea. That idea is then put into the list to be voted on by others. What a fun way to bring fresh ideas to the table - and have immediate visitor feedback on them.

I will add my cautionary, unsolicited advise about asking visitors for ideas. When you ask visitors for ideas, ask about the things they are experts on, not the things you are an expert on.

For example, if you ask visitors "What kind of programs do you want us to do?", you inevitably gets answers that are way out of your budget, mission, or capacity. But what do you expect? You're the experts in program development - you know the profession and the feasibility. On the other hand, if you ask parents/caregivers "What new play items would you like to see in the baby area?", they can fill you in on the latest trends in baby toys that they talk about every week at playgroup.

Keep visitors talking about what they know about, and you can translate that into exceptional experiences that meet their needs.