Early Usage Data on the New Penn State University Libraries' Bento-style Search Results

6/28/2018

Bento search results pages have been a prominent feature of academic libraries for years now. The term "bento" is a metaphor coined by Tito Sierra to refer to the dividing up of search result types into clearly divided boxes. Ideally, academic libraries, like my own at Penn State University, would be able to read a user's mind to understand exactly what it is that a user is attempting to find: a book? a journal title? an article? browsing articles or books by topic or subject? maybe a specific librarian? a newspaper? or a database? And so on. Libraries are sort of a unique space where we hold millions of objects that, while similar, are very different in how and why they are used. For example, digital archives, physical archives, books, online articles, physical media, ebooks, and so on. Not to mention that expanded collection of items available via interlibrary loan, especially those items that can be delivered in a day or two.

The disparate nature of our systems furthers the difficulty in delivering a relevant item for every search. Constance Malpas illustrated this well recently when she gave a talk at the Big Ten Academic Alliance Library Conference (called "Operationalizing the BTAA Collective Collection") on the difficulty of the sheer number of systems, services, standards, vendors and publishers that combine to create frail systems full of edge case problems. And as we grapple with tying these things together simply for our patrons, Google continues to provide basic day-to-day searching that sets the bar high. Users have that Google expectation when they come to a libraries' site (or any site for that matter).

Increasingly, academic libraries present a single search box from their homepage, as opposed to several search boxes for different systems or needs. And despite our wish that this single search box be as good as Google, it simply isn't. Cory Lown, Tito Sierra and Josh Boyer wrote:

"The library search experience must be designed carefully to balance user needs and expectations with the capabilities of library information systems"

I read this, and much of their article, as we need to be honest with ourselves about what we can't deliver and be ruthlessly user centered in our designs with the aid web analytics and usability studies.

Penn State University Libraries' Bento

With that, let's talk about Penn State University Libraries' newest addition to our website services: our own bento search! Search from the main search box at our website. This launched on June 14, 2018. Here's a screenshot:

screenshot of the penn state university libraries search results

Bento interfaces, like ours, have been created in part to solve these problems (bear in mind we are a Summon client):

  1. Overcomes the problems of relevancy in discovery systems that contain too much stuff. Newspaper records are especially problematic in dirtying up relevancy.
  2. Makes different content types clearly visually distinct. Again, discovery systems do this a bit, but in my opinion they are not distinct enough. For example, Summon uses icons and small labels in a single column of results. It's just not simple enough to scan.
  3. Gives control back to libraries. This is important for a lot of reasons, but just one of them is that your library might have some of its own priorities in discovery. For example, highlighting a particular collection in a particular way. Well, when you write the code you can do whatever you want, you're not wedded to customizing within the confines of a vendor discovery system.

Now, I would argue that our new bento search does solve the above problems well. But it's far from perfect. What's next is to find out what the opportunities for improvement and what the problems are in the design and make the changes. Usability testing is certainly one part of this, but, you also need real data from the production website.

I love usability testing. I myself have conducted interviews, and they are instructive and informative, especially from my perspective as a developer. Yet, as the mature user experience experts we are, we know that usability testing isn't perfect, and at best only reveal indicators. When you marry well-run usability tests with usage data, then you can start iterating on design updates. And, of course, you need a great team of librarians, designers and developers.

The Data

The returns are very early, but I've done some light analysis of the first 12 days of its existence based on the framework set forth in Lown, Sierra and Boyer's article. They examined 1.4 million transactions, and I'm taking a look at around 33 thousand, so there's a grain of salt, but that said, the usage data is eerily similar. Here are the returns so far:

33,548 searches (24,564 unique searches)

2,795/day average

Click-through Ratio (CTR):

27,588/33,548 => 82.23%

The CTR is the percentage of page views that resulted in a click of some link within the bento search results page. Clicks anywhere in the footer or header don't count towards this number.

Here's are breakdown of boxes by percent:

box clicked

total clicks

percent of total clicks

percent of total clicks minus search again (refinements)

articles

 10,998

40.06%

55.85%

search again

 7,765

28.28%

 

books and media

 7,143

26.02%

36.27%

news/mag articles

 770

2.80%

3.91%

website

 286

1.04%

1.45%

databases

 207

0.75%

1.05%

best bets

 131

0.47%

0.66%

what’s not here

 176

0.64%

0.89%

libguides

 64

0.23%

0.33%

experts

 48

0.17%

0.24%

total

 27,457

100%

 

total without search again

 19,692

 

100%

We are also monitoring search terms and the number of the result (or see all) clicked within a box. Again, more data needed here, but in general it's a pattern of about, roughly, 80% first link, 10% see all and the rest is the second or third link. 

We are using Google Analytics Event Tracking. Here's the script that does the work. We were using Google Tag Manager for a time before this project, but we noticed a very odd bug where the Tag Manager script was disrupting the file upload process in webforms and content editing, plus our expert who set it up moved on and we did not have a lot of expertise on this tool. We did also notice it was a gateway for other third-party services like CrazyEgg, which loaded on every page, despite the fact that it seemed to be disabled. Regardless, I chose to do this in code because it is simpler.

Some Analysis

It is very early and the sample size is small, but we can tell a few things from the data so far, especially comparing to the Lown, Sierra and Boyer article. Our boxes and features don't line up completely with what NC State University Libraries had when they wrote this, but they are pretty close. For example, both interfaces place articles and books in a prominent location. Here are some observations:

  1. The bento interface is standing up to load and getting used a lot. It's the summer and usage is relatively low compared to fall/spring, but the application code (custom module written on top of Drupal) is standing up to load pretty well so far. The number of searches per day is about the same as what NC State saw.
  2. Users are getting to books and articles they are looking for. Around 83% of page views on the results page leads to a click on something the user finds interesting for whatever reason. 91%, roughly, of those clicks are clicks on books and articles.
  3. Clicks in other boxes are very low. This was expected, but, I had an expectation that they'd be a bit higher. This could be due to some issues with the ways in which we are handling indexing and retrieval for the secondary boxes. I'd say we need more data, perhaps some user testing as well to see if we can figure out how users are seeing these boxes.
  4. Lots of refinements. A common practice on Google is to search and then search again if you're not finding what you're looking for. Perhaps this is happening here too. 28% of searchers are finding a need to change their search term and search again. Is this high percentage a result of "getting used to the new interface" or is it a baseline that can just be expected? Time will tell, and maybe user testing can help us understand. This is not a metric present in the Lown, Sierra, Boyer study.
  5. Best Bets aren't serving many. We try to preemptively handle certain search terms with a heavily emphasized "Best Bet" box (search "web of science" for example). This isn't a new idea, NCSU Libraries did this as do many other bento interfaces (and discovery interfaces, like Summon). However, NCSU saw best bet usage at around 7.81%. We are seeing .66%. This seems like too wide a discrepancy. Perhaps there is an issue here. Again this is something we should keep an eye on and see if usage patterns change or stay the same.

To me, the CTR is the vital sign of the application. If it dips too far below where it's at at the moment, we'll know something's not right. If it goes up any higher, that's assuredly a good thing. In general, at this stage in the game, we can really only understand that the application is being used relatively successfully, and all else is in the "keep monitoring" stage. Having this baseline will be helpful for us in understanding future changes and how they affect usage.

You can see more about what we're up to by reading our Roadmap. Credits: the team working on this project consists of Ruth Kitchin Tillman, Banu Kutlu, Binky Lush and myself.

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.