I am taking a Coursera class on recommender systems right now. It is excellent, just the right blend of mathematics and practice. Exposure to the content of the class motivated this post.
In case you aren’t familiar with them, recommender systems are computing systems which make recommendations to the users of an application. You’ve surely encountered them in your day to day internet travels. Amazon leverages recommender technologies to suggest additional items for viewing. Netflix is a giant recommender system. Pandora, Spotify, Hotwire: all companies which make recommendations to their users with the help of fairly sophisticated algorithms.
Ostensibly, recommender systems are designed to reduce two negative effects associated with large quantities of information: Information Overload and the Paradox of Choice. The Paradox of Choice states that as individuals are given more options for a choice, they become less satisfied with their decision. Information Overload hypothesizes that as more, and more rapidly changing, information becomes available, individuals are overwhelmed by the new information and are no longer able to rely on the old to make decisions. Predictive accuracy falls and the quality of an individual’s decisions deteriorates.
At first glance recommender systems would seem to be designed to prevent these kinds of malfunction. A recommender system reduces our range of options to a more manageable size. Rather than having to consider 100 movies, I can now only consider the top 5 recommended options. Or instead of searching through hundreds of cameras in preparation for a purchase on Amazon, I can refer to only the most recommended. This would seem to be an improvement.
I’m wondering, however, if at some point recommender systems begin to contribute to the paradox of choice and information overload, when they are actually designed to reduce those two effects. While it may be the case that these systems reduce overload and dissatisfied decision making theoretically and in the math, it may not be the case practically and in how the results of a recommendation system are presented to the user of a service, that is, in the user interface.
Consider a site selling products that only allows me to browse items by hierarchy or list without recommendations. One item does not lead to another. Instead, I have to review an item, backtrack, consider my next option, review it, backtrack, consider my next option, and so on. In a sense, the items are discrete units. I can only consider one at a time, with a distinct break between it and my consideration of the next item.
A site like Amazon, however, recommends many items from any single item so that every item is linked to every other in a network with many edges rather than in a tree like hierarchy. In a sense, the items are no longer discrete. In reviewing one item, I am immediately presented with related items for consideration, and it is a single click to get to one of those recommendations, and another, and another, ad infinitum. Or consider how Netflix presents its list of recommendations: essentially a huge matrix of movies from which I need to choose just one to view, which then shows related movies, hardly narrowing my range of choices.
I suspect that many of us have had the experience of browsing Amazon far too long trying to decide which widget to purchase, overwhelmed by the number of options, the reviews, and the possibility of a better deal just one more click away, or paralyzed by the unending list of Netflix recommendations, finally having to settle on a film that you hope is good enough but about which you’re still not quite sure.
My hypothesis is that, within the context of their user interfaces, recommender systems actually contribute to information overload and the paradox of choice rather than reduce those negative effects, which they were originally designed to do. Because of the way contemporary applications present the information returned by recommender systems, and because the recommendation algorithm is run again and again for each newly viewed item, generating an infinitely traversable network of items, recommender systems actually deliver users more choices more quickly than if users were forced to browse items one by one without recommendations.
Is this true, and if so, are there any solutions? So far the Recommender Systems class has not addressed this issue.