Adaptive Web Sites

(this is a slightly expanded transcript of a talk I gave at Oxford in June 2009 about my work there)

Hi! I’m Howard Yeend, my supervisor is Vasile Palade, and the title of my project is:

Implementing Adaptive Web Sites using Machine Learning and Ajax“.

But before I talk about what all those buzzwords mean, I’d like to give a little background information about why this is an important research area, and why I feel it’s the right project for me.

When I was trying to think of a project title, I had a question in mind:

How can we improve the web?

And I think that’s a hugely important question for us to ask.

I don’t need to tell you that the web is strongly interlinked with our daily lives, it’s having a massive impact on the way we interact, the way we make decisions and the way we express ourselves. It’s a very pervasive technology. So it’s important to think about how we can make the web better.

But also in a more general way, I think it’s important for us as Computer Scientists, as students, as geeks, to ask ourselves “How can we improve things”. I mean, that’s kind of our job, isn’t it? That’s why we do what we do – we take technologies and combine them in innovative ways.

We want to make things better.

It’s important to note here that I’m not asking how we could improve the web in 5 years time, or if we had enough users, or if we had the backing of the academic community.

I’m talking about right here, right now – what can we do with the technologies we have available to us, that are robust enough, to improve the web? Well one thing we can do is to implement Adaptive Web Sites.

So that’s really the core of what this project is about. But what are Adaptive Web Sites? I keep saying those words, what do they mean?

Look at what the user is doing on a web site…
…and use that information to improve the web site.

I think that’s a nice phrase to encapsulate what Adaptive Web Sites are all about; you look at what the user is doing, and you use that information to improve the web site. It’s a nice way to explain it. But it’s a little vague, so let’s go a bit deeper into what exactly I’m doing here.

Well first of all I’m actually going to be implementing something. That’s one of the major aims of this project, to actually produce some software at the end of this that you can take away and use on your own web sites. But what am I going to be implementing?

So there was a lot of research in the mid-90′s about Adaptive Web Sites and recommender systems, which is a related field. If you go to amazon, you’ve no doubt seen the little “customers who bought this also bought” feature, that’s a nice little display of a recommender system. But the thing is, in the mid-90′s the technology wasn’t really there. Maybe you can detect that the user was searching google for “pictures of dogs”, and maybe google sent them to your “pictures of cats” page, so we can intercept that and redirect them to our dogs page, to make it more relevant for that user. That’s the kind of adaptation they’re talking about.

And that’s pretty nice, but that scenario doesn’t really happen that often, google have some smart guys working for them, they get it right most of the time.

So this is where the Ajax part comes in, because now, in 2009 we can get a much deeper look into “what the user is doing”.

Sure we can detect that the user hit the pictures of dogs page, but maybe they scrolled down to a particular paragraph.

So we can detect this, and send that information back to our server – while the user is still reading that paragraph – and we can say:

Hey, do we have anything else that this user might be interested in?

And so maybe we have another page on our site about the history of the breed they’re currently looking at, and we can just expand the section they’re reading and insert a little bit of content at the end that they might find useful, just a sentence or two from the relevant page with the option to expand further or link through to that content.

So as they’re reading the page, the page changes in response to what they’re doing.

Maybe they’re hovering over a particular link, perhaps they scrolled down to this or that section, maybe they’re selecting and copying text from the page, or they did a “Ctrl-F” to search for a particular phrase.

And that’s really what this project is all about; collecting this kind of data, theorising about the user’s intent, and then adapting the web site in real time to their needs.

So this kind of technology could be used to improve the relevance of advertising – updating a page’s adverts in real time, or to provide more relevant product and content recommendations, or even just to give web masters a richer insight into what their users are interested in.

Current analytics software, including google analytics, only breaks your content down by page; they’ll say “hey, people spend an average of 30 seconds on this page”.

But what were those users doing during those 30 seconds?

With the technology I’m developing, we’ll be able to say “hey, 50% of visitors nearly clicked this link”, or “600 people only read this one paragraph”, or “most of your visitors scroll down till they find a bold heading”. And this is really powerful data to a content publisher. And it’s my belief that browsing an adaptive web site is going to be a much more useful experience to the end user as well.

In the future, I can see a web where every site links in to every other other site in an adaptive way. Because if we go back to our user who’s interested in dogs, well maybe we actually don’t have any more content that is well matched to what they’re reading. But we can go off to google and pull down the top three results and present them inline to the user. Or we can insert the first paragraph from wikipedia about that topic.

So in this way, when I go to an adaptive web site to research a topic, I’m not just looking at whatever content that site might have, but I’m also looking at everything the web has to offer on that topic.  That’s the real power of adaptive web sites to me.

So that’s my project; Implementing Adaptive Web Sites using Machine Learning and Ajax.

I hope you enjoyed hearing about it.

UPDATE October 2009:

I’ve now completed and passed my MSc, attaining a distinction for this project. I will eventually be writing up my conclusions, which include some novel results. There’s also talk of publication and/or a chapter in a forthcoming book on web usage mining and adaptive web sites, so we’ll see what happens with that. If you’d like to talk about adaptive web sites, or alternative document similarity metrics (a key finding in my work) then please email

Related Posts:

, , , , , , , ,

  1. #1 by Ido Green on June 15, 2009 - 8:59 am

    Very interesting stuff!
    Will you put your code/library somewhere on the web as open source project?

    • #2 by user24 on June 15, 2009 - 12:40 pm

      yes, that’s my aim. I’m going to try to wrap it all up into a wordpress plugin that you can just install and start using. And I’ll probably try to get a workable solution in place for people who aren’t using WP.

      I should also mention that it’d be really helpful to me to collect data from other people’s sites as well as from puremango, so shoot me an email if you’d like to be a part of that:

      Thanks for the interest!

  2. #3 by Kenneth Holm Nielsen on July 19, 2009 - 2:32 pm

    I’ve been pondering this problem aswell. For a big client I presented a solution of using the google analytics api using the Event tracking part of the api, to collect data about the use of AJAX components.

    Ofcourse directly reacting on this information would be a huge improvement. Also I was pondering a feature to help detect if a user was confused in looking for something specific and somehow help them on the way, either by adapting the web site or initiating a live chat with support or something.

    I would absoltely be interested in the results of your research.

  3. #4 by Kulin on August 8, 2009 - 7:23 am

    A lot of what you’re proposing sounds like it would be really intrusive from an end-user perspective. Its like a ‘flight recorder’ capturing everything the user is doing while on the website. Even if we’re to ignore that for a moment, it boils down to the question of do end-users really want or need an intelligent website which assumes it knows what the user wants. Its going to be interesting to see how the end-user experience can indeed be enriched through the use of this technology, instead of becoming counterproductive.

    • #5 by user24 on August 23, 2009 - 5:34 am

      Yes, there are obviously privacy concerns, but if you look at what amazon are doing with recommender technology, that’s generally accepted by users, even though it’s using some pretty “invasive” metrics about what you’ve looked at and bought to generate those recommendations.

      So I think the key is to provide real benefit to users, to be honest about what you’re collecting, and to allow users to easily “turn it off”, and I think most people will be happy with it.

  4. #6 by Eric on October 13, 2009 - 8:48 pm


    Thank you very much for the information in this blog post. Would you be willing to share more information about your research on the adaptive web concepts you are developing?

    • #7 by user24 on October 13, 2009 - 10:33 pm

      Yes, I’m planning to write up my results in a blog post – I have now finished my MSc (attained a distinction on the adaptive web sites project) and actually ended up with some novel results including the ability to predict up to 30 seconds in advance which link a user will click, and also a new document similarity metric which warrants further research.

      I might not be able to write the blog post for some months though, so please email and we can discuss things.

  5. #8 by Colin on October 29, 2009 - 12:44 pm

    Interesting stuff.
    Makes me more confident that computer science is the right degree for me next year.

  6. #9 by Nisha on December 17, 2009 - 11:56 am

    A big thanks , had no clue what adaptive web site was, but reading your article provided great understanding.

  7. #10 by mona on September 20, 2011 - 9:02 pm

    I want to design an adaptive web site and i want to know first, what are the information which i can collect about users?. Second, how can I analyze the users information?. Third, what kind of adaptation I can apply to the web site??
    please help me for importance.

Comments are closed.