This is Part III of a series of articles on the new era of the Stream, a new phase of the Web.

In Part I, The Message is the Medium, I explored the shift in focus on the Web from documents to messages.

In Part II, Drowning in the Stream, we dove deep into some of the key challenges the Stream brings with it.

Here in Part III, we will discuss new challenges and solutions for keeping up with streams as they become increasingly noisy and fast-moving.

 

Getting Attention in Streams

Today if you post a message to Twitter, you have a very small chance of that message getting attention. What’s the solution?

You can do social SEO and try to come up with better, more attention-grabbing, search engine attracting, headlines. You can try to schedule your posts to appear at optimal times of day. You can even try posting the same thing many times a day to increase the chances of it being seen.

This last tactic is called “Repeat Posting” and it’s soon going to be clogging up all our streams with duplicate messages. Why am I so sure this is going to happen? Because we are in an arms race for attention. In a room where everyone is talking, everyone starts talking louder, and soon everyone is shouting.

Today when  you post a message to Twitter, the chances of getting anyone’s attention are low and they are getting lower.  If you have a lot of followers, the chances are a little better that at least some of them may be looking at their stream at precisely the time you post. But still, even with a lot of followers, the odds are that most of your followers probably won’t be online at that precise moment you post something, and so they’ll miss it.

Scheduled Posting

But it turns out there are optimal times of day to post, when more of your followers are likely to be looking at their streams. A new category of apps, typified by Buffer, has emerged to help you schedule your Tweets to post at such optimal times.

Using apps like Buffer, you can get more attention to your Tweets, but this is only a temporary solution. Because the exponential growth of the Stream means that soon even posting a message at an optimal time will not be enough to get it in front of everyone who should see it.

Repeat Posting

To really get noticed, above the noise, you need your message to be available at more than one optimal time, for example many times a day, or even every hour.

To achieve this, instead of posting a message once at the optimal time per day, we may soon see utilities that automatically post the same message many times a day – maybe every hour – perhaps with slightly different wording of headlines, to increase the chances that people will see them. I call this “repeat posting” or “message rotation.”

Repeat posting tools may get so sophisticated that they will A/B test different headlines and wordings and times of day to see what gets the best clickthroughs and then optimize for those. These apps may even intelligently rotate a set of messages over several days, repeating them optimally until they squeeze out every drop of potential attention and traffic, much like ad servers and ad networks rotate ads today.

But here’s the thing — as soon as anyone starts manually or automatically using repeat posting tactics, it will create an arms race – others will notice it, and compete for attention by doing the same thing. Soon everyone will have to post repeatedly to simply get noticed above the noise of all the other repeat posts.

This is exactly what happens when you are speaking in a crowded room. In a room full of people who are talking at once, some people start talking louder. Soon everyone is shouting and losing their voice at the same time.

This problem of everyone shouting at once is what is soon going to happen on Twitter and Facebook and other social networks. It’s already happening in some cases – more people are posting the same message more than once a day to get it noticed.

It’s inevitable that repeat posting behavior will increase, and when everyone starts doing it, our channels will become totally clogged with redundancy and noise. They will become unusable.

What’s the solution to this problem?

What to Do About Repeat Posting

One thing that is not the solution is to somehow create rules against repeat posting. That won’t work.

Another solution that won’t work is to attempt to detect and de-dupe repeats that occur. It’s hard to do this, and easy to create repeat posts that have different text and different links, to evade detection.

Another solution might be to recognize that repeat posting is inevitable, but to make the process smarter: Whenever a repeat posting happens, delete the previous repeat post. So at any given time the message only appears once in the stream. At least this prevents people from seeing the same thing many times at once in a stream. But it still doesn’t solve the problem of people seeing messages come by that they’ve seen already.

A better solution is to create a new consumption experience for keeping up with streams, where relevant messages are actually surfaced to users, instead of simply falling below the fold and getting buried forever. This would help to ensure that people would see the messages that were intended for them, and that they really wanted to see.

If this worked well enough, there would be less reason to do scheduled posting, let alone repeat posting. You could post a message once, and there would be much better chance of it being seen by your audience.

At Bottlenose, we’re working on exactly this issue in a number of ways. First of all, the app computes rich semantic metadata for messages in streams automatically, which makes it possible to filter them in many ways.

Bottlenose also computes the relevance of every message to every user, which enables ranking and sorting by relevancy, and the app provides smart automated assistants that can help to find and suggest relevant messages to users.

We’re only at the beginning of this and these features are still in early beta, but already we’re seeing significant productivity gains.

Fast-Moving Streams

As message volume increases exponentially in streams, our streams are going to not just going to be noisier, they are going to move faster. When we look at any stream there will be more updates per minute – more new messages scrolling in – and this will further reduce the chances of any message getting noticed.

Streams will begin to update so often they will literally move all the time. But how do you read, let alone keep up with, something that’s always moving?

Today, if you follow a Twitter stream for a breaking news story, such as a natural disaster like the Tsunami in Japan, or the death of Steve Jobs, you can see messages scrolling in, in real-time every second.

In fact, when Steve Jobs died, Twitter hit a record peak of around 50,000 Tweets per minute. If you were following that topic on Twitter at that time, the number of new messages pouring was impossible to keep up with.

Twitter has put together a nice infographic showing the highest Tweets Per Second events of 2011.

During such breaking news events, if you are looking at a stream for the topic, there is not even time to read a message before it has scrolled below the fold and been replaced by a bunch of more recent messages. The stream moves too fast to even read it.

But this doesn’t just happen during breaking news events. If you simply follow a lot of people and news sources, you will see that you start getting a lot of new messages every few minutes.

In fact, the more people and news sources, saved searches, and lists that you follow, the higher the chances are that at any given moment there are going to be many new messages for you.

Even if you just follow a few hundred people, the chances are pretty high that you are getting a number of new messages in Twitter and Facebook every minute. That’s way more messages than you get in email.

And even if you don’t follow a lot of people and news sources – even if you diligently prune your network, unfollow people, and screen out streams you don’t want, the mere exponential growth of message volume in coming years is soon going to catch up with you. Your streams are going to start moving faster.

But are there any ways to make it easier to keep up with these “whitewater streams?”

Scrolling is Not the Answer

One option is to just make people scroll. Since the 1990’s UX designers have been debating the issue of scrolling. Scrolling works, but it doesn’t work well when the scrolling is endless, or nearly endless. The longer the page, the lower percentage of users will scroll all the way down.

This becomes especially problematic if users are asked to scroll in long pages – for example infinite streams of messages going back from the present to the past (like Twitter, above). The more messages in the stream, the less attention those messages that are lower in the stream, below the fold, will get.

But that’s just the beginning of the problem. When a stream is not only long, but it’s also moving and changing all the time, it becomes much less productive to scroll. As you scroll down new stuff is coming in above you, so then you have to scroll up again, and then down again. It’s very confusing.

In long streams that are also changing constantly it is likely that engagement statistics will be very different than for scrolling down static pages. I think it’s likely engagement will be much lower, the farther down such dynamic streams one goes.

Pausing the Scroll is Not the Answer

Some apps handle this problem of streams moving out from under you by pausing auto-scrolling as you read – they simply notify you that there are new messages above whatever you are looking at. You can then click to expand the stream above and see the new messages. Effectively they make dynamic streams behave as if they are not dynamic, until you are ready to see the updates.

This at least enables you to read without the stream moving out from under you. It’s less disorienting that way. But in fast moving streams where there are constantly new updates coming in, you have to click on the “new posts above” notification frequently, and it gets tedious.

For example, here is Twitter, on a search for Instagram, a while after the news of their acquisition by Facebook. After waiting only a few seconds, there are 20 new tweets already. If you click the bar that says “20 new Tweets” they expand. But by the time you’ve done that and started reading them, there are 20 more.

 

Simply clicking to read “20 new tweets” again and again is tedious. And furthermore, it doesn’t really help users cope with the overwhelming number of messages and change in busy streams.

The problem here is that streams are starting to move faster than we can read, even faster than we can click. How do you keep up with this kind of change?

Tickers and Slideshows Are Helpful

Another possible solution to the problem of keeping up with moving streams is to make the streams become like news tickers, constantly updating and crawling by as new stuff comes in. Instead of trying to hide the movement of the stream, make it into a feature.

Some friends and I have tested this idea out in an iPad app we built for this purpose called StreamGlider. You can download StreamGlider and try it out for yourself.

StreamGlider shows streams in several different ways — including a ticker mode and a slideshow mode where streams advance on their own as new messages arrive.

 

The Power of Visualization

Another approach to keeping up with fast moving streams is to use visualization, like we’re doing in Bottlenose, with our Sonar feature. By visualizing what is going on in a stream you can provide a user with instant understanding of what is in the stream and what is important and potentially interesting to them, without requiring them to scroll, skim or read everything first.

Sonar reads all the messages in any stream, applies natural language and semantic analysis to them, detects and measures emerging topics, and then visualizes them in realtime as the stream changes.

It shows you what is going on in the stream – in that pile of messages you don’t have time to scroll through and read. As more messages come in, Sonar updates in realtime to show you what’s new.

You can click on any trend in Sonar that interests you, to quickly zoom into just the messages that relate.

The beauty of this approach is that it avoids scrolling until you absolutely want to. Instead of scrolling, or even skimming the messages in a stream, you just look at Sonar and see if there are any trends you care about. If there are, you click to zoom in and see only those messages. It’s extremely effective and productive.

Sonar is just one of many visualizations that could help with keeping up with change in huge streams. But it’s also only one piece of the solution. Another key piece of the solution is finding things in streams.

Finding Things in Streams

Above, we discussed problems and solutions related to keeping up with streams that are full of noise and constantly changing. Now let’s discuss another set of problems and solutions related to finding things in streams.

Filtering the Stream

For a visualization like Sonar to be effective, you need the ability to filter the stream for the sources and messages you want, so there isn’t too much noise in the visualization. The ability to filter the stream for just those subsets of messages you actually care about is going to be absolutely essential in coming years.

Streams are going to become increasingly filled with noise. But another way to think about noisy streams is that they are really just lots of less-noisy streams multiplexed together.

What we need is a way to intelligently and automatically de-multiplex them back into their component sub-streams.

For example, take the stream of all the messages you receive from Twitter and Facebook combined. That’s probably a pretty noisy stream. It’s hard to read, hard to keep up with, and quickly becomes a drag.

In Bottlenose you can automatically de-multiplex your streams into a bunch of sub-streams that are easier to manage. You can then read these, or view them via Sonar, to see what’s going on at a glance.

For example, you can instantly create sub-streams – which are really just filters on your  stream of everything. You might make one for just messages by people you like, another for just messages by influencers, another for just news articles related to your interests, another for just messages that are trending, another of just photos and videos posted by your friends, etc.

The ability to filter streams – to mix them and then unmix them – is going to be an essential tool for working with streams.

Searching the Stream

In the first article in this series we saw how online attention and traffic is shifting from search to social. Social streams are quickly becoming key drivers for how content on the Web is found. But how are things found in social streams? It turns out existing search engines, like Google, are not well-suited for searching in streams.

Existing algorithms for Web search do not work well for Streams. For example, consider Google’s PageRank algorithm.

In order to rank the relevancy of Web pages, PageRank needs a very rich link structure. It needs a Web of pages with lots of links between the documents. The link structure is used to determine which pages are the best for various topics. Effectively links are like votes – when pages about a topic link to other pages about that topic, they are effectively voting for or endorsing those pages.

While PageRank may be ideal for figuring out what Web pages are best, it doesn’t help much for searching messages, because messages may have no links at all, or may be only very sparsely linked together. There isn’t enough data in individual messages to figure out much about them.

So how do you know if a given message is important? How do you figure out what messages in a stream actually matter?

When searching the stream, instead of finding everything, we need to NOT find the stuff we don’t want. We need to filter out the noise. And that requires new approaches to search. We’ve already discussed filtering above and the ability to filter streams is a per-requisite for searching them intelligently. Beyond that, you need to be able to measure what is going on within streams, in order to detect emerging trends and influence.

The approach we’re taking in Bottlenose to solve this is a set of algorithms we call “StreamRank.” In StreamRank we analyze the series of messages in a stream to figure out what topics, people, links and messages are trending over time.

We also analyze the reputations or influence of message authors, and the amount of response (such as retweets or replies or likes) that messages receive.

In addition, we also measure the relevance of messages and their authors to the user, based on what we know of the user’s interest graph and social graph.

This knowledge enables us to rank messages in a number of ways: by date, by popularity, by relevance, by influence, and by activity.

Another issue that comes up when searching the Stream is that many messages in streams are quite strange looking – they don’t look like properly formed sentences or paragraphs. They don’t look like English, for example. They contain all sorts of abbreviations, hashtags, @replies, and short URLs, and they often lack punctuation and are scrunched to fit in 140 character Twitter messages.

Search algorithms that use any kind of linguistics, disambiguation, natural language processing, or semantics, don’t work well out of the box on these messy messages.

To apply such techniques you need to rewrite them so that they work on short, messy, strange looking messages. This is also something we’ve built in Bottlenose — we’ve built a new natural language processing and topic detection engine in Javascript that is designed specifically to handle these types of streams and messages.

These are some of the new challenges and solutions we’re applying in Bottlenose to make working with streams more productive. They are components of what we call our “StreamOS,” a new high-level Javascript and HTML5 operating system for applications that need to do smart things with streams. We’ll be writing a lot more about this in future articles.