A couple of weeks ago, I reported in this blog on the discussion that emerged about Twine on VentureBeat. That discussion quickly turned into some controversy about Twine's potential "usability issues" and led me to wonder about its bizarre PR approach. Both usability and PR have been issues evoked by others in the past. Nova Spivack, who heads Twine, agreed to provide answers to the usability metric, on a different platform, as by that point the VentureBeat page had become a haven for anonymous users fueling unproductive flame wars.
Keeping Twine to its word, today I am glad to publish the result of my interview with Nova Spivack, Twine's founder and CEO, on this blog. To prevent any accusation of potential bias, given the emotional reactions this topic has provoked on all sides in the past, I reproduce our discussion word-of-word below, in a chronological order.
I encourage you to read the whole lot since quite a bit, especially the traffic and usability hard data, appears at the end. Interestingly, this data shows that page views per unique visitor / month have declined from 16 to 3 in the past 7 months. I don't think Twine had planned for those calculations when they sent me the data, but here it is, confirming an argument I've made all along about the potential for much better usability in Twine. Having shared that with Nova, he sent me data on #pageviews per registered user, which shows an average of 7 pages per month. Overall, I think it fares below expectations, but short of a benchmarking analysis, it's hard to say what the right number should look like, and so I'd like to invite readers in the know to comment on this aspect in particular. I know Twine will be watching closely.
More importantly, moving one step beyond the numbers, I must say that I still am disappointed in the manner Nova seems to undermine the problem of usability and duplicates, asserting it's a side issue that's not been raised by many users, when a number of indicators have shown otherwise.
And then there is the PR. I again found it a little frustrating to exchange with Twine. It still felt antagonizing, like you're either "with" Twine or you're "against" Twine. See our discussion. Throughout our exchange, the positive is highlighted and the constructive is dismissed or toned down. Whatever you can bring up, they have thought of it. The tone, even if improved, still feels corporate-defensive, far from the open, transparent exchange one would expect from such an interactive service. And then all those mentions about "not having the time" managed to convince me that my questions were positively annoying Nova... From my end, funnily enough, I feel like I'm helping them.
As I mentioned previously, a number of "web 3.0" adopters think the same way about Twine's PR. Most, not all, are taking a more prudent approach, given Nova's perceived aura in the community. In a number of cases, this prudence is clearly driven by financial and reputational motives. Alienating the semweb champion, politically, is not a smart move, some said. Just writing these lines, I can't help but wonder what wrath I am bringing upon myself. That shouldn't be. Self-preservation should never trump seeking the truth. So be it, what doesn't kill me...
Moving on... Twine has made some efforts and, in spite of all the above, I want to
thank them for participating in this exercise in transparency.
Bottom line is, first, that the data provided demonstrates there is a usability and engagement issue. What Twine does not see yet, is that admitting its problems would cost less than trying to prove users wrong. Even on the capital raising front, I'd say. Second, it would benefit from catering to future customers, and less on niche improvements for the current audience, that's still made in large parts of geeks like you and me. Well, mostly you, actually, given my blog's current audience... ;)
[Update] I strongly encourage any reader to check Nova's comments on Twitter, unless he erases them like he did last time. He's "going after me" for initially quoting 2 as the #pageviews per user as shown in his table 1, as opposed to 3.4 for February in his updated second table. I'm not sure he gets the overall point. Down from 16, Nova. Yes, registered users are higher, I know, I know. That's still not very high as far as I'm concerned. And you've proved my fears about your PR right there.
[Update 2] I found Nova's comments really asymmetrically threatening and, frankly, quite crazy. Nevertheless, I'll publish his [one] coherent Twitter post as it exposes the way Nova looks at the data: "Audience increased by 17X but avg pageviews decreased by 4.5X for unreg'd, and 1.6X for registered. Hmm. Not so bad, Greg."
That's ok to have this opinion, I'm not saying I'll "never do an interview with you again" b/c of it, Nova. Bottom line is, growth or no growth, if engagement metrics go down like they did, I wouldn't dismiss it and I'd be trying to figure it out. Not go crazy on a small consultant out of Canada, because he's not concluded what you were hoping to hear from all the data you sent him, which he fully published for anyone to make up their own minds. Seriously...
PART 1
[Greg Boutin]
Nova, I’d appreciate any element of information on the following. As
previously mentioned, I’d like to insist that you put something in
writing, just b/c it will be clearer and will give me less margin to
misinterpret what you said. If you don’t want to see something
published, don’t share, or let me know the rational.
I understand that Twine is work in progress and that you may not want
to share those metrics for competitive reasons, however I would
encourage you to address the concerns around engagement, since they are
in the public domain at this point, and I plan to blog about it,
leveraging any information I can get.
- Engagement metrics:
- pages/visits
- avg. # visits per 30 days
- #clicks per visit
- active clickstream time within a session
- the metric “time spent on site” is a poor one unless it is “active time spent on site” shown by click rates
[Nova Spivack] Metrics will probably be next week at the earliest as we have had our hands full this week with other things.
- Usability / focus on relevance
- what are you doing to remove duplicates / reduce #items on homepage and email, beyond enabling twine removal
- timeline?
[Nova Spivack] The number of items on the homepage is a function of how many twines
you join, and how many of those twines you then choose to display in
your interest feed. If you keep up with your twines and use the “Catch
Up” feature on individual twines or “Catch Up All” it reduces the items
by removing those you have read. This is similar to Google Reader.
Beyond this however we have a completely redesigned version of the
Interest Feed that will go live next month and will solve many
usability issues with the current version. This will include a truly
scrollable list of all your twines on the left that will make it much
easier to select what you want to view. You will also be able to drag
and drop twines to make your own preferred order in the list. The look
and feel will be improved significantly as well, and this is just the
beginning of the improvements we will be making to the Interest Feed in
future releases after that.
As for duplicates, we actually do have algorithms that remove
duplicates but once in a while there glitches. There is also a case
where it is non-trivial and where non-removal is actually valuable.
Namely, if an item is cross-posted to several twines that is important
context and so we don’t remove the item – if we did remove it people
might then think it wasn’t there and re-post it to those twines. One
solution would be to view things by item rather than by twine – for
example a list of items and then under each item, links to any twines
they are in. But then you would not be able to easily view the items by
twine. This view exists and is optional – click on “Newest First” above
your interest feed.
[Greg Boutin]
-
will there be a relevance metric that cuts across my twines, so I
can rank all items by what’s most relevant to me, based on a match with
my “most used” or expressly declared tags
[Nova Spivack] We’ve thought of things like this, but for starters, we are working on
rolling out a Most Popular algorithm on the site as a whole, and for
each Twine. Timeline is not firm yet but will be in next few months.
As relevance ranking, we compute that automatically for each query
(it’s a sort order you can choose), and in our recommendations. We
don’t yet have a plan for exactly the feature you suggest.
[Greg Boutin] Thoughts on signal-to-noise ratio
You’ve shown to favor a more encompassing approach to displaying
recently twined items and twine items of possible relevance, as opposed
to showing less information and making it easier to digest that way
- what is your rational for that approach? Do you have clear stats
showing that users prefer exhaustiveness over higher relevance? (keep
reading, I explain why I see a trade-off between those 2 dimensions)
- is that a technical limitation? If so, what is it?
- I imagine you might dispute the above statement. In that case, how
do you rationalize digest emails that contain over 100 items when one
is subscribed to a number of twines. How does that fit within your
usability objective?
[Nova Spivack] The ultimate solution is to have a system for featuring the most
popular items, and then for a phase 2 to enable a Digg-style rating
system within each Twine. This is our plan. Ultimately this is best
since it does not require an editor to insert their bias – the
community can filter their own content to surface what they view as
most valuable.
[Greg Boutin] The key concern driving my question is an observation that, even
though it is incorporating user feedback, Twine is putting more
emphasis on pursuing initiatives that are geek-centric and as result prioritizing
and focusing more resources on those, such as establishing various
kinds of connectors and expanding the incoming sources of data at this
stage, than offering a “tighter” experience to users, with the key
benefit being a smaller set of more relevant items (or at least the
ability to ask for that).
[Nova Spivack] We actually are making product priority decisions based on survey
results that random sample our users about what features are most
important. Our surveys support that what we are focusing on is in line
with user needs. That said we also have our own priorities to link
Twine with other apps. There is much to improve in Twine, but we really
have a very small development team working on the front-end app. In
every release we must make difficult tradeoffs. There are still many
very basic general features we need to add or improve before we work on
more advanced capabilities or edge-cases.
[Greg Boutin] Focus on discovery
The fact that Twine, along with other “semweb” apps, is positioned
on exploration and discovery gives me the impression (and I’m also
channeling that from others) that it is constrained in its ability to
serve highly-relevant content; and I’m not sure that constraint is
real, and that you realize you’re giving that impression, at least to a
number of your users. I strongly suspect that focus on discovery hurts
you, and will increasingly do so, because my assumption is that the
folks in the next segment you have to target (not tech early-adopters –
see crossing the chasm) equal “discovery” to a worse signal-to-noise
ratio. I don’t know whether they’d say or not say that in a survey, I’d
love to test it and probably will (not specifically on twine, just on
key latent needs), and suspect that it could surface in anthropological
tests if it doesn’t in surveys. Bottom line, it should be tested. Have
you and can you share stats?
[Nova Spivack] The main focus in our current positioning is not discovery but
“interest tracking.” That has been working to some extent, but we need
to really expand that in a more consumer-friendly explanation. We’re
working on that. Twine as an application and the underlying platform
can serve highly relevant content – as our recommendations illustrate.
We will be enabling users to view their recommendations, as a distinct
feed, rather than just some, in a future release currently anticipated
for late spring or early summer if not sooner. That will be more
tightly relevant. We hear from many users that they are very happy with
the novelty of the items we are delivering via recommendations, and in
the case of twines we also hear they are generally quite happy with the
content they are getting. In fact, content relevancy has not even
registered on the list of user-feedback priorities we get – although at
least in one area, namely search, we have had a lot of feedback that
users would like improvement.
[Greg Boutin]
I sense that if an app came along that offered the ability to
filter shared content so it serves a much smaller set of documents
across my identified poles of interest more closely matching my profile
and needs, it would make Twine less relevant. Is this something you
have been thinking of this way?
[Nova Spivack] Again, the social filtering features of Twine are not developed very
far yet, but we do plan to enable much more there this year. We’re not
terribly worried about competition at this point, given how hard it is
to do this stuff and get traction. There will inevitably be
competition, but I think we are on the right track.
[Greg Boutin]
Let me put that in other words: focusing on “Discovery” gives you
leeway to show a bunch of content, some of which is less relevant. That
leads you to focus on a lower signal-to-noise ratio than I would
advocate, if you are to resonate with users, especially in the more
mainstream category. As a result, I suspect a majority of potential
Twine users are or will be turned away.
[Nova Spivack] I think StumbleUpon’s success might be an example against your argument
on this point. They are pretty big and I wouldn’t say they are
extremely focused or relevant – at least not to the degree you are
asking for. Consumers seem to like serendipity. Most consumers are not
getting RSS feeds and what Twine delivers is pretty unique for a lot of
them.
[Greg Boutin]
Dreaming for a second, the ultimate Twine would basically deliver
whatever information I need at the precise moment I need it. Note that
sounds very utilitarian, but it may be that the information I need at
that precise moment is to help me learn about a topic.
[Nova Spivack] That would require a very deep understanding of real-time user intent.
That is difficult to infer if users have not taken any action to
demonstrate their context or intent, or without getting access to their
attention data somewhere else. Where users have taken an action – by
sharing, searching, connecting, joining twines, collecting, etc. –
Twine does learn and this personalizes the recommendations they see.
[Greg Boutin]
However, the focus is reversed: right now Twine’s items mostly help me
keep on top of things in a somewhat unfocused manner. If I seek
something specific, I go to Google, not to Twine. My hope is that Twine
gets to know me and develop a picture of me based on my activity and,
maybe, declared needs which I could update daily (e.g. today I’m trying
to build a marketing plan for a restaurant, Twine please serve me
documents that will help me do that)
[Nova Spivack] Google does a good job at search. Twine is not trying to compete with
Google or to be a search engine. We are trying to do what you suggest
above, but can only do that from actions the user takes in Twine. And
this point our capabilities are not (yet) as futuristic as what you
propose. But we’re headed in that direction at least.
[Greg Boutin]
It would not serve me a large set of information by default and
let me do the filtering. By default, it would serve me the tighter set.
[Nova Spivack] That would of course be great. Right now we are serving information at
the “twine-level” and at the “recommendation-level” – what you are
proposing is essentially synthetic twines that are dynamically
assembled and personalized for particular users. It’s something that
would be cool, but we have to solve more basic challenges first.
[Greg Boutin]
My assumption is that with your current technology you could crank
the relevance threshold up so I see less items, but of higher relevance.
Is that a fair assumption or technologically there is still a challenge?
[Nova Spivack] Yes, but only for recommendations and search, not for twines. Twines
show content that users have contributed. In the future, when a twine
is large enough we could certainly analyze it and compute what is more
“central” to the cluster of concepts the twine is about, and perhaps
allow users to filter that way. That is not rocket-science, it is an
application of known machine-learning techniques. However, it is again,
a more advanced feature and we have had no requests for anything like
this (until you). I like the idea though and it would be good to
perhaps think about for later, especially as even more users (and
content) go into Twine. Certainly it would be a nice way to help combat
information overload.
[Greg Boutin]
In that case, I’d guess you should devote more resources to that
problem first before tying up twine to the rest of the world (even
something as cool as twitter-on-twine)
[Nova Spivack] Again, from our perspective, the Interest Feed is the number one
feature users want improvement on, and search is also high on the list.
Advanced filtering is not currently a widely echoed request among our
users, yet. I imagine it will become more important in the future.
[Greg Boutin]
More broadly, a recurring theme often neglected in tech startups
is how they become “potential user-driven” as opposed to just
“user-driven”. “User-driven” can take you in the wrong place, by making
you respond mostly to requests from the early adopters type, who often
favor the technology over its usability, and want to see a lot of
features, often at the expense of solving widely shared problems. On
the other hand, focusing on groups of users you don’t have yet will
often lead you to prioritize usability and simplicity, over more
technology-driven and/or overly-nichy improvements. Is this stg you
thought about at Twine and if so how do you/will you tackle it?
[Nova Spivack] Exactly – in fact, some of the functionality you are asking for is what
I would view as “power user” or “tech early adopter” functionality.
Mainstream consumers are actually not asking for that. They want
something that is simple and easy to use and it is hard for them to
even understand auto-tagging, let alone advanced statistical clustering
and automatic filtering etc. We have to focus on the mainstream users
and what they need first. That is often more around simplicity and ease
of use, and integration with other tools they already use, rather than
next-generation artificial intelligence.
[Greg Boutin] Can I reproduce this email on my blog?
[Nova Spivack] Feel free to share my comments.
PART 2
[Greg Boutin] This is helpful, thanks Nova. I look forward to the metrics. Some follow-up comments:
[Nova Spivack] Can’t spend a lot more time on this, but here is my reply to below. I hope that will be enough.
[Greg Boutin] User-driven development
You say that beside connectivity with other apps, your current users drive feature development. Then you say you want to address the needs of mainstream users first. But your user base doesn’t strike me as mainstream, it’s much more geared towards geeks. One just has to take a look at the top twines http://www.twine.com/explore-top-twines to see that a majority of the topics in there would be of more interest to a Leo Laporte than to Joe the plumber, even an educated Joe. The changes you’ve been making to Twine have given me and some thought leaders I talked to the impression that you are in fact not focused first on targeting that more mainstream base. Which is inevitable if you let current users drive the development (still a step above most start-ups in the space, I’ll give you that) as opposed to potential future users. Your response hints to the fact that you don’t have any mechanism to listen to those potential users. If you do, please elaborate.
[Nova Spivack] Our current users are trending more mainstream and that is where we want to go with the product. We base our thinking on very detailed survey research that we do every quarter as well as in-person user-studies by anthropologists and more. If you remember the beta, and you look at the product today, you can see the progress towards a more mainstream, easy to use product from where we began. The next version, coming soon, will reflect even further learning and improvement towards usability by general audiences – particularly around the Interest Feed, initially. There will still be further sections to work on however in future releases – like making the summary pages of twines better, improving the My Items section, and more. Again, we release iteratively on an ongoing basis. It will take a year to make all the features and improvements that are currently planned. So let’s check back next year on this and see where we are.
[Greg Boutin] Duplicates
As an illustration of the above, it strikes me as odd that you think people would rather see duplicates across all their twine (I haven’t seen those algorithms that remove duplicates – when did you activate them?) than have them removed. I’ve heard that from you months ago and to be transparent, I don’t buy it and don’t think it comes from a survey, but from your own preferences and that of early adopters perhaps. I have a hard time believing that a survey of mainstream users (and even actual users) would show that, and believe in any case that this survey should mention the obvious solution: simply put a mention that duplicate items were removed and give an option to add them back… Which would lead to a much less crowded and much more usable page. Having said that, I’ll give you credit for the “Newest First” view, which I had not experimented with as it seemed to suggest just a simple time-ordering by twine to me. And I’ll take half of that credit back for not making “Newest first” the default view, and not removing the duplicates automatically in the twine-grouped view :) It’s not the relevance-weighted cross-twine view that I think would work best, but it’s definitely the best step towards it given the state of the technology as you describe it.
[Nova Spivack] No survey of ours has formally addressed the particular question of duplicates, however at least some users have chimed in on this in the past through unsolicited comments and discussions and feedback. The question has not come up to an extent that we had considered it as a survey question. Very few users have brought it up. I understand that it is a particular peeve for you however. [Post-interview note from Greg: I just searched for "duplicates" in Twine and there are 125 comments, many asking why this issue is not addressed in spite of the amount of feedback. A number of people have also raised the issue with Nova. Andraz Tori of Zemanta recently commented on twitter: "dupes only problem inside Twine digest. Everybody is screaming about that for half a year now. Nova knows." See the comments here for a user who pointed it out a year ago.] In any case there is already de-duping to some extent but there are probably places to add even more intelligence to the feature.
[Greg Boutin] I, the Power user
(quoting Nova) "Some of the functionality you are asking for is what I would view as “power user” or “tech early adopter” functionality. Mainstream consumers are actually not asking for that. They want something that is simple and easy to use and it is hard for them to even understand auto-tagging, let alone advanced statistical clustering and automatic filtering etc. We have to focus on the mainstream users and what they need first. That is often more around simplicity and ease of use, and integration with other tools they already use, rather than next-generation artificial intelligence”
This feels quite rhetorical, but you might have misread me. Quoting myself, I asked Twine “to show less information and make it easier to digest that way”, “a better signal-to-noise ratio” and a “tighter information set”. Simplicity and ease of use, that is. An email digest that has 100s of items with tons of duplicate is neither simple nor easy to use. And as you yourself states across a number of interviews, mainstream users don’t have to understand the technology that’s behind the hood. I did not ask you to show any of the statistical stuff. Just give users an option to have that cross-twine view and call it “Twine Magic Sauce” and I’m sure everyone will buy it. Who cares whether Google’s algorithm is black or white as long as it catches the mouse?
[Nova Spivack] It’s a good suggestion for an improvement – some kind of Best of My Stuff view perhaps, daily. The algorithm could perhaps show the most popular items you received across all your twines, and deduped. Perhaps with some threshold for number of items you want to see and/or level of popularity you care about.
I like it. I would read that every day! I’ll give it more thought and then perhaps propose as a feature for later.
[Greg Boutin] “Again, from our perspective, the Interest Feed is the number one feature users want improvement on, and search is also high on the list. Advanced filtering is not currently a widely echoed request among our users, yet. I imagine it will become more important in the future.” Wait, you say that the interest feed is the top feature to improve? Isn’t that what I kept asking for? That has everything to do with what you call “advanced filtering”. But of course, if you call it “advanced filtering” and make it an obscure option, that won’t fly, and I’m not surprised it’s not widely echoed yet. Just give a “newest first” option for email delivery to start, or remove the duplicates automatically, and make it standard (with an option for geeks to revert back to their old system).
[Nova Spivack] I think the above solution would work for this too. Basically you are asking for a new view of the content – not “by twine” or “by time” but “by popularity” or “by relevance to me”.
I’m still concerned and a little frustrated about your reaction to all this, but I definitely appreciate the reply, Nova.
PART 3
[Greg Boutin] (commenting on Nova's replies above) The detailed survey you do is conducted on usage data and current users, right (you can just reply if it’s not, else no need to)? As for the “particular peeve of mine” again I think this is one of those issues where people just see the results and turn away. I use that as a bit of a pet proxy to assess usability and responsiveness, as it’s such a striking lack thereof, but I’ve heard it from a number of KOLs, not just me. Really. I look forward to the stats.
[Nova Spivack] There’s not much else to say about that Greg, other than what I’ve
explained. Users have not expressed a significant concern with
duplicates in Twine (Post-interview note by Greg: see my previous note on this, in part 2). Many other features are far higher priority to the
users who have contacted us. That is not to say there is no room for
improvement. I agree there is, but according to our research it is
lower on the (big) list of priorities than other improvements users
want. It’s helpful of course to hear your thoughts on this and we
certainly will take them into account in future design discussions.
[Greg Boutin] And again, you’re focusing on users as opposed to potential users, and expressed needs as opposed to latent needs. As I keep saying, I’m noting your marketing really does not account for potential users, who have very different needs from existing users, and relies on surveys rather than empathic research. I’m not asking for more comments on this, it’s fine, I got it.
PART 4 - HARD DATA!
[Nova Spivack] Hi Greg here is some data you can cite [see table below]. The reason for the slight drop in time-spent is because the unique and pageviews have increased so much, which throws off our averages. Feb data is not in yet so not included. This is from our internal logs.
[Greg Boutin] Thanks Nova, this is interesting. Are you planning to send me more stats, around engagement in particular as I’m hoping to see (especially #clicks per visit / active clickstream time within a session, the metric “time spent on site” is a poor one unless it is “active time spent on site” shown by click rates) or will this be it?
[Nova Spivack] Greg the level of additional data you request is not easy to produce and in any event we haven’t released that to anyone yet. So this is it for now. Have to focus on our work now. This is what we are releasing publicly now. Once we have many more months of data we may drill in and release more publicly. At this point it really is too early to draw any conclusions, positive or negative.
[Greg Boutin] Alright, I needed to know whether I should wait for it. You should easily have access to total visitors per month though, which would be useful.
I’ve calculated the #page views/unique visitor and it’s been declining rapidly, see the row I added. I will be commenting on that and qualify it in the context of the launch, but derive my own conclusions, I don’t want it to come as a surprise, and I imagine that your answer will be the same as below. If not feel free to elaborate or send further data. I’m ok with waiting till next week if you need a bit of time there.
PART 5 - HARD DATA, CT'D
[Greg Boutin] Nova, I’m about to publish the post, a week has passed and I haven’t heard back from you on the #page views/unique visitor declining from 16 to 2 in 7 months. I’d assume you’d want to comment on that. If so, please do so asap.
[Nova Spivack] Let me talk to candice and get back to you this am. But my first reaction is that your simple back of the envelope averages are not very meaningful. Of course as the number of uniques goes way up as it has there will be many first time and one time visitors that skew the average. But for repeat users and active users in fact pageviews are higher.
[Greg Boutin] [Post interview note: this actually is quite an important metric for any web CEO... It shows that while Twine is going mainstream, its "stickiness" factor declines rapidly. A fast-growing number of people just don't engage with the app at all. If I were at Twine, I wouldn't dismiss that as normal. I'd want to know why that is. True, it can be expected that a fast-growing app would experience a decline in page views per unique visitors, but that decline looks dramatic and comes on top of a number of users complaining about their feedback being ignored. Focusing on just the current power users leaves a lot of potential users out]
[Nova Spivack] We can get some data on this that shows what pageviews are really like. Different from what you have estimated.
[Nova Spivack] A few other points – Twine went 1.0 in October. Any data from before October 2008 is not an apples to apples comparison to data or trends after that date – because prior to October 2008, Twine was in private invite-only beta testing with a small group of invited users only. The behavior and metrics of a small group of private beta testers is quite different from that of a public site. Twine has multiplied in size dramatically since the private invite-only beta. It does not really make sense to say that our engagement has declined from what it was before we launched the site publicly in October. Prior to October, our metrics were just for actively engaged registered users in our beta test. Since October, the majority of users of Twine are non-registered guests, for example those who visit directly from search results in Google, etc.
[Nova Spivack] Greg, here is the latest data. February is not over yet, but we have the current numbers as of today. Also we have added in page views per registered user and page views per unique according to our Omniture data.

Recent Comments