Validation & Accuracy

We had a problem recently. Our Campaign Tracking application experienced a bug that resulted in our failing to collect a handful of tweets that rightfully should have been found. Once the problem was identified, we were able to restore almost all the missing tweets. We estimate having “dropped” between 10 and 20 tweets from the campaign, which isn’t too many, but still isn’t perfect.

I was very upfront about the issue with the client whose campaign experienced the problem. They expressed their concern and asked me what I could do to give them more confidence in the numbers GraphEdge reports (because they report those numbers back to their clients).

As a result of this, I decided to implement a Quality Assurance program: we would pay for a competing application to periodically validate that our campaigns were capturing the complete set of tweets at the most basic level of complexity—a single-keyword campaign. My idea was that if we can be sure we’re capturing the full scope of those, then we can internally validate all the fancy features inside of the GraphEdge application (multiple keywords, user-panel restriction, negative keywords, etc).

It was  a great plan, but it relied on one assumption—that a perfect competing application is out there. So far I’ve tested 3 competitors (to remain unnamed!) but rather than validate our own application, all I’ve been able to do is invalidate theirs. So far each of the three I’ve tested has failed to find all the tweets GraphEdge has, and in one case the competitor routinely captured as few as 20% of the daily tweets GraphEdge did.

So it’s nice to not have invalidated our own application, but the problem still exists: how to validate it to satisfy customers? Given the apparent lack of accurate data, is there an “accuracy” sales/marketing angle I can take to close new business? How do other companies satisfy their clients that their data is accurate? Do they care!?

Went looking for answers, but just came up with more questions.

Conversation Discovery now in GraphEdge Pro

Now you can get a sense of how much buzz your campaign generates beyond just the results of keyword searches!

GraphEdge Pro now discovers tweets (entire threads) that are in reply to campaign-relevant tweets, regardless of whether the reply contains a campaign search-term.

These “conversation-tweets” and their authors are integrated into the standard Campaign reporting, and are also reported separately, with all the stats you’d expect to see from a GraphEdge report!

There’s a new “Conversations” tab you can use to view threads, hierarchically rendered. When you manually “exclude” a tweet as being irrelevant to a campaign, the entire part of the thread that is in response to that tweet is also excluded (or re-included, if you’re restoring a tweet).

We’re really excited about this one…. multiple clients are already making use of this new feature. It’s very cool!

Please reach out if you’re interested in adding Campaign Tracking to your existing GraphEdge report, or if you’re interested in learning more about becoming a GraphEdge client. I’m offering free trials of Campaign Tracking for a limited time, so hurry!

Brief Outage

We suffered a brief outage this morning/afternoon for GraphEdge Pro (http://agency.graphedge.com). No data was lost, and we’re back up and running. Please contact support at graphedge dot you-know-where with questions.

Followers’ Followers vs. The REAL 2nd Level Reach

A client and I are in the middle of a discussion about total second-level reach. The question is: how many total followers do my followers have? The answer is far less straight-forward than most analysts/agencies would have you believe.

A naive approach to this question would be to look at each follower and count the number of followers those users have. The problem with that approach is we double/triple/etc. count any individuals that are jointly followed by two/three/etc. of the followers. Many analysts will ignore this as being insignificant and just report the big number. Much easier. And completely wrong.

GraphEdge is less concerned with who is following a set of followers, and more concerned with who else the followers follow. That is, we look at the other people your followers follow, rather than who is following your followers (I know, it gets complicated!). However, from what we do calculate, I can tell you that it is not sufficient to take a total of followers’ followers and report that as second-level reach.

The reason is that it ignores “centrality” (for you academics). Imagine that me and ten of my friends are all following each other. My ten followers will each be following ten people. Add that up, and that’s 100 “connections”. But we know there are really only 10 unique individuals in that extended network (not counting me). That’s a 10% uniques-to-connections ratio.

Centrality tends to increase as the size of the network increases. So growing your network has marginally reduced effect on the size of the extended network.

To demonstrate this effect I took a random sample of 700 or so active GraphEdge accounts and looked at the number of followers, the number of connections to followers’ friends, and the number of unique names the followers were following. Note: for accounts with more than 10,000 followers, we generally take a sample of the followers, rather than looking at each follower’s network, so to keep the numbers clean I limited my sample of 700 accounts to those who had less than 10,000 followers.

This chart shows the total connections (blue dots), given a number of followers (on the X axis), and the number of unique individuals (red dots) in the set of followers’ friends. Notice how the number of unique individuals grows at a much slower rate than total connections, as the number of followers increases.

For example, let’s look at that first pairing at around 9K followers. The blue dot (the highest in the chart) says that those 9,000 or so followers had a total number of connections around 8M (close to 900 “friends” per follower). However, from the red dot directly below it, we can see that there were only 2M unique people in that list of 8M connections (more like 250 unique names per follower).

And this next chart breaks it down by the ratio of uniques-to-connections, given the number of followers.

Our friend with the 9,000 followers had a unique-names count just 25% or so of the total number of connections.

So if someone asks for a summed-total of Twitter followers’ followers: go ahead and present the number, but know (better yet, explain) that the real reach may well be just 30% of the number reported… or less! And if you need to calculate either one of them, let me know, I can help you there!

Too Many Notes

Was bound to happen… by drawing every available data point into our graphs we’ve exceeded the maximum length of a URL, so charts for certain very-long-term accounts aren’t drawing. The fix is simple in concept but less straight-forward than one might think. It’ll give us something to work on tomorrow!

Beta Testing Agency Tools

Beta testing some new tools for the Agency version of GraphEdge. If you’re planning on running a campaign on Twitter in the next week or so, drop me a note, I’d love to have GraphEdge track it for you, free of charge.

Recent bug and fix

So I’ve been so focused on our agency-related products, that I missed a pretty big bug in our public-facing site.

The issue was that our updates weren’t being sent if people didn’t explicitly set their update-frequency in their account settings. Then I fixed the bug a couple of days ago. This had an unfortunate side-effect for a certain group of recent Free Trial users: if they signed up more than 14 days before the bug was fixed, their free trial expired before they received their first update. So they’d click the links in their @messages and the report page tells them they’re expired, they’ll have to pay. That sucks!

This has led to confusion and, in some cases, mistrust. I can understand that. If you’ve encountered this bug, I’m very sorry for the inconvenience. I’m trying to find a solution that’ll let these ‘limbo’ Free Trials continue. Stay tuned.

Good but Not Good Enough

I’ve had a survey.io survey available to some repeat customers for a short time. Results are good (even though the sample size is small), but not where I want the product to be.

The key question in a survey.io survey i: “How would you feel if you could no longer use [product name]?”

Current results for GraphEdge are:

• Very disappointed: 15.4%
• Somewhat disappointed: 76.9%
• Not disappointed (it really isn’t that useful): 0%
• N/A – I no longer use GraphEdge: 7.7%

For Lean Startup/Customer Development gurus, this one question is the main metric for product-market fit in young products, much preferred over it’s ugly cousin, the misleading “Net Promoter Score”.

Marketing guru Sean Ellis visited Dogpatch Cambridge back in December and told us he won’t even start marketing a product until it has 25%-40% “very disappointed”. So while these scores look pretty good, it’s not really where I want the product yet.

Of course, this is a poll of the public use of GraphEdge, not of my PR- and marketing agency partners, who are overwhelmingly positive. So I’m not terribly disappointed to have a “would be very disappointed” score of under 30% from the general public… but I’d rather it be better!

Regarding Net Promoter Score: question 5 asks: “Have you ever recommended [product name] to others?” In this question, GraphEdge scores a whopping 77%. Viewed on its own, this would perhaps give a too-rosy picture of the product’s fit to market. Combined with my “how disappointed” score, however, this tells me that I’m doing better than the competition, but failing to delight. Going to work on that.

New Feature: Raw Data

Very pleased to announce a feature that people have been asking for/about for some time now: access to raw data.

As of today, you’ll be able to get tab/return-delimited text of the raw data that feeds the daily GraphEdge stats/charts, including:

• Date
• Follows
• Unfollows
• Net Change
• Total Followers
• Legitimate Followers

To get the data, click the little “data” icon in the headings of your Followers History or Network Change History blocks in your report.

Doing so will launch a specialized Report containing a Historical Data section, in which is a text box containing the data itself, which you can select and copy into your spreadsheet of choice.

Formatting might look a little funky as presented in the text box, but it’ll be fine once pasted into your external application.

Note that Follows and Unfollows (and hence, Net Change) all apply to Legitimate Followers only. Since users are constantly being reclassified as illegitimate (or legit… it could happen!), these numbers are subject to change day-to-day. For example, let’s say Twitter account @SpamBot starts following you on Feb. 14. He’ll show up as a Follow for Feb 14, until his account is canceled by Twitter and GraphEdge reclassifies that user as illegitimate. After that happens, the user will no longer show up as a legitimate Follow in historical data.

All reporting data is updated nightly.

Thanks to Erin Hollabaugh and @StaplesTweets for lighting a fire under our butts to get this finished & rolled out!

Dispensing with Free Trials

Decided today to dispense with free trials. There are a number of factors which lead to this decision, including:

• Much of the value of the report is realized the first time it is viewed… reducing incentive for conversion to paid service
• The free trial has been very popular, and with a low conversion rate, our limited capacity is expended servicing free trials instead of paid customers
• We’ll be undertaking some new products/services/reports in the coming days/weeks and we need that extra bandwidth to support that effort

For potential API or Agency partners, we may relax the no-free-trials policy, please reach out individually if you represent an agency or third party application interested in possibly folding GraphEdge data and/or reporting into your company/service.

Stay tuned for more information about what we plan to do with all this extra bandwidth!