Validation & Accuracy

We had a problem recently. Our Campaign Tracking application experienced a bug that resulted in our failing to collect a handful of tweets that rightfully should have been found. Once the problem was identified, we were able to restore almost all the missing tweets. We estimate having “dropped” between 10 and 20 tweets from the campaign, which isn’t too many, but still isn’t perfect.

I was very upfront about the issue with the client whose campaign experienced the problem. They expressed their concern and asked me what I could do to give them more confidence in the numbers GraphEdge reports (because they report those numbers back to their clients).

As a result of this, I decided to implement a Quality Assurance program: we would pay for a competing application to periodically validate that our campaigns were capturing the complete set of tweets at the most basic level of complexity—a single-keyword campaign. My idea was that if we can be sure we’re capturing the full scope of those, then we can internally validate all the fancy features inside of the GraphEdge application (multiple keywords, user-panel restriction, negative keywords, etc).

It was  a great plan, but it relied on one assumption—that a perfect competing application is out there. So far I’ve tested 3 competitors (to remain unnamed!) but rather than validate our own application, all I’ve been able to do is invalidate theirs. So far each of the three I’ve tested has failed to find all the tweets GraphEdge has, and in one case the competitor routinely captured as few as 20% of the daily tweets GraphEdge did.

So it’s nice to not have invalidated our own application, but the problem still exists: how to validate it to satisfy customers? Given the apparent lack of accurate data, is there an “accuracy” sales/marketing angle I can take to close new business? How do other companies satisfy their clients that their data is accurate? Do they care!?

Went looking for answers, but just came up with more questions.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: