Big Data has been pushed pretty heavily for the last year or so. The idea of taking large data sets and crunching the information to find new connections is going mainstream.

Why is this, and why now? Well, Big Data is a great way to use the ever-cheapening power of servers, and it’s a very good fit  with the cloud. Running a Big Data exercise needs a lot of power, and the new Big Data publicity suggests using it in a pretty ad-hoc manner - so you may well need a service where you can spin up servers and rent them by the hour.

Analytics used to be a precise discipline, where academic precision persuaded large organisations to hand over big money, because only rich firms could afford the hardware required to generate exclusive results that could give you an edge over the competition.

There has been a lucrative, but limited market for analytics, but now vendors like IBM and HP think that analytics is ready for the big time. Why else did HP buy Autonomy?

As analytics gets sold down-market, it’s taken on a more approachable name: Big Data. But analytics needs more than a name change when it emerges from the glass-house. .

Users have to be convinced they want to use Big Data. They need examples of things it can do, that are actually useful, or else they will be entitled to ask: It may be Big, but is it Clever?

The trouble is, most case studies of Big Data aren’t clever. The tales are nebulous and anecdotal, and they tend to describe magic answers from the Big Data systems which are either completely obvious or else of only minor value. They usually sound spurious and made up - and can even sound intrusive.

One well-worn tale has it that a supermarket ran a tonne of number crunching, analysing buying patterns, and wound up with the suggestion that it should put beer by its nappies (diapers). Dads buy the nappies, apparently, and might buy more beer if it was conveniently placed to jog their memory.

If that story is true, how much actual benefit did it produce? More than the downside caused by placing beer in two places in the store? Or lost business when some shoppers missed the nappies altogether (“they can’t be down here, this is the beer aisle”)?

In a creepy and unconvincing story, a store that started to offer a woman childcare equipment in its regular spam mails (sorry, marketing communications). She thought the system was failing, but she found out her teenage daughter was pregnant, and the store had worked it out before she did. The details of this one either suggest the computer worked it out from what the daughter was buying (on the family account?) or else it cross-referenced with medical records (which is surely illegal).

I get bored with this sort of Big Data folk-lore. I’d like something more concrete. And it turns out that a lot of concrete applications of Big Data may actually come from the Green field.

Intel is investingin a project at two London universities (University College, and Imperial College), to use the Big Data generated by the urban environment, to seek ways to make London greener - cutting out transport and energy waste.

The ideas from Collaborative Research Institute for Sustainable Connected Cities (CRISCC) should be applicable to other cities, and the project could make Intel a player in the Smarter Cities movement, alongside IBM (which has put its seal on a lot of this sort of work in its Smarter Planet marketing). 

CRISCC could also produce some early and verifiably useful results from Big Data. Efficiency is all about shaving tiny amounts off a quantity like consumption of energy and materials - and this is the kind of task analytics has always been good at.

But will the shavings add up to savings though? Only time will tell.