MY FIRST JOB involved a deep dive into Big Data but nobody called it that in 1976. Instead, I toiled in the basement of a data archive, trying to figure out if aviation accidents would happen if newly rated pilots failed to complete recurring proficiency flights.
In my first Big Data exercise, I correlated questions in manila folders to answers spread across computerised records of aviation activity. To get the answers, I had to request computer runs that culminated in a regression analysis in support of a controversial decision. It took three months to crank out the answers.
Today, I could accomplish the same task in less than one work week because I can leverage Big Data. And that treasure trove of data isn't merely a summation of flying hours, it also includes machine-generated data from cockpit cameras, gun camera footage and flight simulator profiles. All these data points are compiled when teaching advanced flight training courses.
I also have access to large swaths of data in my current job as a creative multimedia lecturer and can also use big data analysis to predict drop-off points in an Honours degree programme. Some of those drop-off points represent customer behaviour in the marketplace. In their July 2014 issue, Inc. Magazine featured an article detailing how smaller companies are now allowed to play as well, thanks to decreasing technology costs and increasing user-friendliness of big data software.
So if big data tools are a touchscreen away, do we need to train data researchers or education analysts? Big Datasolutions are so user-friendly that my old Air Training Command, former sales companies and my current education establishment could easily glean all the student and consumer insights without specalised analysts. I don’t think proper data analysts will be replaced anytime soon because their analysis can demonstrate causality (what has happened) but often cannot explain why something has occurred.
Chadwick Martin Bailey cites the widely-covered 2013 Google Flu Trends “Epidemic.” "By running algorithms based on flu-related Google searches and searchers’ locations, Google Flu Trends had been historically accurate in predicting how much of the U.S. population had the flu. However, in 2013, it inaccurately predicted the number. In fact, it predicted twice the number reported by the Centers for Disease Control and Prevention! How did this happen? The widespread media coverage of the severe flu season in the U.S. spread like a virus throughout social media, which led to an increase in flu-related Google searches. Many of these searches were from people who thought they might have the flu—“I’m sniffling! I’m sneezing!”—but didn’t. Since Google Flu Trends didn’t consider the context and wasn’t able to ask Googlers why they were Googling flu-like symptoms, it thought 11% of the U.S. population had the flu when the actual number was closer to 6%."
Mark Hansen of Columbia University summed it up best when he said, “Data is not a magic force in society; it’s an extension of us.”
In my own little world, I follow a slice of Big Data. I follow Twitter Analytics. Those points on Twitter's graphs tell a story of my week. Twitter Analytics tells part of the story of my online life. But to truly understand the major peaks and valleys of my strongest points "engagement" on Twitter, you have to know the context of my engagements. You cannot simply look at a peak of activity and surmise exactly what caused a major uptick by unwrapping the links I mentioned, the conversations I entered or the images I shared. In nealy every case of higly engaged Twitter activity on my account, I was physically present at a meeting, confrence or a social event. My physical presence provides the context underscoring major periods of engagement on Twitter. Glancing at Twitter Analytics alone fails to show the whole story.
I also use engagement analytics through Mailchimp and have learned not to trust merely the action of a mail arriving, content being scrolled and links being clicked inside the e-mail. There's often a very personalised reason why some students do not engage electronically before they fall off the curriculum. Mailchimp analytis may point to what is happening but not why the loss of engagement has happened.
Looking back on my own working life, I can point to several success stories that some people wrongly attribute to Big Data alone. However, I believe the best results accrue when combining data analysis with contextual awareness. When we can teach both the analytical skills and the imperative of context, we help produce a better suite of options for management.
Chadwick Martin Bailey -- "Big Data: We've Only Just Begun" in Loyalty360, September25, 2014.
