What exactly is big data capable of? The following is a well thought out article outlining the basics of big data along with the reasoning as to why its more tangible than you may think.
Cheers!
~The TPUServices™, LLC Team
10 big myths about Big Data
By Maria Korolov, Network World
Network World - Big Data has dominated tech news of late. It
has been touted as a possible solution for everything from intrusion detection
to fraud prevention to curing cancer and setting optimal product prices.
But Big Data, which we’re defining as data collected in
large volumes, at high velocity and in a variety of formats, isn't a cure-all
for every problem. In fact, if companies that believe in some of the myths
surrounding Big Data, could head off in the wrong direction, waste a lot of
time and money, cost a company its competitive position in the market, or
damage a company's reputation.
Here are some of the biggest myths surrounding Big Data.
MYTH 1: Only data scientists can deal with Big Data
In fact, data scientists by themselves are not enough.
“Data scientists by themselves aren't going to be able to
pull off getting the information out of Big Data if you don't know what you're
looking for in the first place,” says Pat Farrell, senior director of data
analytics at Penn Medicine. “You need people who are familiar with the
industry, the domain of knowledge, understand what kind of questions are out
there, what insight would be valuable to your particular industry.”
Penn Medicine, for example, includes both a health system
and a school of medicine. For a long time, the health system has been
collecting clinical data in a data warehouse. Meanwhile, in the school of
medicine, new technology is allowing for the sequencing of human genomes, which
entails a huge amount of data.
“We know there's value in there somewhere, and we finally
have the computing power to access it,” says Farrell. Combining data analytics
with expertise in medicine opens up a brand new field of predictive healthcare,
he says.
MYTH 2: The bigger the data, the bigger the value
It takes time and resources to collect data, house it, and
catalog it, says Farrell. Indiscriminately collecting large masses of data can
divert those resources from more worthy projects.
Farrell recommends that companies have a clear idea of the
specific metric or key performance indicator that they're looking for before
they start collecting data.
“You want to get to the point where you have a handful of
nuggets of wisdom that are valuable to you,” he says. “The data by itself,
sitting there, is not enough.”
MYTH 3: Big Data is for big companies
Large companies may have more internal sources of data, but
even small firms can take advantage of data coming in from social media
platforms, government agencies, and data vendors.
“Regardless of the size of your organization, it’s better to
make decisions based on data than to simply rely on intuition or gut feelings,”
says Darin Bartik, executive director of product management for Dell Software’s
Information Management Solutions.
Smaller companies may make data-driven decisions less often
than their bigger counterparts, he says, but, when they do, they can make
course corrections faster.
“Smaller companies can use best practices to be more
data-driven and actually outpace or outmaneuver bigger, slower competitors,” he
says.
MYTH 4: Collect it now, sort it out later
Storage is getting cheaper all the time, but it's not free.
However, for many companies, the appetite for data is expanding faster than
storage costs are decreasing, says Brad Peters, CEO of San Francisco-based
Birst, a cloud-based business intelligence vendor.
Companies think that if they just collect the data, they'll
figure out what to do with it later, he says. “I see a number of large
corporations collecting boatloads of stuff, their expense on it goes up, and
they don't get any value out of it.”
In fact, with some data sets, the law of diminishing returns
starts to apply. Say, for example, you're polling people to predict an
election. You need a certain number of people to get a representative sample.
But after a point, adding more people won't significantly affect the margin of
error.
“Do you store a bunch of data you may need, that might give
you a couple more digits of precision?” he asks. “Or do you buy more people
power? Do you secure your networks better? We're not going too fast as an
economy, and budgets aren't increasing.”
And it's not just storage costs, says Dean Gonsowski, global
head of information governance and big data management at San Francisco-based
Recommind, which specializes in unstructured data analytics.
For example, it may cost the company if the data gets out,
he says. And having data sitting around in warehouses means that it's subject
to e-discovery arising from court cases.
Finally, the more data, the longer it takes to sort through
it. “When the repositories get into the billions of records, searches take
hours or weeks,” he says. “The volume of information really start clogging
systems that were never built to handle those volumes.”
MYTH 5: All data is created equal
The state of Virginia has been collecting data on student
enrollments, financial aid, and degree awards for the past 20 years. But that
doesn't mean that the data collected 20 years ago and stored in the same data
field is necessarily the same data.
“The biggest problem I deal with, is that just because it's
in the data dictionary, researchers think it's fair game,” says Tod Massa, the
policy research and data warehousing director for Virginia's State Council of
Higher Education. “For example, data on student test scores on the ACT and SAT
were initially only collected on in-state students, then there was a gap, then
it was collected on both in-state and out-of-state students.” Similarly, race
and ethnicity is tracked differently at the K-12 level and in higher education.
In fact, any particular data point might be reported
differently by different institutions, or at different points in time, or by
different people at those institutions. “If you're in an isolated shop or
enterprise that is solely responsible for the data it collects, then you might
have a different situation,” he says. “But then even, I suspect that the
meanings of data change over time.”
As a result, analysts need to have not just statistical
skills, but also local knowledge of the data and knowledge of trends in the
industry as a whole, such as SAT and ACT scores being re-calibrated.
“You can't program all those things into a data repository,”
he says.
The same applies to external data sources, he adds. “Data
collections at the federal level have changed dramatically over the past 50
years,” he says. “Understanding the culture and context of data collection is
really a necessity for using the data well.”
MYTH 6: The more specific the prediction, the better
It's human nature to think that something that is more
specific is more accurate. That `3:12 p.m.’ is more accurate than `sometime in
the afternoon.’ That the meteorologist who predicts that it will definitely
rain on Sunday morning is more accurate than the one who predicts a “fifty
percent chance of showers this weekend.”
In fact, the opposite is true. In many situations, the more
exact prediction is less likely to be accurate.
Say, for example, a customer buys a very specific laptop, in
a very particular configuration. And the only other customer to have bought
that same product in the past also bought a pair of hot pink stilettos.
“A recommendation for hot pink stilettos may be very
specific, but may be too specific – and have a high margin of error,” says
Jerry Jao, CEO of Retention Science, a marketing firm in Santa Monica, Calif.
“This is actually something we see pretty commonly among
business and marketing managers,” he says.
MYTH 7: Big Data equals Hadoop
Hadoop, a popular open-source database for unstructured
data, has been getting a lot of attention lately.
But there are other options.
“There is a whole NoSQL movement,” says Irfan Khan, general
manager and senior vice president at SAP Big Data. “There is MongoDB, Cassandra
– a whole rack of other technologies.”
Some of those technologies may be a better fit for a
particular Big Data project than others.
In particular, Hadoop works by dividing data into chunks,
and working on multiple chunks simultaneously. This approach works on many Big
Data problems, but not all of them.
“While YARN and Hadoop 2 address some of this, sometimes you
need to deal with things in ways that Hadoop isn't ideal for,” says Grant
Ingersoll, CTO at Redwood City-based LucidWorks, a Big Data consulting firm.
“People need to keep a level head and decide what is best for them, not just
what is the shiny object that all the cool kids are using.”
MYTH 8: End users don't need direct access to Big Data
With Big Data moving in at a high speed, from a wide variety
of sources, and in large volumes, it might seem that it is just too complicated
for regular employees to deal with.
But that's not necessarily the case.
Take, for example, all the data generated by the devices in
an intensive care unit. Heart rates, respiration data, EKG readings. Too often,
though, the doctors and nurses can only see a patient's current readings.
“I can't look and see what it was 10 minutes ago, or draw a
tend line for what it's going to be an hour from now,” says Anthony Jones,
chief marketing officer of Philips Healthcare’s Patient Care & Clinical
Informatics.
But being able to see the historical data for a patient can
be very valuable for a medical practitioner making a decision. “The guys
sticking with a core data science team, they're missing a big opportunity,” says
Jones.
The problem today is getting all the different devices that
generate data to talk to each other even though they weren't designed to do
that, and use different platforms, operating systems and programming languages.
And then once you do, to get the data in a useful form to doctors and nurses
right when they need it.
MYTH 9: Big Data is for big problems
The CIO of a major bank recently gave a talk about Big Data,
and was asked about end user self-service.
“And the CIO says, 'I don't believe in that,'” recalls
Peters, CEO of Birst.
That's a common attitude, he says, with some executives
thinking that Big Data only answers certain types of questions. The attitude
can be summed up this way: “The goal of Big Data for us is to solve very few,
very high-value problems with a core set of data scientists. We don't want data
chaos where normal people have access to this information, because I don't
think they need it.”
Peters disagrees with this approach, but says it's common in
many industries. “It's a rampant myth inside large insurance companies that
business users aren't smart enough to handle it.”
MYTH 10: The Big Data bubble will eventually burst
Hype cycles may come and go, but transformative
technological changes stick around. The dot-com crash did not signal the end of
the Internet.
Even when the hype dies down, companies will still have Big
Data to deal with. In fact, they will have more Big Data to deal with than they
ever expected, due to exponential growth – IDC projects that total amount of
data collected will double every two years through 2020.
And it's not just that companies are simply collecting more
of the stuff that they currently collect. Instead, new types of data are likely
to appear, requiring massive amounts of storage.
“We will get to the point where everyone who gets admitted
to a hospital, the hospital maps their genome,” says Anthony Jones, chief
marketing officer of Philips Healthcare’s Patient Care & Clinical
Informatics. “This allows treatment to be customized to the patient. And when
you talk about Big Data, that's a massive amount of data. I don't think a lot
of CIOs really appreciate how much harder things are going to get.”
By thinking of “Big Data” as just a phase, companies can
miss opportunities to capture data elements that could have an impact on their
business down the line, says Bryan Hill, CTO of Cadient Group, an interactive
marketing agency in King of Prussia, Pa.
“The term 'Big Data' is likely to change, just like cloud
computing came up, which is no different than the Web was, or the Internet,” he
says. “The term may change, but the spirit of Big Data is here to stay.”
