How Big Is Big Data?

Big Data is a popular term used to describe the massive collection of data, whether structured, semi-structured, unstructured or raw. Data may be defined as an asset on balance sheet.

According to Gartner, Big Data comprises high volume, velocity and variety of information assets that demand cost-effective, innovative forms of information-processing for enhanced insights and decision-making. Hence, the globally accepted 3 Vs of Big Data are:

  • Volume – amount of data,
  • Velocity – speed of data in and out, and
  • Variety – range of data types and sources which include:  unstructured text documents, picture, video, email, audio, stock ticker data, financial transactions, etc.

However, recent studies have added two more components which describe Big Data, viz:

  • Variability: At times, data flow is highly inconsistent with periodic peaks which hamper the process of handling and managing data effectively.
  • Complexity: As large volumes of data come from multiple sources, data management becomes a challenging task.

In fact, the data sets are so big and complex that it becomes very difficult and challenging to process them using the traditional data processing applications. It is estimated that about 2.5 quintillion bytes of data is created every day.

This implies that about 90% of the world’s total data was created in the last two years. It should be noted here that about 80% of the total data is unstructured – there are data collected from sensors used to gather weather information, social media posts, digital photos and videos, purchase transaction records, to mobile phone’s GPS and many more.

Preparing for a career in Data Science? Take this test to know where you stand!

Both government and private sectors have used Big Data to enhance their productivity. Big Data analytics played a key role in Barack Obama’s successful re-election campaign in 2012. Witnessing the role of Big Data in addressing the problems faced by the government, the Obama Government announced the Big Data Research and Development initiative in the year 2012. The United States Federal government owns six of the ten most powerful supercomputers of the world.

In the private sector, Facebook uses Big Data to handle 50 billion photos from its user’s base. used Linux based technology to handle millions of back end operations every day. uses two data warehouses of 7.5 PB and 40 PB as well as a 40 PB Hadoop cluster for search. FICO Falcon Credit Card Fraud Detection system secures 2.1 billion active accounts across the globe.

Walmart handles 1 million+ customer transactions every hour, which are imported into databases estimated to contain more than 2.5 petabytes of data. According to estimates, the volume of data worldwide doubles every 1.2 years.

Role of Big Data in an Enterprise

The evolution of Big Data databases has enabled enterprises to know the importance of data in their growth and success. These databases have helped enterprises to save money, increase revenue and achieve many other business objectives.  The real challenge faced by the enterprises is finding that critical piece of information that provides the competitive edge. Hadoop helps in managing and handling massive amount of data. It also helps in transforming the data into a more usable structure and format, and extract valuable analytics from it.


Big Data in International Development

The scope of Big Data is not limited either to IT companies or any particular sector. As per Research on Effective uses of Information and Communication Technologies for Development (ICT4D), Big Data technologies can be extremely useful and can make important contributions in solving challenges in international development.  Advancements in Big Data technologies result in the creation of cost- effective opportunities which help in improving decision making process in critical areas of development such as healthcare, employment, law and crime, security, natural calamity etc.

Big Job Opportunities in Big Data

Big data offers huge job opportunities in the IT sector, provided one possesses the right qualifications. A study by Mckinsey & Company in 2011, report that the United States can face an acute shortage of people with deep analytical skills in big data. Companies are and will continue to look for skilled people who can tap Big Data’s promise of competitive advantage.  There are several big data jobs that require skilled professionals.
Chief Data Officer: A person responsible for the overall implementation and execution of Big Data in an organization. He or she holds an important chair in the organization. He or she should be a member of the executive board of an organization, reporting directly to the CEO.

Big Data Scientist: This is going to be one of the most sought after jobs of the 21st century.  As the Big Data industry is witnessing a magical growth, the demand for Big Data Scientist is more than ever.  But, this is not an easy task. In order to become a successful Big Data Scientist, one needs to possess some specialized skills such as natural learning processes, machine learning, conceptual modeling, statistical analysis, predictive modeling and hypothetical testing etc.  
In order to be a successful Big Data Scientist, one needs to master the following capabilities too:

  • Ability to work in a fast- paced multi-disciplinary environment
  • Strong written and verbal communication
  • Be able to develop program databases.
  • Ability to query databases and perform statistical analyses
  • Ability to create examples and demonstrations
  • Ability to work autonomously
  • Good understanding of design and architecture principals

Big Data Analyst: Big Data Analyst assists the Big Data Scientist in performing the necessary jobs.  His job primarily is to work on data in a given system and analyze different data sets. The next job for a Big Data Analyst can be that of a Big Data Scientist. And, hence he needs to possess similar sets of skills and capabilities. The skills include data mining skills (including data auditing, aggregation, validation and reconciliation), advanced modeling techniques, testing, creating and explaining data in clear and concise reports. Testing skills of a big data analyst is very important for the successful analysis of databases.  It is required for a big data analyst to be a successful communicator as he needs to communicate complex findings and ideas in a much simpler language.

Big Data Visualizer:  Big Data Visualizer is a creative job where a person is expected to visualize the data in a way that it becomes understandable for the senior management of an organization. He or she must be able to understand user interface design as well as other visualization skills such as typography, interface design, user experience design and visual art design. His/ her job is to make the abstract information from the data analyses appealing and present it in an understandable way.  Strong knowledge of Java script, HTML, familiarity with modern visualization frameworks such as Gephi , experience with common web libraries such as JQuery, LESS etc., sharp analytical abilities and proven design skills, proficiency in Photoshop, Illustrator, InDesign as well as other Adobe Creative Suite Products are some of the required skills for a Big Data Visualizer.  

Big Data Manager: Big Data Manager acts as a bridge between the technical team members and the strategic management in an organization. He or she leads and manages the teams of big data scientists, big data analysts and big data visualizers. He or she must master the core management skills like communicating effectively and efficiently, building personal relationships with the big data team, flexibility in changing environment and ability to understand, interpret and relate organization’s strategy to the team.

Big Data Solutions Architect: This domain aims to address specific big data problems and requirements.  Big Data Solutions Architect is quite important for an organization as they are trained to describe the structure and behavior of a big data solution. He or she must be familiar with HADOOP. Some of the skills required for a big Data Solutions Architect are ability to clearly articulate pros and cons of various technologies, ability to document used cases, solutions and recommendations, strong written and verbal communication skills, self- starter, ability to work in teams etc. 

​Big Data Engineer: They develop, maintain, test and evaluate big data solutions within an organization. He or she must possess extensive knowledge in different programming or scripting languages like Java, C++, PHP, Ruby, Python etc. Building data processing systems with Hadoop and Hive is another important skill, he needs to possess.

With all that said, it can be claimed that Big Data is definitely getting mainstream in the tech savvy world of today and more and more organizations are investing on it to save a lot of time and effort and still gain that success in their businesses.

Simplilearn has quite a number of courses available in Big Data and Hadoop on offer, so anybody interested in pursuing the same can click here to find out more information.

Here is a training video in Big Data and Hadoop offered by Simplilearn®:


About the Author

Eshna VermaEshna Verma

Eshna writes on PMP, PRINCE2, ITIL, ITSM, & Ethical Hacking. She has done her Masters in Journalism and Mass Communication and is a Gold Medalist in the same. A voracious reader, she has penned several articles in leading national newspapers like TOI, HT, and The Telegraph. She loves travelling and photography.

View More
  • Disclaimer
  • PMP, PMI, PMBOK, CAPM, PgMP, PfMP, ACP, PBA, RMP, SP, and OPM3 are registered marks of the Project Management Institute, Inc.