There is a vast amount of data all around us, coming in from all kinds of sources. This data is divided into various types, all of which will be covered in this tutorial.
Let’s look at the types of Hadoop file formats in the next section.
Types of Hadoop File Formats
Hive and Impala table in HDFS can be created using four different Hadoop file formats:
- Text files
- Sequence File
- Avro data files
- Parquet file format
Let’s learn about each Hadoop file formats in detail.
1. Text files
A text file is the most basic and a human-readable file. It can be read or written in any programming language and is mostly delimited by comma or tab.
The text file format consumes more space when a numeric value needs to be stored as a string. It is also difficult to represent binary data such as an image.
2. Sequence File
The sequencefile format can be used to store an image in the binary format. They store key-value pairs in a binary container format and are more efficient than a text file. However, sequence files are not human- readable.
3. Avro Data Files
The Avro file format has efficient storage due to optimized binary encoding. It is widely supported both inside and outside the Hadoop ecosystem.
The Avro file format is ideal for long-term storage of important data. It can read from and write in many languages like Java, Scala and so on.Schema metadata can be embedded in the file to ensure that it will always be readable. Schema evolution can accommodate changes. The Avro file format is considered the best choice for general-purpose storage in Hadoop.
4. Parquet File Format
Parquet is a columnar format developed by Cloudera and Twitter. It is supported in Spark, MapReduce, Hive, Pig, Impala, Crunch, and so on. Like Avro, schema metadata is embedded in the file.
Parquet file format uses advanced optimizations described in Google’s Dremel paper. These optimizations reduce the storage space and increase performance. This Parquet file format is considered the most efficient for adding multiple records at a time. Some optimizations rely on identifying repeated patterns. We will look into what data serialization is in the next section.
What is Data Serialization?
Data serialization is a way to represent data in the storage memory as a series of bytes. It allows you to save data to disk or send it across the network.
For example, how do you serialize the number 123456789? It can be serialized as 4 bytes when stored as a Java int and 9 bytes when stored as a Java String.
Avro is an efficient data serialization framework, widely supported throughout Hadoop and its ecosystem. It also supports Remote Procedure Calls or RPC and offers compatibility with programming environment without compromising performance. Let’s take a look at the data types supported by Apache Avro and Avro Schemas.
Supported Data Types in Avro
Given below are the primitive data types supported by Avro:
- Null: Null is an absence of a value.
- Boolean: Boolean refers to a binary value.
- Int:int refers to a 32-bit signed integer
- Long: long is a 64-bit signed integer.
- Float: Float is a single-precision floating point value.
- Double:Double is a double-precision floating point value.
- Bytes: Byte is a sequence of 8-bit unsigned bytes.
- String: is a sequence of Unicode characters.
Let’s look at the complex data types supported in Avro schemas.
- Record: Record is a user-defined type composed of one or more named fields.
- Enum:Enum is a specified set of values.
- Array: Array is zero or more values of the same type.
- Map: Map is a set of key-value pairs. A key is a string, while the value is of a specified type.
- Union: Union is exactly one value matching a specified set of types.
- Fixed: Fixed refers to a fixed number of 8-bit unsigned bytes.
Now let's see how Avro schema helps Hive create a table.
Hive Table and Avro Schema
Here is an example that shows the equivalent of a Hive table in an Avro schema.
(id INT, name STRING, title STRING)
Avro also supports setting a default value in a schema as shown in below example.
{"namespace":"com.simplilearn",
"type":"record",
"name":"orders",
"fields":[
{"name":"id", "type":"int"},
{"name":"name", "type":"string"},
{"name":"title", "type":"string"}]
}
Other Avro Operations
{"namespace":"com.simplilearn",
"type":"record",
"name":"orders",
"fields":[
{"name":"id", "type":"int"},
{"name":"name", "type":"string", "default":"simplilearn"},
{"name":"title", "type":"string","default":"bigdata"}]
}
Now, let us discuss how Avro and Parquet file format use Sqoop.
Avro With Scoop
Avro and Parquet file format use Sqoop which is a tool designed to transfer data between Hadoop and "Relational Database Management System" or "RDBMS"
In Sqoop, you can import data to HDFS in the Avro format and export the Avro format to RDBMS. To perform the operation, add the parameter--as-avrodatafile in the Sqoop command.
Hive supports all Avro types. However, Impala does not support complex or nested types with Avro, such as enum, array, fixed, map, union, and record (nested).
In the next section, we will discuss how to import data in Parquet file format using Sqoop.
Parquet With Sqoop
You can import data to HDFS in the Parquet file format and export the Parquet file format to RDBMS using Sqoop. To perform the operation, add the parameter: -as-parquetfile in the Sqoop command. In the next section, we will discuss how to import Mysql to hdfs in Parquet File Format.
Importing MySQL to HDFS in Parquet File Format
We can import Mysql to hdfs in Parquet File Format either by Creating a New Table in Hive and by Reading Parquet Files. Let us look into them in detail.
Creating a New Table in Hive With Parquet File Format
You can create a new table in or Impala stored in the Parquet file format. In Impala, you can use LIKE PARQUET to use column metadata from an existing Parquet data file as shown in the image given below.
In this example, a new table and a new order are being created to access existing Parquet file format data. Parquet files are in binary format and cannot be read easily.
Reading Parquet Files
To read parquet files, you can use parquet-tools as given below.
Summary
Here’s a quick summary of the Hadoop file formats tutorial!
- Hive and Impala tables in HDFS can be created using text files.
- Sequence files, Avro data files, and Parquet file formats.
- Data serialization is a way of representing data in memory as a series of bytes.
- Avro is an efficient data serialization framework and is widely supported throughout Hadoop and its ecosystem.
- Using Sqoop, data can be imported to HDFS in Avro and Parquet file formats.
- Using Sqoop, Avro, and Parquet file format can be exported to RDBMS.
Conclusion
Understanding the types of Hadoop file formats is undoubtedly important. However, Hadoop file formats are one of the many nuances of Big Data and Hadoop. And if you wish to master Big Data and Hadoop, Simplilearn’s certification course is just what you need. On the other hand if you are proficient in this field and wish to scale up your career and become a Big Data Engineer, our Caltech PGP Data Science Program next stop!