Beats are lightweight data shippers that you install on your servers to send operational data to Elasticsearch. They are designed to be simple, efficient, and easy to use. Beats can collect various types of data, such as logs, metrics, network data, and more, and ship them to Elasticsearch for analysis and visualization.

In this section, we will cover:

1. What Beats are and their types.
2. Installing and configuring Beats.
3. Sending data to Elasticsearch using Beats.
4. Practical examples and exercises.

Beats are a family of data shippers for Elasticsearch. Each Beat is a single-purpose agent that collects data and sends it to Elasticsearch. The most commonly used Beats include:

• Filebeat: Collects and ships log files.
• Metricbeat: Collects and ships system and service metrics.
• Packetbeat: Collects and ships network data.
• Winlogbeat: Collects and ships Windows event logs.
• Heartbeat: Monitors services for their availability.

1. Download and Install:

   • Download the Beat you need from the Elastic website.
   • Extract the downloaded file and navigate to the Beat directory.

2. Configuration:

   • Each Beat comes with a default configuration file (beatname.yml). Open this file in a text editor.
   • Configure the output to Elasticsearch by specifying the Elasticsearch host and port.

   Example configuration for Filebeat (filebeat.yml):

   output.elasticsearch:
     hosts: ["localhost:9200"]
   

3. Start the Beat:

   • Start the Beat using the command line.

   ./filebeat -e
   

1. Download Filebeat:

   curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.0-linux-x86_64.tar.gz
   tar xzvf filebeat-7.10.0-linux-x86_64.tar.gz
   cd filebeat-7.10.0-linux-x86_64/
   

2. Edit the Configuration File:

   filebeat.inputs:
   - type: log
     enabled: true
     paths:
       - /var/log/*.log
   
   output.elasticsearch:
     hosts: ["localhost:9200"]
   

3. Start Filebeat:

   ./filebeat -e
   

Once Beats are installed and configured, they will start collecting data and sending it to Elasticsearch. You can verify the data in Elasticsearch using Kibana or by querying Elasticsearch directly.

1. Using Kibana:

   • Open Kibana and navigate to the "Discover" tab.
   • Select the index pattern that matches the data sent by Beats (e.g., filebeat-*).

2. Using Elasticsearch Query:

   curl -X GET "localhost:9200/filebeat-*/_search?pretty"
   

1. Download and Install Metricbeat:

   curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.10.0-linux-x86_64.tar.gz
   tar xzvf metricbeat-7.10.0-linux-x86_64.tar.gz
   cd metricbeat-7.10.0-linux-x86_64/
   

2. Edit the Configuration File:

   metricbeat.modules:
   - module: system
     metricsets:
       - cpu
       - memory
       - network
     enabled: true
     period: 10s
     processes: ['.*']
   
   output.elasticsearch:
     hosts: ["localhost:9200"]
   

3. Start Metricbeat:

   ./metricbeat -e
   

4. Verify Data in Elasticsearch:

   • Use Kibana or Elasticsearch query to verify the data.

1. Download and Install Metricbeat:

   curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.10.0-linux-x86_64.tar.gz
   tar xzvf metricbeat-7.10.0-linux-x86_64.tar.gz
   cd metricbeat-7.10.0-linux-x86_64/
   

2. Edit the Configuration File:

   metricbeat.modules:
   - module: system
     metricsets:
       - cpu
       - memory
       - network
     enabled: true
     period: 10s
     processes: ['.*']
   
   output.elasticsearch:
     hosts: ["localhost:9200"]
   

3. Start Metricbeat:

   ./metricbeat -e
   

4. Verify Data in Elasticsearch:

   • Open Kibana and navigate to the "Discover" tab.
   • Select the index pattern metricbeat-*.

In this section, we covered the basics of Beats, including what they are, how to install and configure them, and how to send data to Elasticsearch. We also provided practical examples and exercises to help you get hands-on experience with Beats. In the next section, we will explore integrating Elasticsearch with other tools in the Elastic Stack.