The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Index Lifecycle Management

Next ►
►

Index Lifecycle Management (ILM) in Elasticsearch is a feature that helps manage the lifecycle of an index, from creation to deletion. It allows you to define policies that automate the management of indices, ensuring that they are efficiently managed and resources are optimized.

Key Concepts

  1. Lifecycle Policies: Define the phases an index goes through during its lifecycle.
  2. Phases: Stages in the lifecycle of an index, such as hot, warm, cold, and delete.
  3. Actions: Operations performed during each phase, like rollover, shrink, freeze, and delete.
  4. Conditions: Criteria that trigger the transition from one phase to another.

Phases of Index Lifecycle

  1. Hot Phase:

    • The phase where the index is actively written to and queried.
    • Actions: Rollover, set priority, force merge.

  2. Warm Phase:

    • The phase where the index is no longer being written to but is still queried.
    • Actions: Allocate, shrink, force merge, set priority.

  3. Cold Phase:

    • The phase where the index is infrequently queried and stored on less expensive hardware.
    • Actions: Allocate, freeze, set priority.

  4. Delete Phase:

    • The phase where the index is deleted.
    • Actions: Delete.

Creating an ILM Policy

Step-by-Step Guide

  1. Define the Policy:
    • Create a policy that specifies the phases and actions for the index lifecycle.
PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "30d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "cold"
            }
          },
          "freeze": {},
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
  1. Attach the Policy to an Index Template:
    • Apply the policy to indices using an index template.
PUT _template/my_template
{
  "index_patterns": ["my_index-*"],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "index.lifecycle.name": "my_policy",
    "index.lifecycle.rollover_alias": "my_index_alias"
  },
  "aliases": {
    "my_index_alias": {}
  }
}
  1. Create an Index with the Template:
    • Create an index that uses the template and the ILM policy.
PUT my_index-000001
{
  "aliases": {
    "my_index_alias": {
      "is_write_index": true
    }
  }
}

Practical Exercise

Exercise: Create and Apply an ILM Policy

  1. Create a new ILM policy named logs_policy with the following phases:

    • Hot Phase: Rollover at 20GB or 7 days, set priority to 100.
    • Warm Phase: Transition after 7 days, set priority to 50.
    • Cold Phase: Transition after 30 days, set priority to 0.
    • Delete Phase: Delete after 90 days.

  2. Create an index template named logs_template that applies the logs_policy to indices matching the pattern logs-*.

  3. Create an index logs-000001 using the logs_template.

Solution

  1. Create the ILM Policy:
PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "20GB",
            "max_age": "7d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
  1. Create the Index Template:
PUT _template/logs_template
{
  "index_patterns": ["logs-*"],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "index.lifecycle.name": "logs_policy",
    "index.lifecycle.rollover_alias": "logs_alias"
  },
  "aliases": {
    "logs_alias": {}
  }
}
  1. Create the Index:
PUT logs-000001
{
  "aliases": {
    "logs_alias": {
      "is_write_index": true
    }
  }
}

Summary

In this section, we covered the basics of Index Lifecycle Management (ILM) in Elasticsearch. We learned about the different phases of an index lifecycle, how to create and apply ILM policies, and how to manage indices using these policies. By automating index management, ILM helps ensure that indices are efficiently managed, optimizing resource usage and maintaining performance.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2024. All rights reserved