“ELK Stack: from text search to logs visualization” by Flowygo is a training course on the use of the ELK stack (Elasticsearch, Logstash, Kibana) for processing, indexing and visualization of data and time series.

The fundamental components of the Elastic suite will be presented, with a practical approach and exercises, deepening the applications and examining a real use case.

Elasticsearch, the main product of the suite, is a professional search engine able to effectively manage Big Data in any application / website. To date, it is the most popular search engine in the world.

Based on the open source Lucene library, Elasticsearch is a search server with support for distributed architectures that provides full-text search functionality with a programming language agnostic interface, i.e. using JSON for data representation and HTTP as communication protocol. Elasticsearch can be used to search any type of document and provides a scalable, almost real-time search system with multitenancy support.

Kibana, on the other hand, is the tool in the suite that allows you to browse and visualize the data contained in an Elasticsearch index. Leveraging the capabilities and speed of search and data aggregation offered by Elasticsearch, Kibana allows you to easily and intuitively create charts and dashboards for big-data analysis.

Course Goals

At the end of the course, participants will be able to install and configure the entire ELK stack, a workflow that will allow for in-depth Big Data analysis and the creation of charts and dashboards that can be updated in real-time.

In particular, after an introduction to search engines, the use of Elasticsearch, the search engine from Elastic, will be described. Open Source and based on state of the art algorithms, Elasticsearch is one of the solutions for the management and research on big-data that is emerging as a leader in the industry and is already used by many organizations of various sizes (from startups to large multinationals).

Next, Logstash, the second component of the suite, will be presented, allowing to populate the search engine indexes. In fact, Logstash allows to retrieve data from various sources, transform and index them within an Elasticsearch instance in an automatic way. Thanks to its plugin architecture, Logstash supports several input modes that will allow you to set up an automatic data transfer system in a few steps. Through a specific plugin, for example, it will be possible to monitor a directory where an application writes its logs, process and eventually transform each new log line and finally store the result in an Elasticsearch index.

Finally, Kibana, the component that allows navigation and analysis of large amounts of data in an intuitive way, will be illustrated. One of the main features of Kibana is the ability to create dashboards with various types of graphs for the generation of effective views of the data; Kibana connects to an instance of Elasticsearch and allows you to make queries even very complex, view in detail the most frequent values within the index, aggregate data on different dimensions and create graphs on the data, in particular time series.

During the exercises, participants will have the opportunity to repeat at any time the operations of the teacher on their notebooks.

Who It Is For

The course is aimed at professionals, IT technicians of private companies or public administrations, researchers, university students, teachers and in general to all those who intend to expand their knowledge in terms of management, navigation and analysis of large amounts of data.

Requirements

For participation is required a good command in the basic use of the computer and a shell (eg, Bash for Unix or Mac systems, or Powershell for Windows systems). No experience in programming languages is required, although a minimum knowledge of Python may be useful to carry out some exercises. Preliminary knowledge related to the use of Docker and JSON format can be useful but are not strictly necessary.

A personal computer and a stable internet connection of adequate speed are required. The software used during the course can be installed on PCs with Windows, Linux, Mac OS operating systems. There are no particular hardware requirements (RAM recommended at least 4 GB, HD at least 50 GB free).

Program

  • Introduction to the ELK stack
    • What is a search engine
    • What is Elasticsearch
    • What is Logstash
    • What is Kibana
    • Use Cases
  • Elasticsearch
    • Instance Configuration
    • Document Management
    • Mapping and Analysis
    • Querying
    • Aggregation
  • Logstash
    • Logstash Configuration
    • Workflow Creation
  • Kibana
    • Kibana configuration;
    • Data navigation;
    • Creating charts (histograms, areas, pie, lines, counters, tables);
    • Dashboard creation
    • Security (users, roles and workspaces)

Mode

The course can be conducted either in the classroom or remotely as needed.

Duration

The course has a total duration of 21 hours. The organization of the hours provided may vary based on need.

Benefits And Material

  • Extensive training materials in digital format
  • Access to repositories with the code of the exercises carried out
  • Technical support for any problems of installation and configuration of software used

Software Used

During the course you will use the software:

  • Elasticsearch
  • Kibana
  • Logstash
  • Beats
  • Flask
  • Docker

All the software used is released under the Apache 2.0 license, therefore completely free and usable without restrictions even within commercial products; it can be installed on Microsoft Windows, Linux, Mac OS platforms without license limitations.

Request for information