Centralized Logging

We have learnt how micro services enable the small, medium or large sized business to achieve success by providing products which are stable, scalable, reliable and quickly respond to business change. Lets understand each of the trait of micro services and how to apply those traits in practice.

Traditional Logging

Applications/services while doing their core function also log the events and needed information. These logs (by searching through it) help us analyze the application behavior, trace the flow and diagnose the defects.

Traditionally each application logs its data on the server, so for debugging either one needs to access the machine to get the logs or those are transferred to a location. This involves help from operations team as development teams wont have access to the production servers. In monolith since all different components are in one box, they all log at the same place. When it comes to debugging it, searching for a particular error across hundreds of log files from multiple different server instances is difficult and time consuming. Since application servers would have limited ability, logs if not archived locally would get overridden. Debugging in such situations is like finding a needle in the haystack.

Need Of Centralized Logging

As the transaction is distributed across multiple micro services, having an ability to be able to log it at a centralized location is essential.

There always was a need to manage logs centrally, with good searching, filtering and visualization ability. Other important aspect is faster searching ability to find precise information especially when all applications are writing centrally.

Centralized logging while it is needed, also demands a standardization of logging. Meaning all application in the echo system should embrace similar way for logging, should agree on what kind of elements would be logged. As its a distributed transaction, what co relation elements would be passed among different services to weave a trace. There needs a consistency in all these aspects. When there is a common log structure defined and key common elements present in all application, it makes the logs query-able and that way provide quick and relevant results. Essentially a guideline to the development teams needs to be created on what kind of information to be logged, when to log (key decision factors e.g. time outs, exception, errors, key business flows), how to log i.e. make use of the logging solution within the micro service. A guideline template is suggested here.

Guideline Template For Logging

  • Category: Information (Info), Error/exception (error), Data to help in debugging (Debug)
  • Application/service name
  • Feature name (specific API add a product)
  • Method name (specific method/function name)
  • Date and Time
  • External reference ID to allow external application to pass their identifier for end to end tracing.
  • Co-relation ID to trace distributed transactions across multiple services.
  • Context ID to trace the behavior of an individual application.
  • Host name where service is running
  • Service name and its instance
  • Message with meaningful and enough information
  • Let us understand the centralized logging solution. There are many options available but let’s look at one of them to understand how does it function.

Applications push the log information to a logging infrastructure. Logging infrastructure has typically three components.

How Does Logging Work?

A log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. This component collects and parses the logs.

A log store components which accepts the log documents, indexes those documents and stores it. These indexes help us. You can imagine finding a number in a gigantic book versus finding it in an organized directory. This component categorizes and indexes the log info documents as per different query-able elements in the documents. It stores the underlying information in the key, value format to enable lightening search.

Third component is visualizer. This interface gives freedom to filter, query and get shaped data. E.g. You can query by a given duration or an identifier and further add criteria to nail down to the specific information. This interface provides easy to use searching mechanism and present data in an easy to interpret manner.

In order to capitalize on centralized logging, you need appropriate instrumentation in the code and that too at an appropriate places and cases.

These could be beginning & conclusion of the service call, request & response, key logic moments, decision flows and values etc.

There are different architecture patterns and there could be more or less components involved.

Let us see the comparison between ELK and Splunk.

Comparison FactorELK (Elastic Search, Logstash, Kibana)Splunk
What is it?ELK is an open source stack that lets you reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time.Splunk software is used for searching, monitoring, and analyzing machine-generated big data, via a Web-style interface.
Key ComponentsLogstash is a data pipe to feed into Elastic Search, Elastic Search acts as a search engine and Kibana is a visualizer. Forwarder pushes data to remote indexer. Indexers store and index the data. Search head works as a front end.
Learning CurveQuick learningModerate curve
Community SupportStrongModerate
VisualizationUI with needed abilities and no user management.Flexible controls and User management.
Cost FreeLiecenced

Usually when you are growing into a space, ELK is a better choice. In cases where the organization is grand and needs defined and managed tooling across, Splunk shines better provided you are ready to pay more.