output-onlinepngtools (1)
Creating dynamic fields to make the most of your Kibana visualisations

Scripted Fields in Kibana

Creating dynamic fields to make the most of your visualisations

by Matthew Mills on 31 January 2020

I must confess, I am terrible when it comes to letting food in my house expire. There has been many a day when I open the fridge only to be greeted by the smell of rotten tomatoes or off chicken. Wouldn’t it be great to track the expiry of all your food items and at a glance see what has or hasn’t expired and which items of food are approaching their expiry date?

This simple example will use Scripted Fields in Kibana to do just that.

Kibana

Kibana is the powerful visualization component of the ELK Stack (Elasticsearch – Logstah – Kibana). It’s a tool that works with Elasticsearch to display your data in realtime. With a simple interface, it is quick and easy to put together visualisations and dashboards in a way that makes sense for just about any application.

Typically one would build visualisations from existing fields in your index, but what if you want to perform calculations on multiple fields, create new fields from existing fields, extract part of your data or perform calculations based on dates in realtime? quanox mercado libre These are just some of the examples where Scripted Fields can help.

DISCOVER THE ELK STACK

Scripted Fields

Scripted Fields are a useful, but often overlooked feature in Kibana. Scripted fields compute data on the fly from the fields in your Elasticsearch indices and can be used in all the different types of Kibana visualisations. ivermectin products for alpacas This post will showcase two examples.

Pre-Requisites

Elasticsearch’s website features a full guide on how to install various components. For the sake of this tutorial, the only components needed are:

Another useful tool is the free Elastic head plugin for chrome – it’s a great way to view your indexes / clusters and a whole lot more:

Populating Data into a New Index

For this example, let’s create a Pantry index to keep track of all of the perishable food items that we purchase for our household:

http://localhost:9200/pantry/ [PUT]
{
      "mappings": {
        "properties": {
                  "Brand":    { "type": "keyword" },
                  "Item":  { "type": "keyword"  },
                  "Price":  { "type": "double"  },
                  "Purchase Date":   { "type": "date"  },
                  "Expiry Date":  { "type": "date"  }
            }
      }
}

We can see that the index is up and running using the Elastic head chrome plugin:

Let’s fill our pantry by posting a few new entries to our index. You can change the dates to whatever you like, just make sure that the Expiry Date is in the future

http://localhost:9200/pantry/_doc/ [POST]
{
          	"Brand": "CLOVER",
          	"Item": "Milk",
          	"Price": "25.99",
          	"Purchase Date": "2020-01-16T10:12:12",
          	"Expiry Date": "2020-02-05T00:00:00"
}
{
          	"Brand": "albany",
          	"Item": "Bread",
          	"Price": "13.99",
          	"Purchase Date": "2020-01-16T10:12:12",
          	"Expiry Date": "2020-02-01T00:00:00"
}
{
          	"Brand": "TASTIC",
          	"Item": "Rice",
          	"Price": "10.99",
          	"Purchase Date": "2020-01-16T10:12:12",
          	"Expiry Date": "2021-08-15T00:00:00"
}
{
          	"Brand": "PnP",
          	"Item": "Chicken Fillets",
          	"Price": "52.99",
          	"Purchase Date": "2020-01-16T10:12:12",
          	"Expiry Date": "2020-02-07T00:00:00"
}
{
          	"Brand": "PnP",
          	"Item": "tomatoes",
          	"Price": "21.99",
          	"Purchase Date": "2020-01-16T10:12:12",
          	"Expiry Date": "2020-03-01T00:00:00"
}

Hit refresh in the Elastic head plugin, there should be 5 documents:

Next we need an index pattern in order to create some visualisations in Kibana

Open up Kibana

Go to Management Index patternsCreate index pattern

Give it the name pantry:

Use Purchase Date as the Time Filter field name:

Click Create Index

Now you should see all our indexed fields. Great – we can now create a variety of useful visualisations with this data, but first let’s manipulate them a bit.

Example 1 – Concatenating Fields

Sometimes we need to work with data that is not in a consistent or complete state for our visualizations. Perhaps we don’t want to re-index our data or maybe we are not the ones indexing it in the first place. Scripted fields can be useful in this scenario.

Click on the Scripted fields tab

Click on Add scripted field:

Let’s visualize what items are costing us the most while grocery shopping. For this simple example we will combine two fields together and format them nicely to display in a pie chart:

Here’s the format we will use:

Language: The Painless scripting language is currently the only option – python and javascript are no longer supported.

Transform: Here I chose Title Case to display our data consistently regardless of how it comes in from the source. 

Script: This is just a simple concat in painless syntax, Kibana automatically applies the transform we selected:

doc['Brand'].value + ' ' + doc['Item'].value;

Let’s create a nice pie chart to display the data. Head over to Visualize via the left-nav:

Select Pie Chart

Choose the pantry index:

Add the below settings, and we can now see that PnP Chicken Fillets are dominating the grocery budget, taking up 42% of it:

Example 2 – Working with Dates in Real-time

We want a nice way to visualise what items are approaching their expiration date and we want to know exactly how long we have left to use them before they go off. So head back to scripted fields tab for the pantry index and add a Time Until Expiry field:

Script: This script just gets the expiry date and subtracts the current time to work out how long is left before expiry:

def expiryDate =  doc['Expiry Date'].value.getMillis();
def now = new Date().getTime();
return expiryDate - now;

Go back to Visualze and create a new Data Table

  • Use a min aggregation on our newly created Time Until Expiry field

  • Add a new bucket with a Terms aggregation
  • Use our first scripted field Title as the Field
  • Order by Until Expiry
  • Order Ascending

Sweet, we can now very scan through our pantry items and see how long we have left to use them before they expire. The best part is these fields update dynamically – look at the same table tomorrow and you will see that Clover Milk has 4 days left for example…

Expanding on the First 2 Examples

What happens though tomorrow when Albany Bread reaches its expiry? We surely only want to show fresh items in the table? liquid ivermectin dosage for horses Ideally we need another scripted field Status that can show Fresh or Expired.

This type of logic is very easy to implement with scripted fields and hopefully these simple example examples will help you with your own visualisations and dashboards.

Disclaimer

Scripted fields are resource-intensive as they are constantly computed in real time – they can negatively affect the performance of your Kibana instance. It is best to use them sparingly and test them in a sandbox first. For more details see: https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

About the Author