Mute uninteresting log noise with Machine Learning

Classifiers

A classifier is a machine learning model that can predict which category (or class) a piece of input data falls into. For example, you can build a classifier to identify if an image is of a dog or a cat.

You can build a classifier from any of kind of data, and Classificationboxsupports numbers, text blobs, keywords, lists and images — or any combination.

Classifying log items

If we can build a classifier that knows the difference between noise and not-noise in lines of log text, then we could apply this knowledge automatically to filter out or turn down the noise. If it gets really good, it could even form the basis of a notification system, but let’s not get ahead of ourselves.

Later, I’ll show you a simple Go program that just pipes lines of log streams through it, dimming the noise and highlighting the useful stuff.

To start with, we need some training data. Getting noise is easy enough, but for classifiers to work best they need a balanced number of examples — that is, a similar number of examples for each class in the training data.

Assuming we have the data, we then teach the classifier each example with some code like this: (this is Go code, btw)

cb.Teach(ctx, "model1", classificationbox.Example{
    Class: class,
    Inputs: []classificationbox.Feature{
        classificationbox.FeatureText("line", line),
    },
})

The line and class variables would contain the example log line, and whether we consider it to be noise or not, respectively.

Best practice is to use a random 80% of your training data to teach the model, and the remaining 20% to validate it. To see this in action, read Build a machine learning image classifier from photos on your hard drive very quickly.

Making predictions

Once the training is complete and our model is ready, we can then start to ask it to predict which class each line of log text is most likely to fall into.

With a simple HTTP request, this is done like this:

POST /classificationbox/models/1/predict
{
    "limit": 1
    "inputs": [{
        "key": "line",
        "type": "text",
        "value": "Log line goes here"
    }]
}

You can of course also use the Classificationbox Go SDK here too if you like.

Here we are asking the model to pick one class (via the limit field) based on the log text line. The key and type should match the examples, but obviously the value can be different.

The response will look something like this:

{
    "success": true,
    "classes": [{
            "id": "noise",
            "score": 0.97
    }]
}

We can see that the model predicts that the log entry is noise. For non-noise, it will return the other class.

Because Classificationbox can make predictions very quickly, you can incorporate this capability into a variety of different places in your stack.

Once interesting approach is to build a little bash tool that you can pipe log streams through and ask it to hide or dim the noise, highlighting the bits we care about.

A tool to pipe log items through

When exploring this idea, I wrote this Go program that you can use in the terminal to pipe through log files, or even the output of running programs, and have the output colour change depending on whether that item is considered interesting or not.

You could run it in a variety of ways using unix pipes to stream the data:

cat logfile.txt | logclass

or something like:

sometool | logclass | notifytool

Here’s the code

https://blog.machinebox.io/mute-uninteresting-log-noise-with-machine-learning-daa19f6e222