AI Security — the concept of Underfitting in Machine Learning

Underfitting occurs when a model or algorithm fails to capture the complexity of the underlying data. Underfitting may happen in machine learning when the model or algorithm is too simple for the underlying data trends, like a linear model for a non-linear problem, or one that works well in a limited range of values but has poor predictive accuracy outside that range.¹

Now that we have a fair idea of what underfitting means in AI models, let’s dive a bit deeper into the concept. I’ll try to illustrate it with an example and get into the weeds a bit more as we go through this together.

First up, have your AI Security lab up and running. If you don’t have one, you can use this post to set one up and come back here once ready.

Setup and load data for testing

For this purpose, we will use the Fashion MNIST dataset.

Fashion-MNIST is a dataset of Zalando’s article images consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes.²

Create a python file (*.py) at this point and call it something like uf.py and keep adding the code provided below to it for testing.

Let’s start by loading the data and preparing it:

import tensorflow as tf
from tensorflow.keras.datasets import fashion_mnist

# Following code loads the dataset for us
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

# Folowing code is noramlysing the images from the dataset
train_images = train_images / 255.0
test_images = test_images / 255.0

# Training the model, reshaping: 
train_images = train_images.reshape(-1, 28, 28, 1)
test_images = test_images.reshape(-1, 28, 28, 1)

Now, we will define a simple neural network:

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
    tf.keras.layers.Dense(10, activation='relu'),  
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

Next, we start training the model. In order to demonstrate the concept of ‘underfitting’, we will deliberately use a very small number of ‘epochs’ — which should result in underfitting (or atleast make it easier to underfit the model).

Spoiler alert: this iteration will not result in proper underfitting :) and that is by design.

history = model.fit(train_images, train_labels, epochs=5, validation_split=0.2)

Now we are ready to evaluate our model.

test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)

Let’s put this to test!

Execute this section of the code:

import tensorflow as tf
from tensorflow.keras.datasets import fashion_mnist


(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()


train_images = train_images / 255.0
test_images = test_images / 255.0


train_images = train_images.reshape(-1, 28, 28, 1)
test_images = test_images.reshape(-1, 28, 28, 1)

Result:

Execute this section next:

This should run with no errors.

Next, train the model. We are using 5 epochs in this iteration of our code:

At this point (if everything above worked as intended), we are good to test our model and see the results.

How do we know if underfitting has occurred? 
We are expecting a low accuracy result. Lower the accuracy, 
higher the chances of underfitting.

Ok, now as you can see above, the accuracy is… not bad! Looking at the results above, it is safe to say that underfitting did not take place to a level that we were wanting to achieve.

So, let’s change a few things and try again.

But before we do that, let’s visualise the results a bit more. This will make comparing the results easier later on.

Run the following code in order to do that:

import matplotlib.pyplot as plt

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)

# Accuracy
plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.plot(epochs, acc, 'bo', label='Training accuracy')
plt.plot(epochs, val_acc, 'b', label='Validation accuracy')
plt.title('Training and validation accuracy')
plt.legend()

# Loss
plt.subplot(1, 2, 2)
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

You should get the following visual representation of the results:

Ok, now let’s get back to tweaking our code to underfit the model.

Let’s try reducing the number of neurons:

model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
    tf.keras.layers.Dense(8, activation='relu'),  # Further reduce the number of neurons
    tf.keras.layers.Dense(10, activation='softmax')
])

Execute this code and repeat everything else from our last test.

As you can see from the results below, the accuracy has actually increased!

As a next step, let’s reduce the number of epochs.

Ok, as you can see above, now the accuracy has dropped but still not enough.

At this point, there are two things we can do to further underfit the model:

1. Apply excessive regularisation and lowering the learning rate
2. Reduce the number of neurons even more!

Let’s have a look at option 1:

Here’s the code for option 1 (with 2 epochs, let’s stick with that):

from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.regularizers import l2

model = tf.keras.Sequential([
    Flatten(input_shape=(28, 28, 1)),
    Dense(8, activation='relu', kernel_regularizer=l2(0.01)),  
    Dense(10, activation='softmax')
])

Let’s take a look at the results:

Ok — better

Now let’s add the second part of option 1, decrease the learning rate when compiling:

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

Yes! We have a winner:

The accuracy has gone down considerably as you can see:

I think we can do better though… let’s try option 2.

Let’s reduce the number of neurons to 2 and run the code.

Voila! We have achieved underfitting as you can see from the result below:

And that result was with 5 epochs!

The model in its current state is not able to classify new data from the test set and is struggling to keep up. It also lacks complexity and learning abilities that make it a less effective model — caused mostly by underfitting.

I hope this post has helped you understand and visualise the concept of underfitting in an easy to understand way.

There are more ways of achieving this and I’d like to encourage you to explore those and let me know what you find in the comments!

References:
1. https://c3.ai/glossary/data-science/underfitting/
2. https://www.tensorflow.org/datasets/catalog/fashion_mnist