Simplest Neural Net

Be patient when running the model. For the first 30 seconds it is training the perceptron and there is no output shown.

This is the simplest neural net. It has a single neuron. Shiffman calls the neuron class, Perceptron. Our neuron will have three inputs. I suppose it could be even simpler if it had only two or even just one input. We will train the neural net to recognize whether a point with values x and y is above or below a line.

We will use a class called Trainer to train the neuron. The neuron has to be trained with right answers. Often humans are the source of the right answers. You are training neural nets whenever you do a Captcha.

Have you noticed that the Captcha’s have been getting trickier over the years? That’s because neural nets are getting smarter. To prove you are human and to be useful for training smarter neural nets, you have to answer trickier Captchas. We are very soon going to be at the point where neural nets are better at answering Captchas than humans.

Rather than manually training our neuron, we we will write a class that knows the the right answers and let it train the neural net.

The way Shiffman did his Trainer class, each Trainer is like a single data point (perhaps analogous to a single Captcha answer). A trainer knows a set of inputs and the right answer for those inputs.

See also an implementation with a few embellishments: Simplest Neural Net with Trainer Error

Implementation

class Perceptron {

  float[] weights;
  float learningRate = 0.01;

  // Constructor. Takes one variable which is the number of
  // inputs. Initializes the weights of the neuron randomly.
  // Later the weights will be trained.
  Perceptron(int n) {
    weights = new float[n];
    for (int i = 0; i < n; ++i) {
      weights[i] = random(-1, 1);
    }
  }

  // Our neuron will weight its inputs and compute a sum.
  // From the sum it will produce an answer.

  int feedforward(float[] inputs) {
    float sum = 0.0;
    for (int i = 0; i < weights.length; ++i) {
      sum += inputs[i] * weights[i];
    }
    int output = activate(sum);
    return output;
  }

  int activate(float sum) {
    return sum > 0 ? 1 : -1;
  }

  void learn(float[] inputs, int correctAnswer) {
    int whatPerceptronCurrentlyGets = feedforward(inputs);
    int error = whatPerceptronCurrentlyGets - correctAnswer;
    if (error == 0) return;
    if (error == 2) {
      // Neuron guessed too high.
      // Turn down any positive inputs.
      // Turn up any negative inputs.
      for (int i = 0; i < weights.length; ++i) {
        weights[i] -= 2 * learningRate * inputs[i];
      }
    } else {
      // If we got here, error must be -2
      // Neuron guessed too low.
      // Turn up any negative weights!
      // Turn down any positive weights.
      for (int i = 0; i < weights.length; ++i) {
        weights[i] += 2 * learningRate * inputs[i];
      }
    }
  }
}

class Trainer {

  float[] inputs;
  int answer;

  Trainer(float x, float y, int a) {
    inputs = new float[3];
    inputs[0] = x;
    inputs[1] = y;
    inputs[2] = 1;
    answer = a;
  }
}

int frameCount = 0;

Trainer[] trainers;
static final int NUMBER_OF_TRAINERS = 1800;

Perceptron perceptron;

void setup() {
  size(600, 400);
  frameRate(30);
  // Make "neural net." It is just a single perceptron.
  perceptron = new Perceptron(3);
  // Make trainers
  trainers = new Trainer[NUMBER_OF_TRAINERS];
  for (int i = 0; i < trainers.length; ++i) {
    // Make a random x:
    float x = random(-300, 300);
    // Make a random y:
    float y = random(-200, 200);
    int correctAnswer = y > 0 ? -1 : 1;
    trainers[i] = new Trainer(x, y, correctAnswer);
  }
}

void draw() {
  if (frameCount >= NUMBER_OF_TRAINERS) {
    // If we are done training, we start classifying
    float x = random(0, 600);
    float y = random(0, 400);
    float inputs[] = new float[3];
    inputs[0] = x - 300;
    inputs[1] = y - 200;
    inputs[2] = 100;
    int perceptronsAnswer = perceptron.feedforward(inputs);
    if (perceptronsAnswer == 1) {
      fill(0, 0, 255);
    } else {
      fill(0, 255, 0);
    }
    rect(x - 2, y - 2, 4, 4);
  } else {
    // We are training
    Trainer trainer = trainers[frameCount];
    perceptron.learn(trainer.inputs, trainer.answer);
  }
  frameCount++;
}