Luke Richardson - Electrical Engineer

1. The Motivation

I wanted to build something that would make neural networks intuitive rather than just mathematically correct. Most explanations focus on equations without showing how networks actually learn and adapt in practice.

The Core Insight

Neural networks are dynamic systems that evolve and adapt. Seeing a network learn to distinguish between a circle and spiral, or recognize handwritten digits, demonstrates how complex behavior emerges from simple computational rules.

This playground started with two classic problems that demonstrate non-linear classification. As I built it, I realized I needed a more direct connection between human input and machine learning. That's when I added handwriting recognition.

Why These Problems Matter

Circle Classification: The simplest non-linear boundary, showing why we need hidden layers
Spiral Classification: A problem requiring complex, intertwined decision boundaries
Handwriting Recognition: Direct interaction between human input and machine learning

2. Mathematical Foundation

The mathematics behind neural networks is straightforward. At its core, we're doing repeated matrix multiplications with non-linear transformations. The complexity emerges from how these simple operations combine.

The Forward Pass

Each layer transforms its input through a simple formula. For layer $l$ :

a^{(l)} = f(W^{(l)} \cdot a^{(l-1)} + b^{(l)})

This simple operation, repeated across layers, can approximate any continuous function. The activation function $f$ is crucial. Without it, we'd just have a linear transformation, regardless of how many layers we stack.

Activation Functions

I implemented three activation functions, each with different characteristics:

Sigmoid

\sigma(x) = \frac{1}{1 + e^{-x}}

Smooth, bounded, but suffers from vanishing gradients. Great for understanding, less great for deep networks.

Tanh

\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}

Zero-centered, which helps with convergence. My go-to choice for these problems.

ReLU

ReLU(x) = \max(0, x)

Simple, fast, and surprisingly effective. The workhorse of modern deep learning.

Backpropagation

Backpropagation is the chain rule applied systematically. The network learns by computing how each weight contributes to the final error and adjusting accordingly.

\frac{\partial L}{\partial W^{(l)}} = \frac{\partial L}{\partial a^{(l)}} \cdot \frac{\partial a^{(l)}}{\partial z^{(l)}} \cdot \frac{\partial z^{(l)}}{\partial W^{(l)}}

Each weight learns by computing how its change affects the final error. The error signal propagates backward through the network, telling each parameter how to improve.

3. Classic Problems: Circles and Spirals

I started with two problems that demonstrate why neural networks are necessary. Linear classifiers fail on these, but neural networks can learn the complex decision boundaries.

The Circle Problem

The circle problem is simple: classify points as inside or outside a circle. This requires a curved decision boundary, which is impossible with linear models.

Problems.ts - Circle Classification

circle: {
  generateData(): TrainingData[] {
    const data: TrainingData[] = [];
    for (let i = 0; i < 100; i++) {
      const x = Math.random() * 2 - 1;  // Random point in [-1, 1]
      const y = Math.random() * 2 - 1;
      // The magic: classify based on distance from origin
      const target = x * x + y * y < 0.5 ? [1] : [0];
      data.push({ input: [x, y], target });
    }
    return data;
  }
}

Why This Is Hard

A linear classifier can only draw straight lines. But the optimal decision boundary for this problem is a circle—a fundamentally curved shape. The network needs to learn that the relationship $x^2 + y^2 < 0.5$ defines the boundary.

The Spiral Problem

The spiral problem is more challenging. Two spirals wind around each other, creating a complex decision boundary that requires multiple hidden layers to learn.

Problems.ts - Spiral Classification

spiral: {
  generateData(): TrainingData[] {
    const data: TrainingData[] = [];
    const n = 100;
    const maxRadius = 1;
    
    for (let i = 0; i < n; i++) {
      for (let j = 0; j < 2; j++) {  // Two spirals
        const r = (i / n) * maxRadius;
        const theta = (i / n) * 4 * Math.PI + j * Math.PI;  // Offset by π
        
        const x = r * Math.cos(theta);
        const y = r * Math.sin(theta);
        
        data.push({
          input: [x, y],
          target: [j]  // Spiral 0 or Spiral 1
        });
      }
    }
    return data;
  }
}

Why This Is Difficult

The two classes are completely intertwined. There's no simple geometric shape that can separate them. The network must learn to recognize the spiral pattern itself, which requires sophisticated pattern recognition.

Training on these problems shows how the decision boundary evolves from random noise to the correct shape, sometimes oscillating before converging to the right pattern.

4. Adding Handwriting Recognition

The circle and spiral problems were useful for demonstrating concepts, but I wanted something more interactive that would connect the mathematics to direct human input. That's when I decided to add handwriting recognition.

From 2D to 784D

Moving from 2-input problems to handwriting meant jumping from 2 dimensions to 784 (28×28 pixels). This is a qualitative change that brings new challenges like the curse of dimensionality and much larger parameter spaces.

The Architecture Decision

For handwriting, I needed a network that could handle 784 inputs and output 3 classes (digits 1, 2, and 3). After experimentation, I settled on:

• Input layer: 784 neurons (28×28 pixels)
• Hidden layers: 3 layers of 16 neurons each
• Output layer: 3 neurons (one per digit)
• Activation: Tanh for hidden layers, raw outputs for classification

The Interactive Canvas

Building the drawing canvas required capturing mouse and touch events, converting them to a 28×28 pixel grid, and providing real-time feedback. I added anti-aliasing to make the drawings look more natural and improve recognition accuracy.

HandwritingCanvas.tsx - Drawing Logic

const drawPixel = (x: number, y: number, intensity: number = 1.0) => {
  const newPixels = [...pixels];
  const index = y * 28 + x;
  
  // Set the main pixel
  newPixels[index] = Math.min(1.0, newPixels[index] + intensity);
  
  // Add anti-aliasing to neighboring pixels
  const neighbors = [
    { dx: -1, dy: 0, factor: 0.3 },
    { dx: 1, dy: 0, factor: 0.3 },
    { dx: 0, dy: -1, factor: 0.3 },
    { dx: 0, dy: 1, factor: 0.3 },
    // Diagonal neighbors with less influence
    { dx: -1, dy: -1, factor: 0.1 },
    { dx: 1, dy: -1, factor: 0.1 },
    { dx: -1, dy: 1, factor: 0.1 },
    { dx: 1, dy: 1, factor: 0.1 },
  ];
  
  neighbors.forEach(({ dx, dy, factor }) => {
    const nx = x + dx;
    const ny = y + dy;
    if (nx >= 0 && nx < 28 && ny >= 0 && ny < 28) {
      const nIndex = ny * 28 + nx;
      newPixels[nIndex] = Math.min(1.0, newPixels[nIndex] + intensity * factor);
    }
  });
  
  setPixels(newPixels);
}

Real-time Prediction

The network predicts what you're drawing in real-time. Every stroke updates the prediction, creating immediate feedback between human input and the machine learning model.

The Training Data Challenge

I started with synthetic training data, but quickly realized that real handwriting is much more varied and messy. So I built a data collection system that lets users draw their own digits and contribute to the training set. The network learns from actual human handwriting, not idealized examples.

5. Real-time Visualization Challenges

Making neural network training visual and interactive presented unique challenges. I needed to balance educational value with performance, creating something that was both informative and smooth to interact with.

The Performance Challenge

Running neural network training in the browser while maintaining 60fps visualization required careful optimization. Every training step needs to update multiple canvases, recalculate decision boundaries, and maintain responsive user interaction.

The Animation Loop Strategy

I use requestAnimationFrame to create a smooth training loop. Each frame performs one training step, updates the visualizations, and schedules the next frame. This creates the illusion of continuous learning while keeping the browser responsive.

Canvas Rendering Optimization

Each visualization component uses HTML5 Canvas with careful attention to performance. The key insight was to handle device pixel ratio properly and minimize unnecessary redraws.

NetworkCanvas.tsx - Optimized Rendering

const render = () => {
  const rect = canvas.getBoundingClientRect();
  const dpr = window.devicePixelRatio || 1;
  
  // Set actual canvas size accounting for device pixel ratio
  canvas.width = rect.width * dpr;
  canvas.height = rect.height * dpr;
  ctx.scale(dpr, dpr);
  
  // Clear with background color
  ctx.fillStyle = '#0a0a0a';
  ctx.fillRect(0, 0, rect.width, rect.height);
  
  // Draw connections with weight-based opacity
  for (let i = 0; i < network.layers.length - 1; i++) {
    for (let j = 0; j < network.layers[i]; j++) {
      for (let k = 0; k < network.layers[i + 1]; k++) {
        const weight = network.weights[i][k][j];
        const opacity = Math.min(Math.abs(weight), 1);
        
        ctx.strokeStyle = weight > 0 
          ? `rgba(255, 255, 255, ${opacity * 0.5})`
          : `rgba(115, 115, 115, ${opacity * 0.5})`;
        ctx.lineWidth = Math.min(Math.abs(weight) * 3, 4);
        
        // Draw connection line
        ctx.beginPath();
        ctx.moveTo(x1, y1);
        ctx.lineTo(x2, y2);
        ctx.stroke();
      }
    }
  }
};

The Decision Boundary Challenge

For the 2D problems, I render the decision boundary by sampling the network's output across a grid of points. This creates a beautiful visualization of how the network "sees" the input space, but it's computationally expensive. I had to find the right balance between resolution and performance.

6. Implementation Insights

Building this playground taught me several important lessons about neural networks, web performance, and the intersection of education and technology.

The Value of Immediate Feedback

Immediate feedback significantly improves learning. When you can see the network's decision boundary evolving in real-time, or watch your handwriting being recognized as you draw, abstract concepts become concrete and understandable.

TypeScript for Neural Networks

Implementing neural networks in TypeScript was surprisingly pleasant. The type system caught many bugs early, especially around matrix dimensions and data flow. The performance was adequate for educational purposes, though I wouldn't recommend it for production ML workloads.

Hyperparameter Sensitivity

Making the hyperparameters adjustable revealed how sensitive neural networks can be. A learning rate that's too high causes wild oscillations. Too few neurons and the network can't learn complex patterns. Too many and it overfits quickly.

What Worked Well

• Learning rates between 0.01-0.1
• 3-4 hidden layers for handwriting
• Tanh activation for most problems
• Real training data vs synthetic
• Visual feedback during training

Common Pitfalls

• Learning rates > 0.5 (chaos)
• Too few neurons (underfitting)
• Sigmoid in deep networks (vanishing gradients)
• Ignoring data preprocessing
• Training without validation

Browser Performance Considerations

Running neural networks in the browser taught me about the delicate balance between educational value and performance. Some key insights:

Batch size matters: Processing 100 samples per frame keeps things smooth
Canvas optimization: Device pixel ratio handling prevents blurry visuals
Memory management: Proper cleanup of animation frames prevents leaks
State synchronization: React state updates need careful timing with canvas renders

7. Lessons Learned

Building this neural network playground was both a technical challenge and a journey of discovery. Here are the key insights that emerged from the process.

Interactive Learning

Interactive learning is more effective than passive reading. Seeing a decision boundary evolve, or watching your handwriting get recognized in real-time, creates intuitive understanding that equations alone cannot provide.

Understanding Through Experience

When someone sees the spiral decision boundary form, or watches their handwritten digit get correctly classified, neural networks stop being mysterious black boxes and become understandable tools.

Technical Insights

Visualization is expensive: Real-time neural network visualization requires careful performance optimization
Data quality matters more than quantity: 50 good handwriting samples beat 500 synthetic ones
Hyperparameters are sensitive: Small changes can mean the difference between learning and chaos
User experience drives learning: Smooth interactions are crucial for maintaining engagement

Future Improvements

If I were to rebuild this, I'd consider using WebGL for better performance, implement more sophisticated data augmentation, and add support for convolutional layers. The core insight that immediate visual feedback improves learning would remain central to the design.

Key Takeaway

The best educational tools let you experience concepts rather than just explaining them. Neural networks are about pattern recognition and learning from data. Making that process visible and interactive builds intuition that goes beyond memorizing formulas.

Building an Interactive Neural Network Playground

Table of Contents