What Is a Neural Network? | IBM What is a neural network? Machine learning Welcome Introduction Overview Machine learning types Machine learning algorithms Data science for machine learning Statistical machine learning Linear algebra for machine learning Data visualization for machine learning Uncertainty quantification Bias variance tradeoff Bayesian Statistics Singular value decomposition Feature Engineering Overview Feature selection Feature extraction Vector embedding Latent space Dimensionality reduction Principal component analysis Linear discriminant analysis Upsampling Downsampling Synthetic data Data leakage Supervised learning Overview Regression Linear regression Lasso regression Ridge regression State space model Time series Autoregressive model Classification Overview Decision trees K-nearest neighbors (KNNs) Naive bayes Random forest Support vector machine Logistic regression Ensemble learning Overview Boosting Bagging Gradient boosting Gradient boosting classifier Self-supervised learning Overview Transfer learning Unsupervised learning Overview Clustering Overview K means clustering Hierarchical clustering A priori algorithm Gaussian mixture model Anomaly detection Semi-supervised learning Overview Recommendation engine Collaborative filtering Content based filtering Reinforcement learning Overview Reinforcement learning human feedback Deep reinforcement learning Deep Learning Overview Neural networks Overview Backpropagation Encoder-decoder model Recurrent neural networks Long short-term memory (LSTM) Convolutional neural networks Transformer models Overview Attention mechanism Grouped query attention Positional encoding Autoencoder Mamba model Graph neural network Generative AI Overview Generative model Generative AI vs. predictive AI Large language models (LLMs) Overview Reasoning models Small language models Instruction tuning LLM parameters LLM temperature LLM benchmarks LLM customization LLM alignment Tutorial: Multilingual LLM agent AI image generation Diffusion models Variational autoencoder (VAE) Generative adversarial networks (GANs) Multimodal AI Overview Vision language models Tutorial: Build an AI stylist Tutorial: Multimodal AI queries using Llama Tutorial: Multimodal AI queries using Pixtral Tutorial: Automatic podcast transcription with Granite Tutorial: PPT AI image analysis answering system Retrieval augmented generation (RAG) Overview GraphRAG Tutorial: Build a multimodal RAG system with Docling and Granite Tutorial: Evaluate RAG pipline using Ragas Tutorial: RAG chunking strategies Tutorial: Graph RAG using knowledge graphs Tutorial: Inference scaling to improve multimodal RAG AI code generation Overview Vibe coding AI agents Visit the 2025 Guide to AI Agents Model training Overview LLM training Loss function Training data Model parameters Optimization algorithms Overview Gradient descent Stochastic gradient descent Model hyperparameters Hyperparameter tuning Learning rate Fine tuning Overview Parameter efficient fine tuning (PEFT) LoRA Tutorial: Fine tuning Granite model with LoRA Regularization Foundation models Overfitting Underfitting N-shot learning Few shot learning Zero shot learning Knowledge distillation Meta learning Data augmentation Continual learning Catastrophic forgetting Machine learning libraries Overview Scikit-learn XGboost PyTorch MLOps Overview AI lifecyle AI inference Model deployment Machine learning pipeline Data labeling Model governance Model risk management Model drift AutoML Model selection Federated learning Distributed machine learning AI stack Natural language processing Overview Natural language understanding Text classification Overview Sentiment analysis Tutorial: Spam text classifier with PyTorch Machine translation Text mining Overview Information retrieval Information extraction Topic modeling Latent semantic analysis Latent Dirichlet Allocation Named entity recognition Word embeddings Bag of words Intelligent search Speech recognition Stemming and lemmatization Text summarization Conversational AI Conversational analytics Natural language generation Computer vision Overview Image classification Object detection Image segmentation Instance segmentation Semantic segmentation Optical character recognition Image recognition Visual inspection By Fangfang Lee What is a neural network? A neural network is a machine learning model that stacks simple "neurons" in layers and learns pattern-recognizing weights and biases from data to map inputs to outputs. Neural networks are among the most influential algorithms in modern machine learning and artificial intelligence (AI). They underpin breakthroughs in computer vision, natural language processing (NLP), speech recognition and countless real-world applications ranging from forecasting to facial recognition. While today’s deep neural networks (DNNs) power systems as complex as transformers and convolutional neural networks (CNNs), the origins of neural networks trace back to simple models such as linear regression and how the human brain digests, processes and decides on the information presented to it. How do neural networks work? On a high level, the inspiration for neural networks comes from the biological neurons in the human brain, which communicate through electrical signals. In 1943, Warren McCulloch and Walter Pitts proposed the first mathematical model of a neuron, showing that simple units could perform computation of a function. Later, in 1958, Frank Rosenblatt introduced the perceptron, an algorithm designed to perform pattern recognition. The perceptron is the historical ancestor of today’s networks: essentially a linear model with a constrained output. In the following section, we will dive into how neural networks borrow inspiration from the human brains to make decisions and recognize patterns.   A neural network can be understood through a simple example: spam detection. An email is fed into the network, and features such as words or phrases like “prize,” “money,” “dear” or “win” are used as inputs. The early neurons in the network process the importance of each signal, while later layers combine this information into higher-level cues that capture context and tone. The final layer then computes a probability of whether the email is spam, and if that probability is high enough, the email is flagged. In essence, the network learns how to transform raw features into meaningful patterns and use them to make predictions. This process is powered by two fundamental concepts: weights and biases. Weights act like dials that control how strongly each input feature influences the decision—a word like “prize” may be given more weight than a common word like “hello.” Biases are built-in values that shift the decision threshold, allowing a neuron to activate even if the inputs themselves are weak. Together, these model parameters determine how each neuron contributes to the overall computation. By adjusting these values during training, the network gradually learns to make accurate predictions—in this case, whether an email is spam or not. Mathematically, a neural network learns a function    by mapping an input vector    to a predict a response    What distinguishes neural networks from other traditional machine learning algorithms is their layered structure and their ability to perform nonlinear transformation.   Join 100,000+ subscribers for the latest tech news Stay up to date on the most important—and intriguing—industry news on AI, automation, data, quantum, infrastructure and security with the Think Newsletter, delivered twice weekly.  Thank you! You are subscribed. A neural network is comprised of: Input layer: holds the raw features   . Hidden layers: consist of artificial neurons (or nodes) that transform inputs into new representations. Mathematically, hidden layers are expressed as the input features, multiplied by their associated weights and added bias to pass from one layer to the next layer, eventually arriving at the final output layer. This is where the linear transformation between input and output happens.  Output layer: After performing the linear transformation in the hidden layer, a nonlinear activation function (tanh, sigmoid, ReLU ) is added to produce the final prediction (such as a number for regression, or a probability distribution for classification).   A standard feedforward neural network with 3 hidden layers. Neural network training Just like other machine learning algorithms, a neural net requires rigorous training to perform well on testing. To train a network, a single neuron computes:      Where:   = input feature,   = weight,    = bias,    = weighted sum (linear transformation),    = activation function (nonlinear transformation),    = output,    represents an activation function at the output layer that transforms the linear combination to fit the decision of the function. Using this architecture, the input features X are transformed into an output Y, serving as a predictive machine learning model.   The power of a neural network comes from its ability to learn the right weights and biases from data. This is done by comparing the network’s prediction   to the true label    and measuring the error using a loss function. For example, in classification tasks, the loss might measure how far the predicted probability is from the correct answer. To minimize this loss, the network uses an algorithm called backpropagation. The neural net trains in four steps: Forward pass: Inputs flow through the network, computing linear combinations, passing through the nonlinear activation function and producing an output prediction. Error calculation: The loss function measures the difference between prediction and truth. Backward pass (backpropagation): The error is propagated backward through the network. At each neuron, the algorithm calculates how much each weight and bias contributed to the error using the chain rule of calculus. Weight update: The weights and biases are adjusted slightly in the direction that reduces the error, using an optimization method like gradient descent. This process is repeated many times over the training dataset. Each pass helps the network “tune” its internal parameters so that its predictions get incrementally closer to the correct answers. Over time, the network converges to a set of weights and biases that minimize error and generalize well to unseen data. Backpropagation, coupled with gradient descent, is the engine that makes neural networks work. It enables networks with millions (or even billions) of parameters to learn meaningful patterns from massive datasets.   However, despite practitioners’ effort to train high performing models, neural networks still face challenges similar to other machine learning models—most significantly, overfitting. When a neural network becomes overly complex with too many parameters, the model will overfit to the training data and predict poorly. Overfitting is a common problem in all kinds of neural networks, and paying close attention to bias-variance tradeoff is paramount to creating high-performing neural network models.   Modern neural network architectures—such as transformers and encoder-decoder models—follow the same core principles (learned weights and biases, stacked layers, nonlinear activations, end-to-end training by backpropagation). They differ mainly in how inputs are mixed across layers. Instead of fully connected mixing alone, transformers use attention to form data-dependent weighted combinations of representations, alongside residual connections, normalization and positional encodings to enrich wiring built on the same fundamentals. Types of neural networks While multilayer perceptrons are the foundation, neural networks have evolved into specialized architectures suited for different domains: Convolutional neural networks (CNNs or convnets): Designed fo…