From Spatial Patterns to Sequential Patterns

Images are naturally organized in two-dimensional space, which makes CNNs highly effective at capturing spatial patterns. However, many real-world problems involve sequences, where the order of information is important. A sentence, for example, consists of words arranged in a particular order; a speech signal unfolds over time; and stock prices form a time-dependent sequence. In such problems, understanding the current input often requires information from earlier inputs.

Recurrent Neural Networks (RNNs) are designed to handle this kind of sequential data. Unlike a conventional feedforward network, which processes each input independently, an RNN maintains a form of memory that carries information from previous time steps into the current computation. This allows the network to use both the current input and relevant information from the past when producing its output.

Understanding Sequential Data

Consider the sentence: “The movie was excellent because the acting was outstanding.”

To understand the meaning of the word “outstanding,” the model benefits from information about the words that appeared earlier in the sentence. Processing each word independently would lose much of this context.

An RNN processes the sequence one element at a time while maintaining a hidden state. At each time step, the hidden state is updated using the current input and the previous hidden state. In this way, information can flow through the sequence.

A simplified representation is:

Input → RNN → Hidden State → RNN → Hidden State → RNN → Output

The hidden state acts as a running summary of information encountered so far.

The Basic RNN Architecture

At each time step t, an RNN receives two main inputs:

• Current input $x_t$— the element currently being processed.
• Previous hidden state $h_{t-1}$ — information carried from the preceding time step.

The network combines these to produce a new hidden state $h_t$, which can then be used to generate an output.

Here, $W_x$ controls how the current input influences the hidden state, while $W_h$ determines how information from the previous hidden state is carried forward. The bias b provides an additional adjustable parameter, and the tanh activation introduces non-linearity.

The important idea is that the same weights are reused at every time step. This allows the network to process sequences of different lengths while maintaining a consistent mechanism for handling each element.

How an RNN Learns

During training, the RNN produces outputs from the sequence and compares those predictions with the expected targets. The resulting loss is then propagated backward through the unrolled network.

This training method is called Backpropagation Through Time (BPTT). It is essentially backpropagation applied across the different time steps of the sequence. The gradients from multiple time steps contribute to updates of the shared RNN parameters.

Because the same weights are used repeatedly, an update can be influenced by information from many positions in the sequence. This allows the model to learn relationships between events that occur at different points in time.

However, this repeated propagation of gradients also creates an important training challenge. As sequences become longer, gradients can become extremely small or extremely large as they are propagated through many time steps. These problems are known as the vanishing-gradient and exploding-gradient problems, and they can make it difficult for a basic RNN to learn long-term dependencies.

Unrolling an RNN Through Time

An RNN becomes easier to understand when its recurrent structure is unrolled across time steps. Although the network contains a single recurrent cell whose parameters are shared, unrolling displays how that same cell processes each element of a sequence one after another.

Consider a sequence containing four inputs:

$x_1, x_2, x_3, x_4$

The RNN processes them sequentially. The first input produces a hidden state $h_1$, which is passed to the next time step. The second input is then combined with $h_1$ to produce $h_2$, and this process continues throughout the sequence.

The information flow can be represented as:

$x_1 \rightarrow h_1 \rightarrow h_2 \rightarrow h_3 \rightarrow h_4$

while each hidden state also receives the corresponding input:

$x_1 \rightarrow h_1,\quad x_2 \rightarrow h_2,\quad x_3 \rightarrow h_3,\quad x_4 \rightarrow h_4$

The crucial point is that the same RNN cell and the same weights are reused at every time step. The network does not create a completely new set of parameters for each position in the sequence. This parameter sharing makes RNNs efficient and allows them to handle sequences of varying lengths.

For example, when processing a sentence, the first hidden state may capture information about the beginning of the sentence. As additional words are processed, subsequent hidden states incorporate information from both the current word and the accumulated context.

This recurrent flow gives the RNN a form of short-term memory. Information from earlier time steps can influence later predictions, allowing the network to model relationships that would be difficult to capture if every input were processed independently.

The output structure can also vary depending on the task. An RNN may produce an output at every time step, such as when labeling each word in a sentence, or produce a single output after processing the entire sequence, such as when classifying the sentiment of a complete sentence.

Many-to-Many and Many-to-One RNNs

The recurrent structure of an RNN can be adapted to different sequence-based tasks depending on how inputs and outputs are organized. Some problems require an output at every time step, while others require the entire sequence to be summarized into a single prediction.

Many-to-One

In a many-to-one configuration, the network receives a sequence of inputs but produces a single output. The hidden state progressively collects information from the sequence, and the final representation is used to make the prediction.

For example, in sentiment analysis, the RNN can process each word in a sentence and use the information accumulated in the final hidden state to classify the entire sentence as positive, negative, or neutral.

Sequence → Single Output

Words → RNN → Final Hidden State → Sentiment

Many-to-Many

In a many-to-many configuration, the network produces an output for each time step. This is useful when every element of the sequence needs to be analyzed or labeled individually.

For example, in part-of-speech tagging, each word in a sentence can be assigned a grammatical category. The RNN processes the sequence while producing a corresponding output for each word.

Sequence → Sequence

Input₁ → RNN → Output₁
Input₂ → RNN → Output₂
Input₃ → RNN → Output₃

Another variation uses a complete input sequence to generate a complete output sequence, as in sequence-to-sequence tasks such as machine translation. In such systems, an encoder can process the input sequence and a decoder can generate the output sequence.

The choice of configuration depends on the nature of the problem:

Configuration	Input	Output	Example
Many-to-One	Sequence	Single value	Sentiment analysis
Many-to-Many	Sequence	Sequence	Part-of-speech tagging
Sequence-to-Sequence	Sequence	Different sequence	Machine translation

These configurations demonstrate the flexibility of recurrent networks. By changing how inputs and outputs are connected to the recurrent structure, the same fundamental mechanism can be applied to a wide variety of sequential problems.

The Hidden State: How an RNN Remembers Information

The defining feature of an RNN is its hidden state. It acts as a running representation of information encountered earlier in a sequence. At each time step, the network combines the current input with the previous hidden state to produce a new hidden state.

Here, $x_t$ is the current input, $h_{t-1}$ contains information carried from the previous time step, and $h_t$ becomes the updated representation passed forward. The parameters $W_x, W_h$, and b are learned during training.

Consider processing the sentence:

“The cat sat on the mat because it was tired.”

As the RNN encounters each word, its hidden state is continuously updated. By the time it reaches “tired,” the hidden state contains information influenced by earlier words in the sentence. This allows the network to use context rather than treating each word as an isolated input.

However, the hidden state is not a perfect memory. In a basic RNN, information from distant time steps must pass through many repeated transformations before it can influence the current state. As sequences become longer, important information from the beginning can gradually become weaker.

This limitation is particularly important when the relationship between two elements of a sequence spans many time steps. A basic RNN may remember recent information reasonably well but struggle to preserve information from much earlier in the sequence.

This limitation is closely connected to the vanishing-gradient problem, which can make learning long-term dependencies difficult. Specialized recurrent architectures such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks were developed to address this limitation by providing more effective mechanisms for controlling how information is retained and discarded.

Vanishing and Exploding Gradients in RNNs

When an RNN processes a long sequence, information must be carried through many recurrent steps. During training, the gradients used to update the network’s weights also have to travel backward through these time steps. Repeated multiplication during this process can cause the gradients to become extremely small or extremely large.

When gradients become very small, the problem is called the vanishing-gradient problem. When they become excessively large, it is known as the exploding-gradient problem.

Vanishing Gradients

With vanishing gradients, the gradient can shrink significantly as it moves backward through the sequence. As a result, earlier time steps receive very small updates. The network therefore struggles to learn relationships between events that are far apart in the sequence.

For example, consider:

“The students who studied very hard throughout the semester finally passed the examination.”

To correctly understand the relationship between “students” and “passed,” information may need to be retained across many intermediate words. A basic RNN can struggle to preserve such long-range information when gradients vanish.

Exploding Gradients

The opposite problem occurs when gradients grow excessively during backpropagation. Large gradients can cause extremely large weight updates, making training unstable. The loss may fluctuate dramatically or even become numerically undefined.

A common practical technique for controlling exploding gradients is gradient clipping. Instead of allowing a gradient to grow without limit, its magnitude is restricted to a predefined threshold before the weights are updated.

These problems reveal an important limitation of the basic RNN architecture: although its hidden state provides a mechanism for carrying information through a sequence, it does not provide reliable long-term memory. This limitation motivated the development of more advanced recurrent architectures that can control which information should be retained, updated, or discarded.

Long Short-Term Memory (LSTM) Networks

The limitations of basic RNNs in retaining information over long sequences led to the development of Long Short-Term Memory (LSTM) networks. LSTMs are a specialized type of recurrent neural network designed to preserve important information for longer periods while reducing the difficulty of learning long-term dependencies.

Instead of relying on a single hidden state to carry all information, an LSTM maintains a cell state that acts as a dedicated information pathway. The network uses specialized structures called gates to control what information should be retained, what new information should be added, and what information should be removed.

An LSTM contains three main gates:

• Forget gate — determines which information from the previous cell state should be discarded.
• Input gate — determines which new information should be stored.
• Output gate — determines which information from the cell state should be exposed as the current hidden state.

The cell state provides a pathway through which important information can flow across many time steps. The gates continuously regulate this pathway, allowing the network to preserve useful information while filtering out information that is no longer relevant.

For example, when processing a long sentence, an LSTM can learn to retain information about the subject introduced near the beginning of the sentence and use it much later when making a prediction. This ability makes LSTMs considerably better suited to many long-sequence problems than a basic RNN.

The strength of an LSTM therefore comes from controlled memory. Rather than simply carrying the entire previous state forward, it learns what to remember, what to forget, and what information to use for the current output.

Long Short-Term Memory Gates

The power of an LSTM comes from the way its gates regulate the flow of information. At each time step, the network examines the current input and the previous hidden state, then decides how the internal memory should change.

An LSTM first computes a candidate cell-state update, usually written:

$\tilde{c}_t=\tanh(W_cx_t+U_ch_{t-1}+b_c)$

Then the input gate controls how much of that candidate enters the cell state.

Forget Gate

The forget gate determines which information from the previous cell state should be retained and which should be discarded. It produces values between 0 and 1, where a value close to 1 indicates that information should be retained and a value close to 0 indicates that it should be forgotten.

Input Gate

The input gate controls how much new information should be added to the cell state. The LSTM first identifies potentially useful new information and then determines which parts should actually be stored.

Cell State Update

The cell state is updated by combining the retained previous information with the selected new information. This creates a controlled memory pathway that can carry important information across many time steps.

Output Gate

The output gate determines which information from the updated cell state should be used to produce the current hidden state. The hidden state is then passed to the next time step and can also be used to generate an output.

The overall information flow can be summarized as:

Previous Cell State → Forget → Retain Important Information → Add New Information → Updated Cell State → Output Gate → Hidden State

The gates are learned during training, so the LSTM determines for itself which information is useful for the task. This makes the architecture more flexible than manually defining rules for what a sequence should remember or forget.

For example, while processing a sentence, the forget gate may reduce the influence of information that is no longer relevant, while the input gate can store a newly introduced subject or important context. Later, the output gate can expose the relevant information when it becomes useful for making a prediction.

This controlled flow of information is what allows LSTMs to handle long-term dependencies more effectively than basic RNNs.

Gated Recurrent Unit (GRU)

Another important recurrent architecture is the Gated Recurrent Unit (GRU). Like an LSTM, a GRU is designed to address the difficulty that basic RNNs have with learning long-term dependencies. However, it uses a simpler internal structure and fewer gates.

A GRU combines the ideas of maintaining useful information and controlling its flow through two main mechanisms: the update gate and the reset gate.

Update Gate

The update gate determines how much of the previous hidden state should be retained and how much new information should be incorporated. The update gate controls how much information from the previous hidden state is retained and how much new information is incorporated. In the common GRU formulation, a larger update-gate value gives more weight to the candidate hidden state, while a smaller value gives more weight to the previous hidden state.

Reset Gate

The reset gate determines how strongly the previous hidden state should influence the creation of new information. When the reset gate reduces the influence of the previous state, the GRU can effectively focus more on the current input.

A simplified GRU information flow can be represented as:

Current Input + Previous Hidden State → Reset Gate & Update Gate → Candidate Hidden State → New Hidden State

Unlike an LSTM, a GRU does not maintain a separate cell state. Its hidden state serves as both the memory representation and the information passed to the next time step.

This simpler structure can make GRUs computationally lighter than LSTMs while still providing an effective mechanism for handling longer-term dependencies. In practice, the choice between an LSTM and a GRU depends on factors such as the dataset, sequence length, computational resources, and the specific task.

Both architectures demonstrate an important evolution beyond the basic RNN: rather than allowing information to flow through the sequence without explicit control, they learn how much information to retain, update, and discard.

RNNs vs. LSTMs vs. GRUs

Basic RNNs, LSTMs, and GRUs all process sequential information by carrying information from one time step to another, but they differ in how they manage that information.

A basic RNN uses a hidden state to carry information forward. Its structure is relatively simple and computationally efficient, but it can struggle to preserve information over long sequences because of vanishing and exploding gradients.

An LSTM introduces a separate cell state and three gates—forget, input, and output—to provide more controlled memory management. This makes it particularly useful when important information needs to be retained across many time steps.

A GRU simplifies this approach by using two main gates—update and reset—and does not maintain a separate cell state. It often provides a good balance between computational efficiency and the ability to learn long-term dependencies.

Architecture	Memory Mechanism	Gates	Complexity	Long-Term Dependencies
RNN	Hidden state	None	Low	Limited
LSTM	Hidden state + cell state	3 main gates	Higher	Better suited
GRU	Hidden state	2 main gates	Moderate	Better suited

The choice is therefore not simply about which architecture is newer or more complex. A basic RNN may be sufficient for short sequences and simpler tasks, while LSTMs and GRUs are better suited to situations where information must be maintained over longer periods.

These recurrent architectures form an important foundation for understanding sequential deep learning. However, recurrent processing has an inherent limitation: information is processed step by step, making it difficult to efficiently handle very long sequences and highly parallel computation. This limitation eventually motivated the development of attention-based architectures and Transformers, which changed how modern neural networks process sequential information.

Training RNNs with Backpropagation Through Time

Training an RNN follows the same fundamental learning principle used by other neural networks, but the recurrent structure introduces an additional dimension: time. Because the same weights are reused at every time step, the network must learn how those shared parameters influenced the predictions throughout the entire sequence.

During the forward pass, the RNN processes the sequence one element at a time. Each hidden state incorporates information from the current input and the previous hidden state. The network may produce an output at every time step or produce a final output after processing the sequence, depending on the task.

The predictions are compared with the target values to calculate the loss. Backpropagation then determines how the network’s parameters contributed to that loss. Because the network has been unrolled across time, the gradients are propagated backward through the sequence. This process is called Backpropagation Through Time (BPTT).

The gradients generated at different time steps contribute to updates of the same shared parameters.

Conceptually, the training cycle is:

Sequence → Forward Pass → Predictions → Loss → Backward Through Time → Gradients → Weight Update → Improved Predictions

For short sequences, this process can work effectively. However, as the sequence becomes longer, the repeated gradient calculations can lead to vanishing or exploding gradients. This is one of the main reasons basic RNNs struggle with long-term dependencies and why architectures such as LSTMs and GRUs became important alternatives.

In practice, RNN training is also performed using mini-batches of sequences, allowing multiple sequences to be processed together while maintaining their temporal order within each sequence. The optimizer then updates the shared network parameters based on the gradients accumulated from the batch.

RNN Applications in Real-World Problems

The ability of RNN-based architectures to process information in sequence makes them useful for problems where order, timing, and context influence the meaning of the data. Although newer architectures have replaced RNNs in many modern applications, RNNs and their variants remain important for understanding sequential modeling.

Natural Language Processing

RNNs can process text one word or token at a time while maintaining information about earlier words. This makes them useful for tasks such as sentiment analysis, text classification, language modeling, and sequence labeling.

For example, while analyzing a sentence, the hidden state can carry information about words that appeared earlier, helping the network interpret the current word in context.

Speech and Audio Processing

Speech is naturally sequential because sound signals change over time. RNNs can process successive portions of an audio signal and use previous information to interpret the current portion. This has made recurrent architectures useful in speech recognition and audio classification.

Time-Series Forecasting

Many datasets consist of observations collected at regular intervals, such as temperature, electricity demand, sales, or financial measurements. RNNs can use previous observations to help predict future values.

For example:

Past Measurements → RNN → Hidden State → Future Prediction

The model can learn patterns such as trends, recurring behavior, and relationships between observations at different time steps.

Sequence Classification

Sometimes the goal is to assign a single category to an entire sequence. In this case, the RNN processes the complete sequence and uses the accumulated representation to make the final prediction.

Examples include sentiment classification, activity recognition, and certain types of anomaly detection.

The common principle across these applications is that the model does not treat each observation as independent. Instead, it considers the relationship between the current input and information from earlier time steps.

This ability to model sequential dependencies made RNNs an important milestone in deep learning and provided the foundation for more advanced approaches to sequence modeling.

Limitations of RNNs

Although RNNs introduced an effective way to model sequential information, they have several limitations that become increasingly important as sequence length and model complexity grow.

Difficulty with Long-Term Dependencies

A basic RNN repeatedly passes information through its hidden state. When a sequence is very long, information from earlier time steps may gradually lose its influence. This makes it difficult for the network to connect events that are separated by many intermediate steps.

LSTMs and GRUs reduce this problem through their gating mechanisms, but they still process sequences sequentially.

Sequential Computation

RNNs process one time step after another:

$x_1 \rightarrow x_2 \rightarrow x_3 \rightarrow \cdots \rightarrow x_T$

Because the computation at one time step depends on the previous hidden state, later steps generally cannot be processed until earlier steps have been computed. This limits the amount of parallel processing available during training.

Training Difficulties

Long sequences can lead to vanishing or exploding gradients during Backpropagation Through Time. Techniques such as gradient clipping, careful initialization, and gated architectures can help, but training can still be more challenging than with architectures designed for greater parallelism.

Limited Context Representation

The hidden state is expected to carry useful information from the sequence. As the sequence becomes longer, compressing all relevant information into this evolving representation can become difficult. Important details may be weakened or lost.

These limitations do not make RNNs ineffective. Rather, they highlight why the field continued to develop more sophisticated approaches. LSTMs and GRUs improved memory management, while later attention mechanisms provided a different way to connect information across distant positions in a sequence.

The development from RNNs to attention-based models represents an important shift in deep learning: instead of relying primarily on a recurrent hidden state to carry information forward one step at a time, models can directly determine which parts of a sequence are most relevant to each other.

From Recurrent Memory to Attention

Recurrent Neural Networks introduced an important approach for working with sequential data by allowing information from earlier time steps to influence later computations. Through the hidden state, an RNN maintains a running representation of the sequence and uses that context when producing predictions.

The development of LSTMs and GRUs addressed important limitations of basic RNNs by introducing mechanisms that provide greater control over how information is retained, updated, and discarded. These architectures made it more practical to learn longer-term dependencies and became useful for language processing, speech recognition, time-series forecasting, and other sequential tasks.

However, RNNs also have important limitations. Their sequential nature makes parallel processing difficult, while very long sequences can still create challenges in preserving useful information and training the network effectively. These limitations encouraged researchers to explore a different way of modeling relationships within sequences.

Rather than passing information primarily from one time step to the next, attention mechanisms allow a model to directly examine different parts of a sequence and determine which information is most relevant to the current computation.

This idea led to the development of Transformers, which have become one of the most influential architectures in modern artificial intelligence and form the foundation of many contemporary language and multimodal models.