Part 16: RNN Use Cases - Time Series, Sequence Classification and Labelling - A Friendly Guide
Part 16: RNN Use Cases — Time Series, Sequence Classification, and Labelling — A Friendly Guide
In Parts 14 and 15, you learned how RNNs and LSTMs work.
Now let us use them on real problems.
But before we dive into specific tasks, we need to understand something important: there are different patterns for how sequences can go in and come out of an RNN.
0. Sequence modeling patterns — what goes in, what comes out
An RNN takes a sequence as input. But what it outputs can be very different depending on the task.
There are three main patterns:
| Pattern | Input | Output | Example tasks |
|---|---|---|---|
| Many-to-One | Many tokens | One answer | Sentiment analysis, spam detection, time series forecast |
| Many-to-Many | Many tokens | Many tokens | POS tagging, NER, summarization, translation |
| One-to-Many | One input | Many tokens | Image captioning, music generation |
Pattern 1 — Many to One
Input: a sequence of many tokens (many steps) Output: a single answer for the whole sequence
The RNN reads every step and builds memory. At the very last step, you take the final memory (the final hidden state) and pass it through a linear layer to get one answer.
Real examples:
- Sentiment analysis — input: “The food was not bad at all” (7 words) → output: Positive
- Spam detection — input: email words → output: Spam or Not Spam
- Time series forecasting — input: last 7 days of temperature → output: tomorrow’s temperature
- Document classification — input: article → output: Sports / Politics / Tech
All classification tasks where you want one label for the whole sequence fall into Many-to-One.
Pattern 2 — Many to Many
Input: a sequence of many tokens Output: a sequence of many tokens (one per input step, or a different-length output sequence)
There are two sub-types:
Sub-type A — Same length output (one tag per input token)
The RNN produces an output at every step, not just the last one. You take the hidden state at each step and pass it through a linear layer to get a tag for that position.
Real examples (same-length):
- Part-of-speech (POS) tagging — input: “The cat sat” → output: “DET NOUN VERB” (one tag per word)
- Named entity recognition (NER) — input: “Barack Obama visited Paris” → output: “PERSON PERSON O LOCATION”
- Sequence labelling — any task where every input token gets its own label
Sub-type B — Different length output (encoder-decoder / seq2seq)
The input sequence and output sequence can be different lengths. You encode the whole input into a final memory vector, then use a separate decoder RNN to generate the output sequence one token at a time.
Real examples (different-length):
- Machine translation — input: “Bonjour le monde” (French, 3 words) → output: “Hello world” (English, 2 words)
- Text summarization — input: long document (500 words) → output: summary (50 words)
- Dialogue generation — input: “How are you?” → output: “I am doing well, thank you.”
Both sub-types fall into Many-to-Many. The difference is whether output length equals input length or not.
Pattern 3 — One to Many
Input: a single item (not a sequence) Output: a sequence of many tokens generated one at a time
There is no sequence coming in — just one thing (an image, a number, a label). The decoder RNN generates output step by step, using its own previous output as input to the next step.
Real examples:
- Image captioning — input: one photo → output: “A cat sitting on a wooden chair” (many words)
- Music generation — input: one starting note → output: a sequence of notes
- Story generation — input: one prompt word → output: many sentences
The key: the input is not a sequence. The model generates the output sequence from scratch, conditioned on just one thing.
All three patterns at a glance
Now let us look at how to implement each pattern in PyTorch, starting with the most common ones.
1. Time series forecasting — predicting what comes next (Many-to-One)
What is time series forecasting?
Time series forecasting means predicting future values based on data collected over time.
Unlike regular data — where each row is independent — time series data is sequential. Each value depends on what came before. The order matters.
Examples of time series data:
| Data | Time step | What you want to predict |
|---|---|---|
| Daily stock prices | One day | Tomorrow’s price |
| Hourly temperatures | One hour | Next hour’s temperature |
| Monthly sales | One month | Next month’s sales |
| Website traffic | One day | Tomorrow’s visitor count |
| Patient heart rate | One second | Next second’s reading |
Why sequential order matters
Imagine you shuffle the daily temperatures for the past month randomly. The numbers are still there — but the pattern is destroyed. You can no longer tell if temperatures are rising, falling, or cycling. A model trained on shuffled data would learn nothing useful.
This is exactly why a normal neural network cannot do time series forecasting well — it sees all inputs at once without any sense of order. An RNN reads values one at a time, in order, building memory as it goes. That is what makes it suitable.
What patterns does a time series contain?
Good forecasting comes from recognising the patterns hidden in past data. There are four main types:
1 — Trend: a long-term direction, either rising or falling.
Example: Monthly sales slowly increasing over a year as the business grows.
2 — Seasonality: a repeating pattern at regular intervals.
Example: Electricity usage spikes every summer (air conditioning) and every winter (heating). It repeats at the same time every year.
3 — Cycles and fluctuations: irregular rises and falls caused by external factors.
Example: Sales jumping unexpectedly after a news story, or dipping during an economic downturn. Less regular than seasonality.
4 — Anomalies: unusual values that do not fit the pattern at all.
Example: A sudden spike in server traffic that indicates a security incident or a viral post.
An RNN can learn to recognise all four of these patterns from the sequence of past values — because it processes them in order and remembers what it has seen.
Why RNNs are good at this
A simple model (like linear regression) can follow a straight trend. But it cannot capture:
- Patterns that depend on what happened 7 steps ago (weekly seasonality)
- Patterns that change over time (an upward trend that accelerates)
- Interactions between trend and anomaly
An RNN remembers past steps. Its hidden state at step 30 contains information about steps 1–29. This lets it capture complex, time-dependent relationships that simpler models miss.
The task in one line
Given the last N values in a sequence, predict the next value.
x₁ → x₂ → x₃ → x₄ → x₅ → [final hidden state] → [Linear] → predicted x₆
This is a Many-to-One pattern: many input steps, one output prediction.
Step by step example — weekly sales forecasting
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Fake weekly sales data (units sold)
sales = [120, 135, 128, 142, 150, 145, 160, 155, 170, 165,
180, 175, 190, 185, 200, 195, 210, 205, 220, 215]
# Step 1 — Create sequences
# Input: 4 weeks. Target: week 5.
SEQ_LEN = 4
X, y = [], []
for i in range(len(sales) - SEQ_LEN):
X.append(sales[i : i + SEQ_LEN])
y.append(sales[i + SEQ_LEN])
# Step 2 — Convert to tensors
# Normalise: divide by 200 so numbers are between 0 and 1
scale = 200.0
X = torch.tensor(X, dtype=torch.float32).unsqueeze(-1) / scale # (N, 4, 1)
y = torch.tensor(y, dtype=torch.float32).unsqueeze(-1) / scale # (N, 1)
print(f"Training samples: {len(X)}")
print(f"X shape: {X.shape} y shape: {y.shape}")
# Step 3 — Build the model
class SalesForecaster(nn.Module):
def __init__(self):
super().__init__()
self.lstm = nn.LSTM(input_size=1, hidden_size=32, batch_first=True)
self.linear = nn.Linear(32, 1)
def forward(self, x):
out, (h, c) = self.lstm(x)
return self.linear(h.squeeze(0)) # use final hidden state
model = SalesForecaster()
loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)
# Step 4 — Train
for epoch in range(500):
pred = model(X)
loss = loss_fn(pred, y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if epoch % 100 == 0:
print(f"Epoch {epoch} loss: {loss.item():.6f}")
# Step 5 — Predict next week
model.eval()
with torch.no_grad():
last_4 = torch.tensor([[[205.0], [220.0], [215.0], [225.0]]]) / scale
predicted = model(last_4).item() * scale
print(f"\nGiven sales: 205, 220, 215, 225")
print(f"Predicted next week: {predicted:.0f} units")
Key decisions for time series
| Decision | Recommendation |
|---|---|
| Sequence length | Start with 4–10 steps. Try longer if the model is bad. |
| Normalise data | Always divide by a scale factor. Values between 0 and 1 train much better. |
| Use LSTM not RNN | LSTM remembers longer patterns — almost always better. |
| Loss function | MSELoss for continuous values (temperature, sales, prices) |
2. RNNs and language — understanding text as a sequence
Before we look at specific NLP tasks, it helps to understand why RNNs are a natural fit for language.
Language is temporal — order changes meaning
Human language unfolds over time. When you read or listen, words arrive one after another. Each word builds on what came before.
The order of words is what creates meaning:
“The dog chased the cat” — the dog is the one doing the chasing. “The cat chased the dog” — completely different meaning. Same words, different order.
This means a model that processes all words at once — ignoring order — cannot understand language properly. An RNN processes words one at a time, left to right, building memory as it goes. That matches how humans read.
The hidden state as evolving context
As the RNN reads each word, its hidden state is updated. Think of the hidden state as a running summary of everything read so far.
The hidden state at the end contains context from the whole sentence. This is what gets passed to the classification layer.
Short-range vs long-range dependencies
RNNs can capture two kinds of relationships in text:
Short-range — words that are close together and must be read as a unit.
“New York City” — “New” and “York” only make sense together. Without remembering “New”, “York” means nothing.
Long-range — a word near the end depends on something said much earlier.
“The book that I borrowed from the library last week was fascinating.” — “was fascinating” describes “book”, even though many words come between them.
Basic RNNs struggle with long-range dependencies (the vanishing gradient problem from Part 14). LSTM and GRU handle them much better, which is why they are used for most NLP tasks.
3. Sequence classification — one label for the whole sequence (Many-to-One)
The task: Read a whole sequence and output one single label.
Examples:
- Read all words in a review → positive or negative (sentiment analysis)
- Read all words in an email → spam or not spam
- Read all words in a support ticket → which department to route to
- Read a sentence → detect emotion (happy, sad, angry)
- Read a sentence → detect sarcasm or humour
How it works — sentiment analysis in detail
Same as forecasting — the RNN reads the whole sequence. You take the final hidden state and classify it.
But for classification there are multiple possible classes (positive, negative, neutral), so we use a softmax layer to get a probability for each class:
word₁ → word₂ → ... → wordₙ → [final hidden state]
↓
[Linear layer]
↓
[Softmax]
↓
P(positive)=0.82 P(negative)=0.12 P(neutral)=0.06
↓
Pick highest: POSITIVE
The LSTM remembers the word “not” when it reaches “bad”, so it correctly classifies “not bad” as positive — something a bag-of-words model would get wrong.
Other tasks that use this exact same pattern:
- Emotion recognition (happy / sad / angry / surprised)
- Humour detection (funny / not funny)
- Sarcasm detection
- Topic classification (sports / politics / technology)
Step by step example — sentiment classification
import torch
import torch.nn as nn
# Vocabulary — every word the model knows
word_to_idx = {
"the": 0, "food": 1, "was": 2, "good": 3, "bad": 4,
"not": 5, "really": 6, "very": 7, "service": 8, "great": 9,
"awful": 10, "excellent": 11, "ok": 12, "terrible": 13,
"<pad>": 14
}
VOCAB_SIZE = len(word_to_idx)
# Sentiment labels: 0 = negative, 1 = positive
sentences = [
(["the", "food", "was", "great"], 1), # positive
(["the", "food", "was", "terrible"], 0), # negative
(["the", "food", "was", "not", "bad"], 1), # positive (tricky!)
(["the", "service", "was", "excellent"], 1), # positive
(["the", "service", "was", "not", "good"], 0), # negative (tricky!)
(["really", "awful", "food"], 0), # negative
(["very", "good", "service"], 1), # positive
]
def encode(words):
return torch.tensor([[word_to_idx[w] for w in words]])
X = [encode(s[0]) for s in sentences]
y = torch.tensor([s[1] for s in sentences], dtype=torch.float32)
# Model: word → embedding → LSTM → final hidden state → Linear → Sigmoid → probability
class SentimentClassifier(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(VOCAB_SIZE, 16) # each word → 16 numbers
self.lstm = nn.LSTM(input_size=16, hidden_size=32, batch_first=True)
self.linear = nn.Linear(32, 1) # hidden state → 1 score
self.sigmoid = nn.Sigmoid() # score → probability (0 to 1)
def forward(self, x):
emb = self.embedding(x) # (1, seq_len, 16)
out, (h, c) = self.lstm(emb) # h: (1, 1, 32)
return self.sigmoid(self.linear(h.squeeze(0))).squeeze()
model = SentimentClassifier()
loss_fn = nn.BCELoss() # binary cross-entropy for two-class problem
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
# Train
for epoch in range(400):
total_loss = 0
for xi, yi in zip(X, y):
pred = model(xi)
loss = loss_fn(pred, yi)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
if epoch % 100 == 0:
print(f"Epoch {epoch} loss: {total_loss/len(X):.4f}")
# Test — including tricky cases
model.eval()
test = [
["the", "food", "was", "not", "bad"], # tricky positive
["really", "terrible", "service"], # negative
["very", "good", "food"], # positive
]
with torch.no_grad():
for words in test:
x = encode(words)
prob = model(x).item()
label = "POSITIVE" if prob >= 0.5 else "NEGATIVE"
print(f"'{' '.join(words)}' → {label} (confidence: {prob:.2f})")
Expected output:
'the food was not bad' → POSITIVE (confidence: 0.78)
'really terrible service' → NEGATIVE (confidence: 0.08)
'very good food' → POSITIVE (confidence: 0.91)
The LSTM correctly handles “not bad” as positive because it remembers “not” when it reads “bad”.
4. Sequence labelling — one label per step (Many-to-Many, same length)
The task: Read a sequence and output one label at every position.
Examples:
- Read each word in a sentence → label each word as Noun, Verb, Adjective, etc. (POS tagging)
- Read each word in a sentence → label each word as Person, Location, Organisation, or Other (Named Entity Recognition)
- Read each character in a DNA sequence → label each position as coding or non-coding
- Read each frame in a video → label each frame as “action” or “no action”
How it works
For labelling, you do not take just the final hidden state. You take the output at every step.
word₁ → [LSTM] → output₁ → [Linear] → label₁
word₂ → [LSTM] → output₂ → [Linear] → label₂
word₃ → [LSTM] → output₃ → [Linear] → label₃
In PyTorch, nn.LSTM returns output — the hidden state at every step. Use this for sequence labelling.
Step by step example — part-of-speech tagging
import torch
import torch.nn as nn
# Vocabulary and tags
word_to_idx = {
"the": 0, "cat": 1, "sat": 2, "on": 3, "mat": 4,
"dog": 5, "ran": 6, "fast": 7, "a": 8, "big": 9,
"<pad>": 10
}
tag_to_idx = {"DET": 0, "NOUN": 1, "VERB": 2, "PREP": 3, "ADJ": 4}
IDX_TO_TAG = {v: k for k, v in tag_to_idx.items()}
# Training data: (sentence, tags) — one tag per word
training_data = [
(["the", "cat", "sat", "on", "mat"], ["DET", "NOUN", "VERB", "PREP", "NOUN"]),
(["a", "dog", "ran", "fast"], ["DET", "NOUN", "VERB", "ADJ"]),
(["the", "big", "dog", "ran"], ["DET", "ADJ", "NOUN", "VERB"]),
(["a", "cat", "sat", "on", "mat"], ["DET", "NOUN", "VERB", "PREP", "NOUN"]),
]
def encode_sentence(words):
return torch.tensor([word_to_idx[w] for w in words])
def encode_tags(tags):
return torch.tensor([tag_to_idx[t] for t in tags])
# Model
class POSTagger(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(len(word_to_idx), 16)
self.lstm = nn.LSTM(input_size=16, hidden_size=32, batch_first=True)
self.linear = nn.Linear(32, len(tag_to_idx)) # one score per tag
def forward(self, x):
emb = self.embedding(x.unsqueeze(0)) # (1, seq_len, 16)
output, _ = self.lstm(emb) # (1, seq_len, 32)
return self.linear(output.squeeze(0)) # (seq_len, num_tags)
model = POSTagger()
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
# Train
for epoch in range(300):
total_loss = 0
for words, tags in training_data:
x = encode_sentence(words)
y = encode_tags(tags)
pred = model(x) # shape: (seq_len, num_tags)
loss = loss_fn(pred, y) # compare each position
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
if epoch % 100 == 0:
print(f"Epoch {epoch} loss: {total_loss/len(training_data):.4f}")
# Test — tag a new sentence
model.eval()
test_sentence = ["the", "big", "cat", "sat"]
with torch.no_grad():
x = encode_sentence(test_sentence)
pred = model(x)
predicted_tags = pred.argmax(dim=1).tolist()
print("\nWord-by-word tagging:")
for word, tag_idx in zip(test_sentence, predicted_tags):
print(f" {word:10s} → {IDX_TO_TAG[tag_idx]}")
Output:
Word-by-word tagging:
the → DET
big → ADJ
cat → NOUN
sat → VERB
5. Language modelling — predicting the next word (Many-to-One repeated)
What is a language model?
A language model is a model that understands natural language well enough to predict what word comes next.
Given an input sequence of words, it outputs a probability distribution over every word in the vocabulary — a score for each possible next word.
Input: "For space tourist return to"
Output: P(Earth)=0.61 P(Mars)=0.18 P(space)=0.09 P(orbit)=0.07 ...
→ Pick highest: "Earth"
This is how autocomplete works. The model reads what you have typed so far and predicts the most likely next word.
Decoding strategies — how to pick the next word
Once the model produces probabilities, how do we choose the next word?
Greedy decoding — always pick the word with the highest probability.
- Fast and simple.
- Can get stuck in repetitive or dull outputs.
Beam search — keep the top-K most likely sequences at each step, explore all of them, then pick the best overall.
- More expensive but produces better, more coherent text.
- Used in machine translation and text summarisation.
Sampling — randomly pick a word according to the probability distribution.
- Produces more varied, creative output.
- Used in chatbots and story generation.
import torch
import torch.nn as nn
import torch.nn.functional as F
# Tiny vocabulary for demonstration
words = ["for", "space", "tourist", "return", "to", "earth", "mars", "orbit", "<eos>"]
w2i = {w: i for i, w in enumerate(words)}
i2w = {i: w for w, i in w2i.items()}
VOCAB = len(words)
class LanguageModel(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(VOCAB, 16)
self.lstm = nn.LSTM(input_size=16, hidden_size=32, batch_first=True)
self.linear = nn.Linear(32, VOCAB) # output one score per vocab word
def forward(self, x, hidden=None):
emb = self.embedding(x) # (batch, seq, 16)
out, hidden = self.lstm(emb, hidden) # out: (batch, seq, 32)
logits = self.linear(out) # (batch, seq, VOCAB)
return logits, hidden
model = LanguageModel()
# --- Greedy decoding: generate 3 words after a prompt ---
model.eval()
prompt = ["for", "space", "tourist", "return", "to"]
with torch.no_grad():
x = torch.tensor([[w2i[w] for w in prompt]])
logits, hidden = model(x)
# Take logits at the last step only
last_logits = logits[0, -1, :] # (VOCAB,)
probs = F.softmax(last_logits, dim=-1)
next_word_idx = probs.argmax().item() # greedy: pick highest probability
print(f"Prompt: '{' '.join(prompt)}'")
print(f"Predicted next word: '{i2w[next_word_idx]}'")
print(f"Top 3 probabilities:")
top3 = probs.topk(3)
for prob, idx in zip(top3.values, top3.indices):
print(f" {i2w[idx.item()]:10s}: {prob.item():.3f}")
6. Machine translation — sequence to sequence (Many-to-Many, different length)
What is machine translation?
Given a sentence in one language (the source), generate the translation in another language (the target).
Source (English): "We are attending upgrade classes"
Target (Hindi): "Hum upgrade classes attend kar rahe hain"
The input and output are both sequences — but they can be different lengths. This is a Many-to-Many problem with variable output length.
How it works — Encoder + Decoder
This architecture has two parts:
Encoder — reads the entire source sentence word by word. At the end, the final hidden state becomes a context vector — a compressed summary of the whole source sentence.
Decoder — takes the context vector as its starting hidden state. It then generates the target language one word at a time, using its own previous output as the next input.
The decoder generates one word at a time. Each generated word becomes the input for the next step. It stops when it generates a special <eos> (end of sentence) token.
Key point — same architecture, different use
The encoder and decoder are both LSTM networks. But:
- The encoder uses the Many-to-One pattern — reads all source words, outputs one context vector
- The decoder uses the One-to-Many pattern — takes one context vector, generates many target words
Together they form a seq2seq (sequence-to-sequence) model.
import torch
import torch.nn as nn
# Tiny vocabulary for both languages
src_vocab = {"we": 0, "are": 1, "attending": 2, "classes": 3, "<sos>": 4, "<eos>": 5}
tgt_vocab = {"hum": 0, "upgrade": 1, "attend": 2, "kar": 3, "rahe": 4, "hain": 5,
"<sos>": 6, "<eos>": 7}
SRC_V, TGT_V = len(src_vocab), len(tgt_vocab)
class Encoder(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(SRC_V, 16)
self.lstm = nn.LSTM(16, 32, batch_first=True)
def forward(self, x):
emb = self.embedding(x)
_, (h, c) = self.lstm(emb) # we only want the final hidden state
return h, c # this is the context vector
class Decoder(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(TGT_V, 16)
self.lstm = nn.LSTM(16, 32, batch_first=True)
self.linear = nn.Linear(32, TGT_V)
def forward(self, x, hidden):
emb = self.embedding(x) # (1, 1, 16) — one word at a time
out, hidden = self.lstm(emb, hidden) # out: (1, 1, 32)
logits = self.linear(out.squeeze(1)) # (1, TGT_V)
return logits, hidden
encoder = Encoder()
decoder = Decoder()
# --- Forward pass (inference sketch) ---
encoder.eval()
decoder.eval()
with torch.no_grad():
# Source sentence: "we are attending classes"
src = torch.tensor([[src_vocab["we"], src_vocab["are"],
src_vocab["attending"], src_vocab["classes"]]])
context_h, context_c = encoder(src) # compress source to context vector
# Decoder starts with <sos> token and the context vector
dec_input = torch.tensor([[tgt_vocab["<sos>"]]]) # start token
hidden = (context_h, context_c)
i2w_tgt = {v: k for k, v in tgt_vocab.items()}
print("Generated translation:")
for _ in range(6): # generate up to 6 words
logits, hidden = decoder(dec_input, hidden)
next_word = logits.argmax(dim=1).item()
word = i2w_tgt[next_word]
if word == "<eos>":
break
print(f" {word}")
dec_input = torch.tensor([[next_word]]) # feed output as next input
Other tasks that use this exact seq2seq pattern:
- Text summarisation — source: long document → target: short summary
- Dialogue generation — source: user message → target: bot reply
- Code generation — source: English description → target: code
7. Which output to use — a clear guide
This is the most common source of confusion. Here is a simple rule:
output, (h_n, c_n) = lstm(x)
| Task | Which output to use | Why |
|---|---|---|
| Time series forecasting | h_n — the final hidden state |
You want one number at the end |
| Sequence classification | h_n — the final hidden state |
You want one label at the end |
| Sequence labelling | output — hidden state at every step |
You want one label at every position |
# For forecasting or classification:
out, (h_n, c_n) = lstm(x)
result = linear(h_n.squeeze(0)) # use h_n → shape (batch, hidden)
# For labelling:
out, (h_n, c_n) = lstm(x)
result = linear(out) # use output → shape (batch, seq_len, hidden)
8. Choosing the right sequence model — summary
| Problem | Pattern | Model | Key detail |
|---|---|---|---|
| Predict next value | Many-to-One | LSTM | Use h_n (final hidden state) |
| Sentiment / spam / emotion classification | Many-to-One | LSTM | Use h_n + softmax |
| Label each token (POS, NER) | Many-to-Many (same length) | Bidirectional LSTM | Use output (all steps) |
| Language modelling / autocomplete | Many-to-One (repeated) | LSTM | Output logits over vocab, apply softmax |
| Machine translation / summarisation | Many-to-Many (different length) | Encoder-Decoder LSTM | Encoder → context vector → Decoder generates |
| Very long sequences (1000+ steps) | Any | Transformer (covered later) | Attention replaces hidden state |
9. Common patterns in real projects
Pattern 1 — Normalise your time series data
# Always normalise time series before feeding to LSTM
mean = X.mean()
std = X.std()
X_normalised = (X - mean) / std # values centred around 0
# After predicting, scale back
prediction_real = prediction_normalised * std + mean
Pattern 2 — Use a DataLoader for sequences
from torch.utils.data import Dataset, DataLoader
class SequenceDataset(Dataset):
def __init__(self, X, y):
self.X = X
self.y = y
def __len__(self):
return len(self.X)
def __getitem__(self, idx):
return self.X[idx], self.y[idx]
dataset = SequenceDataset(X, y)
loader = DataLoader(dataset, batch_size=16, shuffle=True)
for X_batch, y_batch in loader:
pred = model(X_batch)
# ...
Pattern 3 — Add dropout to prevent overfitting
lstm = nn.LSTM(
input_size=1,
hidden_size=64,
num_layers=2,
dropout=0.3, # 30% dropout between layers
batch_first=True
)
Summary
Three main RNN use cases, each slightly different:
Time series forecasting:
past values → LSTM → final hidden state → Linear → next value
Sequence classification:
all words → LSTM → final hidden state → Linear → one label
Sequence labelling:
all words → LSTM → output at every step → Linear → label per word
- Always normalise time series data before training.
- Use
h_n(final hidden state) for forecasting and classification. - Use
output(all hidden states) for sequence labelling. - For tasks where context from both directions matters, use Bidirectional LSTM.
- Stack 2 LSTM layers and add dropout for better generalisation on real datasets.