Deep Learning for Generative AI — Part 6: Introduction to the Transformer

In Part 5 we ended with a wish: what if we could compute all the representations at once, in parallel, instead of one word at a time? The Transformer is the answer to that wish.

This part has no math. Just two ideas, explained as simply as possible:

1. The Transformer still has an encoder and a decoder — the jobs are the same, only the machinery changes.
2. The new machinery is built on Query, Key, Value — which you already understand from everyday life, as we will see with a group project analogy.

1. Relax — the encoder-decoder is not going away

Everything you learned in Parts 2–4 still applies. The Transformer keeps the same big picture:

input sentence → [ENCODER: understand it] → [DECODER: generate output, one word at a time] → output sentence

Think of how a human translates: you read the input sentence, build an understanding of it in your head, and then produce the translation one word at a time. That is still exactly the plan.

What changes is how each box does its job. The RNN inside is thrown away. Attention — which used to be a helper on the side — becomes the only mechanism.

2. The encoder’s job: make every word’s representation richer

The encoder receives the input words as vectors (the embeddings from Part 1):

Input:  x1, x2, ..., xn        e.g.  x1 = vector for "I"
                                     x2 = vector for "love"
                                     x3 = vector for "cats"

These starting vectors are not good enough on their own. Why? Because each one describes its word in isolation. But the meaning of a word depends on the words around it:

"bank" in "river bank"    → means the side of a river
"bank" in "bank account"  → means a financial institution

Same starting vector. Different meanings. The vector must be enriched by its neighbours.

So the encoder’s job is:

x1, x2, ..., xn   →   [ENCODER]   →   z1, z2, ..., zn

plain word vectors                    RICH representations —
(each word alone)                     each z captures its word
                                      PLUS its relationships
                                      with every other word

This philosophy is not new — it is the same “representation of each word depends on other words” idea from the RNN encoder. The difference is how the dependency is computed: not by passing a memory left to right, but by an attention mechanism where every word directly looks at every other word — all at once, in parallel.

3. The decoder’s job: generate the output, one word at a time

The decoder takes the rich representations z1...zn and produces the output words:

z1, z2, ..., zn   →   [DECODER]   →   y1, y2, ..., ym

Note the letters: input has n words, output has m words. They do not have to match — “I love cats” (3 words) becomes “J’aime les chats” (4 tokens). Some languages need more words, some fewer. Same as Part 2.

Words are still generated one at a time. And to generate the next word y_i, the decoder has exactly two sources of information:

To generate the next word, the decoder looks at:

  1. The encoder outputs z1...zn        → "what does the INPUT say?"
  2. The words generated so far y1...y_{i-1}  → "what have I SAID so far?"

This matches how you translate. You do not translate word 1, then word 2, mechanically. At every moment you keep two things in mind: what the source sentence means and what you have already written — because the next word must fit both.

So the decoder needs TWO kinds of attention

One attention mechanism per information source:

┌──────────────────────────────────────────────────────┐
│                      DECODER                          │
│                                                       │
│  Attention #1: SELF-ATTENTION                         │
│    looks at →  y1 ... y_{i-1}                         │
│    answers →  "what have I generated so far?"         │
│                                                       │
│  Attention #2: CROSS-ATTENTION (encoder-decoder)      │
│    looks at →  z1 ... zn                              │
│    answers →  "which parts of the input matter now?"  │
└──────────────────────────────────────────────────────┘

Attention #2 is essentially the attention you already know from Part 3 — the decoder looking back at the encoder. Attention #1 (self-attention) is the new star of the show, and the group project analogy below is the intuition for it.

4. The group project analogy: understanding Query, Key, Value

Here is the single best intuition for how Transformer attention works. Think back to any successful group project you have done.

The setup

You are a team of 5. You are given a task — say, understanding a difficult book. The plan every good team uses:

Phase 1: everyone works INDEPENDENTLY (and in parallel!)
Phase 2: everyone meets to DISCUSS and share

Phase 1: independent work — everyone builds three things

Say tonight, between 7 and 8 PM, all five of you read chapter 5 — at the same time, independently. Nobody waits for anybody. (Remember this — it matters later.)

While reading, three things happen to each of you:

Concrete example — you just watched a lecture on Transformers:

Your VALUE:  "I understood the overall encoder-decoder structure"      (what you know)
Your QUERY:  "But how does the decoder work in translation exactly?"   (what you're missing)
Your KEY:    "I can explain how attention weights are computed"        (what you can offer)

Every team member walks away from their reading with their own V, their own Q, and their own K.

Phase 2: the discussion — queries meet keys

The next day you all meet. You put your questions (queries) on the table. Now the natural thing happens:

The team members whose insights (keys) match your questions (queries) will answer you.

Your query:      "How does the decoder work in translation?"

Friend 3's key:  "I can explain decoders"           → STRONG match → they answer, you listen A LOT
Friend 5's key:  "I know about attention weights"    → partial match → you listen a bit
Friend 2's key:  "I only understood the same things  → no match     → you gain little from them
                  you already know"

How much you attend to each friend depends on how well their key matches your query. That is literally where the word “attention” comes from.

What you walk away with: collective knowledge

After the discussion, your understanding is no longer just your own value. It is a weighted blend of everyone’s values:

your new understanding =
      0.70 × friend 3's value     (they answered most of your questions)
    + 0.20 × friend 5's value     (they helped a little)
    + 0.05 × friend 2's value     (barely helped)
    + 0.05 × friend 4's value     (barely helped)

Sounds familiar? This is exactly the weighted sum / convex combination from Part 4. The weights are decided by query-key matching, and what gets blended are the values.

And here is the beautiful part: everyone does this simultaneously. You attend to friends 3 and 5. Friend 2 attends to friends 1 and 6. Each person ends up with their own custom blend — their own enriched understanding.

The full mapping

This is self-attention: every word plays both roles at once — it asks its own questions of all the other words, and it answers the other words’ questions. After one round of this “discussion,” every word’s representation is enriched by exactly the words that were relevant to it.

Before self-attention:   "bank" = generic vector (could mean anything)

"bank" asks its neighbours: "what kind of bank am I?"
"river" has a matching key:  "I can tell you — the water kind!"

After self-attention:    "bank" = enriched vector, blended with "river"'s value
                         (now clearly the river bank)

A concrete example: what does “it” mean?

Self-attention is especially good at resolving words whose meaning depends on other words. Take this sentence:

"The animal did not cross the street because it was too tired."

What does “it” refer to? To a human it is obviously the animal. But the starting vector for “it” is just a generic placeholder — it does not know it is an animal, a street, or anything else.

After self-attention, the vector for “it” becomes a weighted blend of the other words. In a trained model the attention weights might look roughly like this:

it →  animal  (0.70)  ← strongest: "it" is the animal
       tired   (0.15)
       street  (0.08)
       cross   (0.04)
       because (0.02)
       the     (0.01)

The model did not learn English grammar explicitly. It learned that, in this kind of sentence, “it” usually points back to a noun like “animal,” so it borrows most of “animal”’s meaning when building “it”’s representation. This is the famous attention-visual effect: the vector for one word is “baked together” from the words it attends to.

5. Why this beats the RNN: the sequential group project

Now imagine running the group project the way an RNN works:

Day 1: Friend 1 reads the chapter, writes a summary, passes it to Friend 2.
Day 2: Friend 2 reads the chapter + the summary, updates it, passes it on.
Day 3: Friend 3 does the same...
...

Ten team members, deadline in 4 days — impossible. Each person waits for the previous one. Information from Friend 1 gets diluted by the time it reaches Friend 10 (the telephone-game problem from Part 5). Only one person works at any moment.

The Transformer’s way:

Everyone reads tonight, IN PARALLEL.        ← all Q, K, V computed at once
Tomorrow, one discussion session.           ← attention: queries matched to keys
Everyone leaves with enriched knowledge.    ← weighted sums of values

This solves precisely the drawbacks from Part 5: no more sequential computation, no more telephone-game signal loss — any word can directly attend to any other word, no matter how far apart.

6. Putting it together: the Transformer at a glance

        INPUT: "I love cats"
              ↓
   ┌─────────────────────────┐
   │        ENCODER          │   every word computes its Q, K, V in parallel,
   │   (self-attention)      │   holds a "discussion," and walks away with
   │                         │   an enriched representation
   └─────────────────────────┘
              ↓
        z1, z2, z3            (rich representations of "I", "love", "cats")
              ↓
   ┌─────────────────────────┐
   │        DECODER          │   attention #1 (self):  what have I said so far?
   │  (two attentions)       │   attention #2 (cross): what does the input say?
   │                         │
   └─────────────────────────┘
              ↓
        OUTPUT: "J'" → "aime" → "les" → "chats"   (one word at a time)

7. Summary

• The Transformer keeps the encoder-decoder concept: encoder understands the input, decoder generates the output one word at a time, output length can differ from input length.
• The encoder converts plain word vectors x1...xn into rich representations z1...zn, where each word’s representation is enriched by all other words.
• The decoder uses two attentions: self-attention over what it has generated so far (y1...y_{i-1}), and cross-attention over the encoder outputs (z1...zn).
• Query, Key, Value = your questions, your insights, your understanding — from the group project analogy.
• Attention = your queries get answered by whoever’s keys match them, and you walk away with a weighted blend of the group’s values.
• Everything is computed in parallel — like team members reading independently at the same time — which is exactly what the RNN could not do.

Next up: watch the encoder enrich a real sentence, word by word, in Part 7: Transformer Case Study — Python Program. (The actual math of self-attention — how Q, K, V are computed with learned matrices — comes in Part 9.)