Causal vs
Full Attention.

Who can see which token?

This planned seminar and practical work explores how attention masking changes the information available to a transformer at each position. Causal self-attention restricts a token to its own position and earlier positions. Full, bidirectional self-attention can use all permitted input positions.

The goal is to connect the mathematics of attention with practical PyTorch experiments, then study how information access interacts with the chosen task and evaluation protocol.

Research status: upcoming. No completed study, supervisor, publication, or experimental outcome is claimed. The matrix above illustrates a binary access pattern, not learned attention scores.

The mask is part of the problem.

Attention is often introduced as a common mechanism across transformer architectures. But the context a model receives is a defining design choice. In autoregressive generation, future tokens are unavailable at inference time. In representation-learning settings, the full input can be useful context.

Understanding this difference matters for choosing a training objective, preventing leakage, and interpreting what a model has learned.

Attention, made explicit.

The query-key dot products measure compatibility. Scaling by the square root of the key dimension helps control their magnitude. The additive mask M uses 0 for allowed positions and negative infinity for forbidden positions. Softmax converts the allowed scores into weights before combining the values.

Same operation. Different access.

Causal self-attention

A lower-triangular mask permits position i to attend only to positions j ≤ i. This matches left-to-right generation: the model predicts the next token using the prefix available so far. Decoder-style transformers commonly use this structure for autoregressive modeling.

Full self-attention

Every non-padding input position can attend to the other allowed input positions. The resulting representations can incorporate context from both directions. Encoder-style transformers use this to reason about a sequence that is already available.

A valid comparison must respect the information available for the task. Allowing the full-attention model to see a prediction target would create target leakage. A difference in access is not, by itself, evidence of a superior architecture.

Questions to test, carefully.

1. Read the theoretical background. Establish attention semantics, objectives, and the settings being compared.
2. Implement self-attention. Build queries, keys, values, scaled scores, and both mask types.
3. Choose tasks and datasets. Separate generative objectives from representation-learning objectives and define exactly what inputs are available.
4. Control the experiment. Match data splits, optimization budget, model dimensions, and random-seed reporting where the comparison permits it.
5. Evaluate and visualize. Inspect access masks, task-specific metrics, runtime, and qualitative behavior without conflating them.
6. Discuss limitations. State what the experiment can establish and what remains task-dependent.

A mask in a few lines.

import math
import torch

def self_attention(q, k, v, causal=True):
    # q, k, v: [batch, heads, tokens, features]
    scores = q @ k.transpose(-2, -1)
    scores = scores / math.sqrt(q.size(-1))

    if causal:
        n = q.size(-2)
        future = torch.triu(
            torch.ones(n, n, device=q.device, dtype=torch.bool),
            diagonal=1,
        )
        scores = scores.masked_fill(future, float("-inf"))

    weights = torch.softmax(scores, dim=-1)
    return weights @ v

This educational sketch assumes equal query and key sequence lengths and no padding. A complete implementation also needs appropriate padding masks, numerical checks, and tests for the chosen training setup.

The study is still ahead.

Experimental results will be added after the research begins. The eventual discussion will connect task performance, information access, and representation behavior. Conclusions will follow the evidence.

JEPA and world models are a separate research interest, with a different question: predicting future representations rather than reconstructing every detail of an input.

References.