Multimodal AI

Run a complete example · 01

Look up and pool two tokens

A tiny English example uses only lists and a dictionary; the explanation applies to representations in any language.

text = "red kite"
tokens = text.split()
vocabulary = {"red": 0, "kite": 1}
embedding_table = [[1.0, 0.0], [0.0, 2.0]]
ids = [vocabulary[token] for token in tokens]
vectors = [embedding_table[token_id] for token_id in ids]
pooled = [sum(vector[j] for vector in vectors) / len(vectors) for j in range(2)]
print("tokens:", tokens)
print("IDs:", ids)
print("pooled:", pooled)

python tokens-embeddings-toy-token-table.py

tokens: ['red', 'kite']
IDs: [0, 1]
pooled: [0.5, 1.0]

Read the code step by step

split separates at whitespace. The dictionary converts each token to an ID; square brackets look up a key or list position. The next comprehension retrieves each feature list. range(2) visits feature positions 0 and 1, sum adds matching entries and len gives two tokens. This is hand-set lookup and mean pooling, not a trained text encoder.

Run a complete example · 01

Compare two descriptions

The feature lists are hand-set to isolate cosine similarity, not produced by a trained encoder.

import math
def unit(values):
    length = math.sqrt(sum(x * x for x in values))
    if length == 0:
        raise ValueError("A zero vector has no direction")
    return [x / length for x in values]
image = unit([3, 4])
for name, values in [("A", [0, 2]), ("B", [2, 0])]:
    text = unit(values)
    similarity = sum(a * b for a, b in zip(image, text))
    print(f"{name}: {similarity:.2f}")

python modalities-alignment-cosine-alignment.py

A: 0.80
B: 0.60

Read the code step by step

def defines a reusable function. sum adds squared entries and math.sqrt gives length. The if guards division by zero. A list comprehension divides every entry by that length. The loop tests two named descriptions. zip pairs corresponding entries; multiplying and adding gives 0.80 and 0.60. These are ranking scores, not correctness probabilities.

Run a complete example · 01

Combine only available evidence

Use two supplied scores and handle one missing modality.

scores = {"image": 0.8, "audio": 0.4}
weights = {"image": 0.75, "audio": 0.25}
def fuse(available):
    total = sum(weights[name] for name in available)
    return sum(weights[name] * score for name, score in available.items()) / total
print(f"both={fuse(scores):.2f}")
print(f"image only={fuse({'image': 0.8}):.2f}")
print("early feature list:", [0.2, 0.9] + [0.6, 0.1])

python early-late-fusion-weighted-fusion.py

both=0.70
image only=0.80
early feature list: [0.2, 0.9, 0.6, 0.1]

Read the code step by step

Each dictionary names a score or weight. fuse sums weights for present keys only. items gives name–score pairs for the weighted sum; dividing by total makes available weights sum to one. The last + joins two lists, illustrating concatenation rather than numeric addition. The example assumes nonempty available evidence and positive total weight. It is a rule, not a trained multimodal model.

Run a complete example · 01

Check the time before matching

Count samples and compare supplied unit embeddings.

duration = 4
original_rate = 48000
new_rate = 16000
original_count = duration * original_rate
new_count = duration * new_rate
print("counts:", original_count, new_count)
print("wrong relabelled duration:", original_count / new_rate)
audio = [0.8, 0.6]
text = [0.6, 0.8]
print(f"similarity={sum(a * b for a, b in zip(audio, text)):.2f}")

python audio-text-alignment-audio-duration.py

counts: 192000 64000
wrong relabelled duration: 12.0
similarity=0.96

Read the code step by step

Multiplication counts 192000 original samples and 64000 required samples at the new rate. Dividing the old count by the new rate exposes the incorrect 12-second interpretation. zip pairs unit-embedding entries; products 0.48 and 0.48 sum to 0.96. The code neither resamples a waveform nor runs Whisper or CLAP; it checks timing and scoring arithmetic.

Run a complete example · 01

Score the correct partners both ways

Use a two-pair similarity table to compute the stated loss.

import math
similarity = [[0.8, 0.2], [0.1, 0.7]]
temperature = 0.2
scores = [[x / temperature for x in row] for row in similarity]
def loss(row, correct):
    weights = [math.exp(x - max(row)) for x in row]
    probability = weights[correct] / sum(weights)
    return -math.log(probability)
row_loss = sum(loss(row, i) for i, row in enumerate(scores)) / 2
columns = list(zip(*scores))
column_loss = sum(loss(column, i) for i, column in enumerate(columns)) / 2
print(f"image to text={row_loss:.4f}")
print(f"text to image={column_loss:.4f}")
print(f"combined={(row_loss + column_loss) / 2:.4f}")

python clip-contrastive-contrastive-table.py

image to text=0.0486
text to image=0.0543
combined=0.0515

Read the code step by step

Nested list comprehensions divide every table entry by temperature. loss subtracts the maximum before exp for stable arithmetic, then finds the correct share and its negative log. enumerate supplies each diagonal index. zip(*scores) groups entries by column: the star passes each row as a separate argument to zip. The two averages differ because competitors differ. This computes a loss on invented scores without training CLIP.

Run a complete example · 01

Let a query mix two values

All query, key and value lists are invented; this is one attention calculation.

import math
query = [1, 0]
keys = [[2, 0], [0, 2]]
values = [[10, 0], [0, 4]]
scores = [sum(q * k for q, k in zip(query, key)) / math.sqrt(2) for key in keys]
weights = [math.exp(s - max(scores)) for s in scores]
weights = [w / sum(weights) for w in weights]
output = [sum(weight * value[j] for weight, value in zip(weights, values)) for j in range(2)]
print("weights:", [round(w, 4) for w in weights])
print("output:", [round(x, 4) for x in output])

python cross-attention-fusion-attention-mix.py

weights: [0.8044, 0.1956]
output: [8.0443, 0.7823]

Read the code step by step

The first comprehension makes one scaled dot-product score per key. Exponentials followed by division create weights summing to one. In the output comprehension, j selects output coordinate 0 then 1; zip pairs each weight with its value list. Each coordinate is a weighted sum. The output is [8.0443,0.7823], not a copied patch or an object label. No trained transformer is run.

Run a complete example · 01

Find and check the top two

Use three supplied unit embeddings and independently checked relevance.

query = [1.0, 0.0]
images = {"kite": [0.8, 0.6], "umbrella": [1.0, 0.0], "kite-detail": [0.6, 0.8]}
scores = {name: sum(q * x for q, x in zip(query, vector)) for name, vector in images.items()}
ranked = sorted(scores, key=scores.get, reverse=True)
top_two = ranked[:2]
relevant = {"kite", "kite-detail"}
recall = len(set(top_two) & relevant) / len(relevant)
print("ranking:", ranked)
print(f"Recall@2={recall:.2f}")

python image-text-search-search-ranking.py

ranking: ['umbrella', 'kite', 'kite-detail']
Recall@2=0.50

Read the code step by step

The dictionary comprehension computes a dot product for each image. sorted uses each name’s score and reverse=True puts the largest first. [:2] takes the first two entries. set removes repeated names, & finds relevant returned items, and len counts them. Recall is 0.50 despite a high-scoring first result. This is ranking arithmetic, not a trained image-search service.

Run a complete example · 02

Evaluate retrieval with several relevant results per query

Mina searches an image collection with short text descriptions. One query has two valid pictures, another has one, and a third has no labeled match in the collection. A repeated image ID also appears in the ranked output. She removes duplicates, writes down which metric she means by recall, and reports unscored queries separately instead of hiding them.

from math import isclose

def retrieval_report(queries, k):
    if k <= 0:
        raise ValueError("positive k required")
    recalls, hits, reciprocal = [], [], []
    found_total = relevant_total = unscored = 0
    for ranked, relevant in queries:
        ranked = list(dict.fromkeys(ranked))  # Preserve the first occurrence.
        relevant = set(relevant)
        if not relevant:
            unscored += 1
            continue
        found = len(set(ranked[:k]) & relevant)
        recalls.append(found / len(relevant))
        hits.append(int(found > 0))
        first = next((i for i, item in enumerate(ranked, 1) if item in relevant), None)
        reciprocal.append(1 / first if first is not None else 0.0)
        found_total += found
        relevant_total += len(relevant)
    count = len(recalls)
    if not count:
        return {"scored": 0, "unscored": unscored, "macro_recall": None,
                "micro_recall": None, "hit_rate": None, "mrr": None}
    return {"scored": count, "unscored": unscored,
            "macro_recall": sum(recalls) / count,
            "micro_recall": found_total / relevant_total,
            "hit_rate": sum(hits) / count,
            "mrr": sum(reciprocal) / count}

queries = [(["a", "a", "x", "b"], {"a", "b"}),
           (["z", "c", "d"], {"c"}),
           (["x", "z"], set())]
report = retrieval_report(queries, 2)
assert isclose(report["macro_recall"], 0.75)
assert isclose(report["micro_recall"], 2 / 3)
assert report["hit_rate"] == 1.0 and report["mrr"] == 0.75
assert retrieval_report([([], {"a"})], 2)["mrr"] == 0.0
assert retrieval_report([([], set())], 2)["macro_recall"] is None
print(f"scored: {report['scored']}; unscored: {report['unscored']}")
for metric in ["macro_recall", "micro_recall", "hit_rate", "mrr"]:
    print(f"{metric}: {report[metric]:.6f}")

python image-text-search-multi-query-retrieval-report.py

scored: 2; unscored: 1
macro_recall: 0.750000
micro_recall: 0.666667
hit_rate: 1.000000
mrr: 0.750000

Follow the execution

1. Remove duplicate IDs before cutting off the top k. A repeated image must not occupy two result slots in this protocol.

2. For the first query, top two unique results are a and x. Only one of its two relevant images is found.

3. The second query finds its only relevant image at rank two. Its recall is one, but reciprocal rank is one half.

4. The query with no labeled relevant image is counted as unscored. It contributes to the coverage report, not to a division by zero.

Read the code step by step

For query q, let R_q be its relevant set and T_q the top k unique results. Fractional recall is |T_q intersect R_q|/|R_q|. Macro recall averages these fractions equally across Q scored queries. Micro recall divides total retrieved relevant items by total relevant items. Here macro=(1/2+1)/2=0.75, while micro=(1+1)/(2+1)=2/3.

Hit@k averages the indicator that at least one relevant result is retrieved. Here it is 1. MRR averages 1/r_q, where r_q is the first relevant rank in the full supplied, deduplicated list; a missing match contributes zero. Here MRR=(1+1/2)/2=0.75. Some image-text benchmarks call a hit-within-k measure Recall@k. With one positive per query it equals fractional recall; with multiple positives it need not. Always publish the definition.

Two of three queries are scored. The third may expose missing annotations or an answer absent from the collection. Excluding it does not measure whether a system can correctly abstain; that needs a separate protocol with known unanswerable queries. Similarity scores rank candidates, but do not prove relevance. Compare systems using the same corpus, deduplication, annotations and cutoff. Cost is linear in the supplied ranked lists plus relevance sets with average constant-time hash operations.

Run a complete example · 01

Compare two possible responses

Supplied conditional probabilities isolate sequence scoring; no text or image model is loaded.

import math
correct = [0.4, 0.5, 0.9]
incorrect = [0.5, 0.7, 0.9]
probability = math.prod(correct)
mean_loss = -sum(math.log(p) for p in correct) / len(correct)
print(f"correct probability={probability:.3f}")
print(f"incorrect probability={math.prod(incorrect):.3f}")
print(f"mean loss={mean_loss:.4f}")

python conditional-generation-response-probability.py

correct probability=0.180
incorrect probability=0.315
mean loss=0.5716

Read the code step by step

math.prod multiplies all entries. The generator takes the natural log of each checked-token probability; sum adds them and len supplies token count three. The negative average is 0.5716. The incorrect response scores 0.315 versus 0.180 for the correct one. These hand-set values demonstrate why model likelihood and visual truth need separate checks.

Run a complete example · 01

Keep a word beside its row

Each tuple stores text and a box in pixel coordinates.

words = [("Friday", (10, 40, 80, 60)), ("Room 3", (100, 40, 170, 60)), ("Room 8", (100, 80, 170, 100))]
target_y = 50
selected = []
for text, box in words:
    centre_y = (box[1] + box[3]) / 2
    if abs(centre_y - target_y) <= 5:
        selected.append((box[0], text))
selected.sort()
print("row:", [text for left, text in selected])
print("normalized x:", 1000 * 100 / 500)

python document-ocr-qa-document-row.py

row: ['Friday', 'Room 3']
normalized x: 200.0

Read the code step by step

The loop unpacks text and box. Indices 1 and 3 select top and bottom, whose average is row centre. abs and <= test vertical distance. append stores the left edge with each accepted word. sort orders by left edge; the final comprehension prints only text. The row is Friday, Room 3. These are supplied boxes, not OCR predictions.

Run a complete example · 01

Count a changed word

Start with a manually checked one-substitution example.

reference = "bring the new box".split()
substitutions, deletions, insertions = 1, 0, 0
wer = (substitutions + deletions + insertions) / len(reference)
print("reference words:", len(reference))
print(f"WER={wer:.2f}")
short_reference_count = 2
print(f"three insertions WER={3 / short_reference_count:.2f}")

python speech-tasks-word-error-rate.py

reference words: 4
WER=0.25
three insertions WER=1.50

Read the code step by step

split makes the four-word reference list; len counts its words. The next line assigns already verified edit counts. Add all edits before dividing by reference length. The last example gives 1.50 because insertions are not limited by reference length. These arithmetic examples do not implement speech recognition or a general edit-distance algorithm.

Run a complete example · 01

Find a delayed clap

Binary event positions replace real audio and video features.

visual = [0, 1, 0, 0, 0]
audio = [0, 0, 1, 0, 0]
scores = {}
for shift in [-1, 0, 1]:
    total = 0
    for t in range(len(visual)):
        j = t + shift
        if 0 <= j < len(audio):
            total += visual[t] * audio[j]
    scores[shift] = total
best_shift = max(scores, key=scores.get)
print("shift scores:", scores)
print(f"audio delay seconds={best_shift * 0.1:.1f}")

python audio-video-sync-sync-offset.py

shift scores: {-1: 0, 0: 0, 1: 1}
audio delay seconds=0.1

Read the code step by step

The outer loop tests three shifts; the inner loop visits video positions. j is the audio position to compare. The chained comparison checks its bounds before indexing. Multiplication counts a coincident pair of ones, and += accumulates it. max selects the shift with the highest score, giving +1 or 0.1 seconds of delay. This does not run SyncNet or alter media files.

Run a complete example · 02

Align events with timestamps, offsets and a tolerance

Mina reviews a video with an audio event log. The video has uneven frame intervals. The audio clock reads 20 milliseconds ahead of the video clock. Multiplying time by a guessed frame rate gives wrong matches. She converts the audio timestamps to the video clock, finds the nearest observed frame, and leaves distant events unmatched.

from bisect import bisect_left
from math import isclose

def nearest_frame(timestamps, event_time, offset, tolerance):
    if tolerance < 0 or not timestamps:
        raise ValueError("nonempty timestamps and nonnegative tolerance required")
    if any(a >= b for a, b in zip(timestamps, timestamps[1:])):
        raise ValueError("frame timestamps must be strictly increasing")
    target = event_time - offset  # audio_time = video_time + offset
    insertion = bisect_left(timestamps, target)
    candidates = [i for i in (insertion - 1, insertion) if 0 <= i < len(timestamps)]
    best = min(candidates, key=lambda i: (abs(timestamps[i] - target), i))
    error = abs(timestamps[best] - target)
    return (best, error) if error <= tolerance else None

video = [0.0, 0.041, 0.083, 0.126, 0.168]
events = [0.061, 0.104, 0.300]
matches = [nearest_frame(video, event, 0.020, 0.010) for event in events]
assert matches[0][0] == 1 and isclose(matches[0][1], 0.0, abs_tol=1e-12)
assert matches[1][0] == 2 and isclose(matches[1][1], 0.001)
assert matches[2] is None
assert nearest_frame([0.0, 2.0], 1.0, 0.0, 1.0)[0] == 0
for event, match in zip(events, matches):
    if match is None:
        print(f"audio {event:.3f}: unmatched")
    else:
        index, error = match
        print(f"audio {event:.3f}: frame {index}, error {error:.3f} s")

python audio-video-sync-timestamp-nearest-alignment.py

audio 0.061: frame 1, error 0.000 s
audio 0.104: frame 2, error 0.001 s
audio 0.300: unmatched

Follow the execution

1. State the clock relation: audio time equals video time plus 0.020 seconds. Therefore subtract the offset.

2. Binary search finds the insertion point. Only the frames immediately before and after it can be nearest.

3. Choose the smaller distance, breaking exact ties toward the earlier frame.

4. Reject the last event: 0.300-0.020=0.280, which is 0.112 seconds from the final frame.

Read the code step by step

Let a be an audio timestamp, delta a known clock offset and v_i an observed video timestamp. Search for i_star=argmin_i |v_i-(a-delta)|. Accept only if that minimum is at most tolerance tau. Here tau=0.010 seconds. The first two adjusted events are 0.041 and 0.084; their frame errors are 0 and 0.001 seconds.

The sign of delta comes from the declared clock equation, not a guess. Use one time unit everywhere. Frame index alone does not encode presentation time when frame intervals vary. This program assumes finite numeric timestamps and a known constant offset. Clock drift may require a calibrated scale as well, such as audio_time=scale*video_time+offset.

Each call validates ordering in O(F), then searches in O(log F), so the full function is O(F). A production index can validate once and reuse the sorted timestamps for O(log F) queries. This is nearest-point matching, not interpolation, one-to-one assignment, or learned audio-visual synchronization. Some applications should match time intervals instead. A nearby frame is not proof that the sound and visible event have the same cause.

Run a complete example · 01

Sample a video and compare intervals

Use timestamps only, without any video files.

timestamps = list(range(0, 12, 2))
short_event = (5.2, 5.7)
seen = any(short_event[0] <= t <= short_event[1] for t in timestamps)
predicted = (5, 9)
reference = (4, 8)
intersection = max(0, min(predicted[1], reference[1]) - max(predicted[0], reference[0]))
union = (predicted[1] - predicted[0]) + (reference[1] - reference[0]) - intersection
print("sample times:", timestamps)
print("short event sampled:", seen)
print(f"temporal IoU={intersection / union:.2f}")

python video-temporal-grounding-temporal-overlap.py

sample times: [0, 2, 4, 6, 8, 10]
short event sampled: False
temporal IoU=0.60

Read the code step by step

range starts at zero, stops before 12, and steps by two. any is True if at least one timestamp falls inside the short event; here none does. min of the ends and max of the starts find the shared interval. max with zero prevents a negative duration. Adding both lengths and subtracting overlap gives union five. This evaluates supplied intervals, not a video model.

Run a complete example · 01

Adjust a connector, keep the feature

Try two weights without any calculus or model library.

fixed_feature = 2.0
target = 1.0
for weight in [1.0, 0.75]:
    output = weight * fixed_feature
    error = output - target
    loss = error ** 2 / 2
    print(f"weight={weight:.2f}; feature={fixed_feature:.1f}; loss={loss:.3f}")

python visual-instruction-tuning-frozen-feature.py

weight=1.00; feature=2.0; loss=0.500
weight=0.75; feature=2.0; loss=0.125

Read the code step by step

The feature and target are assigned once. The for loop tests two connector weights. Multiplication produces the output; subtraction produces the signed error; **2 squares it before dividing by two. Loss falls from 0.500 to 0.125 while the feature remains fixed. This is a trial comparison, not gradient training or a full visual instruction model.

Run a complete example · 01

Return an answer with its evidence ID

Small dictionaries replace a real document store and a trained generator.

query = set("model opening upward".split())
records = [
    {"id": "sheet-A", "terms": "model opening upward arrow", "answer": "arrow-marked opening", "page": 2, "region": "diagram-A"},
    {"id": "sheet-B", "terms": "model paint dry", "answer": None, "page": 3, "region": "paragraph-B"},
]
def overlap(record):
    return len(query & set(record["terms"].split()))
best = max(records, key=overlap)
print("retrieval score:", overlap(best))
if best["answer"] is not None:
    print("answer:", best["answer"])
    print("evidence:", best["id"], "page", best["page"], best["region"])
else:
    print("No checked answer in the retrieved evidence")

python multimodal-rag-retrieve-cite.py

retrieval score: 3
answer: arrow-marked opening
evidence: sheet-A page 2 diagram-A

Read the code step by step

split separates words and set keeps distinct terms. & finds shared terms; len turns them into a count. max calls overlap for each record and selects the highest count. The if requires a checked answer field before printing. The citation preserves the record, page and region. This toy routes supplied evidence and does not implement MuRAG or prove that word overlap establishes relevance.

Run a complete example · 01

Count unsupported mentions and coverage

Use an invented, already checked audit.

mentions, unsupported = 40, 6
captions, captions_with_error = 10, 4
correct_distinct, target_objects = 34, 50
print(f"unsupported mention rate={unsupported / mentions:.2f}")
print(f"captions with errors={captions_with_error / captions:.2f}")
print(f"target coverage={correct_distinct / target_objects:.2f}")
print(f"short revision coverage={19 / target_objects:.2f}")

python evaluation-hallucination-claim-audit.py

unsupported mention rate=0.15
captions with errors=0.40
target coverage=0.68
short revision coverage=0.38

Read the code step by step

The first three lines name different populations: mentions, captions and distinct target objects. Each division uses its matching denominator. Rates are 0.15, 0.40 and 0.68. The last line shows that mentioning only 19 correct targets covers 0.38. These are checked count ratios; the code cannot judge image evidence or implement the full annotation protocol by itself.

Run a complete example · 01

Compare a full pass and a shortlist

Use hypothetical cost units and a simple evidence budget.

candidate_count = 100
cheap_cost, expensive_cost = 1, 10
shortlist_size = 5
full_cost = candidate_count * expensive_cost
cascade_cost = candidate_count * cheap_cost + shortlist_size * expensive_cost
print("costs:", full_cost, cascade_cost)
print(f"saved fraction={1 - cascade_cost / full_cost:.2f}")
evidence_tokens = [80, 60, 90]
budget = 150
used = 0
kept = []
for index, count in enumerate(evidence_tokens):
    if used + count <= budget:
        kept.append(index)
        used += count
print("kept evidence indices:", kept, "tokens:", used)

python efficient-evidence-systems-cascade-budget.py

costs: 1000 150
saved fraction=0.85
kept evidence indices: [0, 1] tokens: 140

Read the code step by step

The first arithmetic compares assumed per-item costs. enumerate gives each evidence item an index and token count. The if checks whether adding it fits the remaining budget. append records accepted indices and += updates used tokens. Items 0 and 1 use 140 tokens, so item 2 is omitted. This greedy budget rule does not judge relevance or run a model; omitted evidence may be essential.