deep-learning

Why cross entropy, of all things: from BCE to KL divergence

Why classification uses cross entropy instead of MSE, traced as a single line from information content and entropy through BCE and MLE to KL divergence.

cross-entropy  kl-divergence  information-theory  loss-function  deep-learning

deep-learning

2026-08-30   ·   14 min read

ROUGE

Recall-Oriented Understudy for Gisting Evaluation

rouge  nlp  evaluation

deep-learning

2026-04-01   ·   8 min read

How to teach your embedding model new words

A step-by-step guide on adding new vocabulary to a Hugging Face sentence-transformer model, including fine-tuning and using it in LangChain.

embeddings  fine-tuning  nlp

deep-learning

2025-10-25   ·   8 min read

Running a local LLM with llama.cpp: build, download, serve, benchmark

A minimal llama.cpp workflow for running GGUF models on your own machine — from the cmake build to an OpenAI-compatible server and llama-bench.

llama-cpp  llm  inference  gguf  local-llm

deep-learning

2025-03-30   ·   4 min read

Pydantic with llm.with_structured_output — pitfalls and safeguards

Combining Pydantic models with LangChain's with_structured_output cleanly — avoiding prompt collisions, validation errors, and runaway evaluation loops.

langchain  pydantic  structured-output  llm

deep-learning

2025-02-18   ·   4 min read

From Seq2Seq to Transformer: from accumulated context to direct reference · Part 4

Self-attention and the Transformer - cutting the sequential chain and referencing directly

Where an RNN passed context along by accumulating it through previous states, the Transformer looks up the tokens it needs directly. How self-attention removes the sequential structure, and how it connects back to attention Seq2Seq.

transformer  self-attention  multi-head-attention  positional-encoding  masked-attention

deep-learning

2023-07-14   ·   4 min read

From Seq2Seq to Transformer: from accumulated context to direct reference · Part 3

The decoder's remaining problem - output context still accumulates

Attention solved the encoder-side information bottleneck. But the decoder itself is still a sequential structure that depends on its previous state. Here is why that becomes the final motivation for the Transformer.

attention  seq2seq  bahdanau-attention  query-key-value  nlp

deep-learning

2023-06-28   ·   3 min read

From Seq2Seq to Transformer: from accumulated context to direct reference · Part 2

Attention Seq2Seq - don't just look at the last state, look directly

How do you fix the vanilla Seq2Seq bottleneck? By letting the decoder reference every encoder hidden state directly, "as needed".

attention  seq2seq  bahdanau-attention  query-key-value  nlp

deep-learning

2023-06-02   ·   4 min read

From Seq2Seq to Transformer: from accumulated context to direct reference · Part 1

Vanilla Seq2Seq - the architecture that leaned on one final state

To understand the basic Seq2Seq architecture you first have to accept one fact: the input is compressed into a single vector. How everything ends up in the last hidden state, and the fundamental limits that follow.

seq2seq  rnn  encoder-decoder  attention  nlp

deep-learning

2023-05-25   ·   3 min read

Softmax and Cross-Entropy

Deriving the softmax Jacobian and the cross-entropy gradient, and why their composition collapses into prediction minus target.

softmax  cross-entropy  backpropagation  deep-learning

deep-learning

2022-01-20   ·   5 min read