#informationtheory

If you've spent any time around machine learning the familiarizing yourself with the concept, you...

Originally published on https://marcindudek.dev/blog/77-questions-identify-every-human/ In short A...

Every time your computer performs a computation, it consumes energy. We all know this intuitively,...

How DNA wires a brain: a ~10B-bit genome must wire 100 trillion connections. The scheme that closes the gap in 300M bits, and why two simpler plans fail first.

Information Theory in AI — Entropy, KL Divergence & Why Cross-Entropy Is the Default...

The series closes by restating the codes-as-priors thesis across all twelve instances and connecting the wire-format side to the Stepanov type-algebra side.

Cryptographic theory assumes random oracles with infinite output. We have 256 bits. This paper explores how we bridge that gap, and what it means that we can.

Every prefix-free code is a hypothesis about the source. The codeword lengths determine an implicit probability distribution; the code is optimal when that prior matches the true source.

The problem of predicting what comes next, from compression to language models

Unary and Elias gamma are the two simplest universal codes. Unary encodes n in n bits; gamma in 2 log2(n)+1 bits. Each implies a different prior over the integers.

Rice and Golomb codes are parametric: a single parameter k (or m) tunes the code to a specific geometric distribution. Choosing k is choosing your prior precisely.

Solomonoff induction, MDL, speed priors, and neural networks are all special cases of one Bayesian framework with four knobs.