#informationtheory

Maksim Barziankou (MxBv) Navigational Cybernetics 2.5 · The Urgrund...

Maksim Barziankou (MxBv) Navigational Cybernetics 2.5 · The Urgrund...

If you've spent any time around machine learning the familiarizing yourself with the concept, you...

Originally published on https://marcindudek.dev/blog/77-questions-identify-every-human/ In short A...

Every time your computer performs a computation, it consumes energy. We all know this intuitively,...

How DNA wires a brain: a ~10B-bit genome must wire 100 trillion connections. The scheme that closes the gap in 300M bits, and why two simpler plans fail first.

Information Theory in AI — Entropy, KL Divergence & Why Cross-Entropy Is the Default...

The series closes by restating the codes-as-priors thesis across all twelve instances and connecting the wire-format side to the Stepanov type-algebra side.

Every prefix-free code is a hypothesis about the source. The codeword lengths determine an implicit probability distribution; the code is optimal when that prior matches the true source.

The problem of predicting what comes next, from compression to language models

Cryptographic theory assumes random oracles with infinite output. We have 256 bits. This paper explores how we bridge that gap, and what it means that we can.

Unary and Elias gamma are the two simplest universal codes. Unary encodes n in n bits; gamma in 2 log2(n)+1 bits. Each implies a different prior over the integers.