JPEG
The usual account is that JPEG throws away pixels. It does not throw away a single one. It rewrites a block of 64 pixels as 64 numbers saying how much of each of 64 fixed patterns is present, rounds most of those numbers to zero, and rebuilds the block from what is left. Every pixel goes in and every pixel comes out; almost none of them comes out the same, and the picture still looks right. That substitution is the whole idea and it is visible in one block.
New to frequencies? Start here
A block of pixels can be written a second way: not as brightness at each position, but as a sum of patterns, from flat, through broad gradients, to fine checkerboards. Same block, same information, different description. Nothing is lost in the translation and it can be reversed exactly.
That rewriting is the whole reason the format works. In the second description most of the value sits in the coarse patterns, and the fine ones are usually near zero and are the ones the eye is worst at. So you can round the fine ones hard and the coarse ones gently, which is throwing away numbers rather than pixels.
Only patterns can be spent
A file gets smaller for one of three reasons. Something in it repeats, so the second copy can be a note saying where the first one was. Or some symbols are commoner than others, so the common ones can be given shorter codes and the rare ones longer. Those two keep every bit, and are called lossless. The third is to throw information away on purpose, which works only where something can be relied on not to miss it: an eye, an ear, or an application that never needed it.
The first two are spending a pattern, and a pattern can only be spent once. Compress an already compressed file and you usually get nothing, or a little worse, because the first pass took what was there. Not always: a different method can find structure the first one was not looking for, which is why some formats chain two. What is certain is the counting argument underneath, and it is about all inputs rather than about yours: no lossless method can shorten every possible input, because shortening some must lengthen others. The machines here are choosing which inputs to be good at.
The machine for this idea on its own is Huffman, if you would rather press it than read about it.
One block of eight by eight, and what survives being compressed
1 The image, in blocks of eight by eight
A block of eight by eight samples, which is what JPEG works on. Pick one with more or less fine detail in it.
The 64 original pixels, 0 to 255
2 The same block written as frequencies instead of pixels
The same 64 numbers, rewritten as how much of each of 64 fixed patterns is present. Top left is the block’s flat level, and JPEG subtracts 128 from every sample before the transform, so the mean of 110 here becomes −18 and the coefficient is eight times that: −144. Down and right is finer detail.
The 64 coefficients, before anything is thrown away
- the first coefficient
Nothing has been lost yet. This step is exactly reversible, and the page checks that when it loads.
3 Quantisation, which is where the throwing away happens
Each coefficient is divided by the number below it and rounded. That rounding is the only lossy step modelled on this page.
The quantiser, scaled to this quality
What is left after dividing and rounding
- rounded to nothing
- what is stored
4 Rebuild it, and look at what did not come back
The block rebuilt from what survived
- pixels that came back identical
- the worst any pixel is out by
- the RMS error
What this page checked when it loaded.
| claim | held | measured |
|---|---|---|
| the transform is reversible to within rounding | yes | worst difference 1.28e-13 |
| a flat block has exactly one non-zero coefficient | yes | DC 256.00, every other under 1e-9 |
| at quality 50 most coefficients are thrown away | yes | 59 of 64 became zero |
| and the rebuilt block is close but almost nowhere identical | yes | 24 of 64 pixels came back unchanged, worst error 4 |
| lower quality throws away more and costs more error | yes | quality 10: 60 zeros, rms 16.9; quality 90: 59 zeros, rms 0.4 |
What is real here, and what is not
One block, in grey, with no file around it
JPEG itself covers greyscale as well as colour and leaves the colour space to the application. A typical photographic encoder splits the image into brightness and colour and then keeps one colour sample for each two-by-two group of brightness samples before the transform starts, because an eye is worse at colour than at brightness. That is 4:2:0, and it is a common choice rather than part of the format: 4:4:4 keeps every colour sample and the standard’s own reference software supports it. Then this runs on every 8 by 8 block, and the result is Huffman-coded with the coefficients read in a zigzag so the zeros end up together. None of that is here. What is here is the one step where information is actually lost, on one block, and the compression that follows it is what turns those zeros into a smaller file.
The transform is computed from its definition, which no encoder does
Each of the 64 coefficients is worked out by summing over all 64 samples, which is 4,096 cosines per block and about as slow as it can be made. Real encoders use a fast algorithm that gets the same answer in a fraction of the work. Slow and exactly right was the correct trade here, because it means the sum written on this page is the sum in the code rather than an optimisation of it.
The quantisation table is copied, so it is checked against the standard
The table is the example luminance table from Annex K of ITU-T T.81. It is not normative, since the standard offers it as one that has been found to work, and it is what almost everything uses. It is a memorised table, which is exactly the kind of thing that rots quietly, so the claims test compares all 64 entries against libjpeg-turbo's independent copy rather than trusting this file. The standard's own PDF is not machine-readable from the archive, and a second implementation is the better witness anyway: it would catch two rows exchanged, which a structural check would not.
Quantisation throws away small numbers, not fine detail
It is tempting to say JPEG discards high frequencies, and the fine stripes here show why that is wrong: they are the finest pattern the block can hold and they come back nearly intact, because their coefficient is enormous. What the quantiser removes is anything small relative to its divisor. A faint texture at the same frequency disappears completely; a bold one at that frequency survives.
Turning the quality up can make one block very slightly worse
The quality setting guarantees one thing and it is not the one people assume: no divisor in the table ever gets larger as quality rises, and that holds for all 64 entries at every step. What it does NOT guarantee is that any particular block comes out closer. A finer divisor can round a coefficient the other way, and on the gradient here the error rises slightly three times on the way from 5 to 100 before ending at zero. Over a whole photograph this averages out; on one block it does not, and the sweep on this page is one block.
Quantisation is the only lossy step HERE, not in every JPEG
The panel above used to say the rounding was the only lossy step in the whole format. On this page it is, because this page is one block of brightness with no colour in it. In a file encoded 4:2:0 the colour was thrown away before the transform ever ran, and that is lossy too. An outside reader caught the overclaim. The distinction worth keeping is between the mechanism this page demonstrates and the encoding choices it does not.
Sources
- ITU-T Recommendation T.81, Digital compression and coding of continuous-tone still images, 1992. The standard: the transform, and the Annex K example tables.
- libjpeg-turbo, jcparam.c. An independent copy of the Annex K luminance table, which is what the claims test compares this page's against entry by entry.
- N. Ahmed, T. Natarajan and K. R. Rao, Discrete Cosine Transform, IEEE Transactions on Computers C-23, 1974. Where the transform itself comes from, eighteen years earlier.
- Logical Art, the studio this belongs to.