Satellite image vector quantization

The dataset this page uses: 27,000 small colour satellite photographs of European land, sorted by the EuroSAT authors (Helber et al.) into ten folders named for what they show. The Dataset page documents the release.

The European Earth-observation satellites; the EuroSAT authors cut these tiles from Sentinel-2A images (see the Dataset page).

One photograph: a fixed-size square of ground, in three colour channels. One tile becomes one row of the design matrix.

The folder a tile was filed in — forest, river, housing and the rest. Here it is an evaluation label only: it is never given to the clustering or to the codebook, and is used afterwards to ask what the description happened to keep.

A grouping of the ten folders into four families (cultivated, natural, built and water), used to colour the maps: the hue shows the family and a shade of it the folder, and every folder is also named in the key, the hover card and the table. The grouping is this page's own reading of the folder names, not a category in the dataset.

The tiles this page uses were drawn folder by folder rather than at random from the whole archive, so each folder contributes the same number of tiles. The archive itself is unbalanced; the sample deliberately is not.

The fixed number that starts the random draw. The same seed reproduces the identical tiles in the identical order, which is what makes the numbers on this page repeatable.

The picture format the archive stores. It has already discarded some detail, so quality figures here are measured against the decoded JPEG.

The table of pooled pixel values: one row per tile, one column per pooled colour value, 768 in all. The Design matrix section and the PCA curve use it; the clustering, the patch codebook and the classifier read the same tiles through other descriptions.

A row X[i,:] is one tile; a column X[:,j] is one pooled position-and-colour value across every tile; a cell x_ij is one number.

Replacing each square block of pixels by the average of that block. It is how a tile is shrunk before it becomes a row — cheap, reversible in shape but not in content, and the reason fine texture is gone from everything downstream.

Reading a two-dimensional picture out as one long list of numbers, row after row. It is a change of bookkeeping, not of information: nothing is lost, but a method reading the flattened list no longer knows which numbers were neighbours.

The grid of thumbnails in the Design matrix section: each row of the matrix shown as the picture of its tile.

The number of rows drawn when the matrix has more rows than the chart can show. It is printed beside the chart, together with how the drawn rows were chosen.

A rotation of the description into new axes ordered by how much of the variation across tiles each one carries, so the leading few can be kept and the rest dropped. On this page it is applied to the pooled design matrix (the variance curve in the K-means clustering section); the clustering uses the CNN descriptor instead. PCA of the raw pixels is also the first step of the original-pixel map and of one classification baseline. PCA is linear: it cannot capture non-linear structure.

The numbers a method sees when it looks at a tile. Two are used for the maps and the clustering: the pooled one, a 16×16 RGB thumbnail flattened to 768 numbers (the design matrix and the PCA curve), and the learned one below (the clustering and the 2D maps in the Design matrix and K-means clustering sections). The patch codebook and the original-pixel map work on the raw pixels. Each panel names its input on the line under its heading.

A description produced by running the tile through a network already trained on a different, labelled photo collection, and keeping the numbers from its next-to-last layer. Here that is ResNet-18 trained on ImageNet, 512 numbers per tile, unit-normalised. It is why the clustering separates land cover much better than the pooled thumbnail: no EuroSAT label is used, but the features were shaped by ImageNet's labels.

The share of the total variation across tiles that a component carries. Added up over the leading components, it answers “how much of the description survives if I keep only this many axes”.

Any recipe that gives each row of the design matrix a position in a space with far fewer axes than the row has columns: here, two, so the whole matrix fits on a screen. What is dropped is dropped: two rows can land on the same spot because they are alike, or because the ways they differ lie in directions the recipe threw away.

A projection fitted to keep each point's near neighbours near, and nothing else. It is very good at making groups visible, but it is not a map: on a t-SNE picture the distance between two far-apart clumps, the area a clump covers and the direction of either axis are outputs of the optimiser, not measurements.

A projection with the same purpose as t-SNE and different machinery: it builds a graph of near neighbours and lays that graph out. As with t-SNE, near is meaningful, far is not, and the axes have no units. It tends to keep clumps more compact than t-SNE does.

t-SNE's main setting: roughly how many neighbours each point is fitted to stay close to. A small value makes many small clumps and a large one makes few broad ones, out of the very same data. It is unrelated to the usage perplexity below, which counts codewords.

UMAP's two main settings. n_neighbors is how much of each point's neighbourhood the layout is fitted to, so it trades local detail against overall shape; min_dist is how tightly points are allowed to be packed, so it changes how dense a clump looks without changing which points are in it. Both are settings, not properties of the data.

An algorithm that splits points into k groups by repeatedly moving k centres to the average of the points nearest them. It always returns k groups, whether or not the data has any; the number k is a setting, not a discovery.

The same algorithm fitted on small random batches instead of all the points at once. Used here because the patches are far more numerous than the tiles.

Replacing each piece of data by the nearest entry of a fixed, short list, and storing the entry's index instead of the data. It is the whole idea this page is about: the picture is no longer stored, only the recipe for reassembling it.

The fixed list itself — the K stored patches every patch of every tile must be replaced by. Learned here by running k-means over patches taken from the sampled tiles.

One entry of the codebook. Its index is a name, not a quantity: codeword 7 is not between 6 and 8, which is why the codebook is drawn as a strip of swatches and never as a scale.

The ten groups k-means finds among the tiles, which the class map in the K-means clustering section colours by. It uses the same algorithm family as the codebook above, but everything else differs. The codebook is fitted on 4×4 patches and exists to rebuild a tile cheaply; a cluster is fitted on the whole tile's 512-number descriptor and is named afterwards, by scoring it against folder names it was never shown. The fit stays at ten clusters, one for each land-cover folder, so that it can be scored against them; its adjusted Rand index and matched-class F1 are printed beside it.

The single codeword a tile is made of more than any other. Quantize a tile at codebook size K and each of its 256 4×4 patches is replaced by its nearest codeword; whichever of them comes up most often is the tile's dominant codeword, and the share of the 256 it accounts for is printed beside it wherever the page names it. It is what the original-pixel map colours by and what the codeword bars count, so changing the codebook size re-quantizes every tile. Read the share: at a small K one codeword can cover most of a tile, and at K = 256 the dominant one is often a modest plurality. A codeword that dominates no tile is not unused: it can appear in many tiles in small amounts.

The small square a tile is cut into before quantization. Patch size and K together decide both the bit cost and how much structure a single codeword can carry.

The failure where almost every patch chooses the same few codewords and the rest are never used. The training loss does not report it; counting the alive codewords does.

A model that learns the description and the codebook together: an encoder compresses the picture, the result is snapped to a learned codebook, and a decoder rebuilds it. The contributed notebook includes one, switched off by default; here it was run at the notebook's own settings. The codebook collapsed and the rebuild did not reach the quality of returning one fixed average picture; the figures are in the VQ-VAE panel at the end of the Vector quantization section.

The two halves of the learned arm. The encoder maps a tile down to a small grid of vectors; the decoder maps a grid of codebook entries back up to a tile. They are trained together, against the difference between the tile that went in and the tile that came out — so neither is meaningful on its own, and neither is ever shown a folder name.

One pass of the training loop over every tile it was given. It is a setting, not a result: a model that has not been trained for long enough and a model that cannot learn this data look identical after too few of them, which is why the epoch count is printed beside the VQ-VAE's score.

The trick that lets the snapping step be trained at all. Choosing the nearest codebook entry has no usable gradient, so the gradient is passed backwards as if the snap had not happened. It is the plain form the contributed notebook implements, without either of the two maintenance measures below.

Moving the codebook entries towards a running average of the vectors assigned to them, instead of learning them by gradient alone. It is one of the two standard measures against collapse; the contributed notebook's code does not implement it.

Detecting codebook entries that nothing is choosing and re-seeding them onto vectors that are actually in the data. The other standard measure against collapse, also absent from the contributed notebook's code. Absent both, a codebook can shrink to a single entry and the decoder can only emit one picture.

The average of the squared differences between the original and the rebuilt picture, pixel by pixel. It treats every pixel alike and knows nothing about what a person would notice.

Peak signal-to-noise ratio: the reconstruction quality figure derived from the mean squared error, on a logarithmic scale in decibels. Higher is closer to the original. It is reported together with the bits per tile it costs.

What one tile now costs to store, counted as the number of patch indices times the bits needed to name one codeword. It is the “rate” half of the trade.

The curve of quality against storage cost across codebook sizes.

How many codebook entries were chosen at least once. Well below K means the bit budget is paying for a vocabulary that is not being used.

The effective number of codewords actually in play, computed from how unevenly the usage is spread. It falls towards one as a codebook collapses.

How much a grouping the algorithm found agrees with the folder names, corrected so that chance agreement scores about zero. It compares two groupings.

Each cluster is paired with the folder it overlaps most, one to one, and the per-pair scores are averaged. It answers “if I had to name each cluster, how well would the names fit?”

The table of how many tiles of each folder landed in each cluster. The pairing above is computed from it, and the class-map table uses it to show which cluster most of each folder went to.

The score of the simplest possible predictor: return one fixed picture, the average of every tile in the sample, whatever the input is. It is measured with the same formula on the same tiles as the models, and a reconstruction has to beat it before its decibel figure means anything. A model that does not has learned nothing tile-specific.

The form the K-means clustering section uses to compare folders and clusters in one picture. Every tile is drawn once, at the position the projection gave it, and two buttons change only what the dots are coloured by: the folder the tile was filed in, or the cluster k-means put it in. Because no dot moves, a dot that changes colour is a disagreement, and a class that keeps a region of its own under both colourings is one the clustering recovers. Clicking a key entry shows only that series and greys every other dot.

How the maps draw ten categories: the hue shows the land group (water blue, natural green, cultivated gold, built brick) and a shade of that hue separates the two or three folders inside a group. A shade is a weaker cue than a hue, so every series is also named in the key, in the hover card and in the table, and clicking a key entry shows only that one.

Nothing on this page fits a model in your browser. Changing a setting switches to a result computed in advance.

A codeword nobody chose is drawn as a short grey mark on the baseline of the usage chart, sorted last, so that it reads as “none” rather than as a very small count.

In the Source evidence table, whether the project's records and the archive itself agree. Each flag carries an icon and a word, not just a colour.