Did rotation help?

The same with-and-without-rotation comparison, run twice against one baseline model, gives opposite answers.

The question

A small neural network (a CNN) learns to sort galaxy images into ten shape classes. Training it on randomly rotated copies of its training images lifted its test accuracy from 44.6% to 55.2%. Does that mean rotation helped?

The answer

Not from these experiments. The comparison was run twice against the same baseline model, which scores 44.61%. With rotation, one run reached 55.25% (+10.6 points) and the other 32.45% (−12.2 points): two runs of the same recipe, 22.8 points apart. 8 of the 10 galaxy classes change direction between the runs. The difference between the two runs is larger than the effect being measured.

What rotation did to the model's own feature space

trained without rotation
trained with rotation

Does that turn into accuracy?

test accuracy without rotation

with rotation

What the two runs agree on, and what they do not

Where these numbers come from