There's actually a wide variety of noise in the data, but the one that really matters is altitude (the barometer can measure the altitude difference between your feet and your head)! Other sources of error are the biases in the sensors themselves (place two phones side by side, get different readings), pressure-controlled buildings, sensor drift, etc.

We have a number of mechanisms to filter out most of the noise, but it's not easy.