Aiming at the problem that geomagnetic sensor is susceptible to interference

a pre-processing total least square iteration method is proposed for calibration compensation. Error model for geomagnetic sensor is analyzed and the correction model is proposed

characteristics of model are analyzed and converted into estimation problem of nine parameters. The geomagnetic data is filtering processed by Hilbert Huang transform（ HHT） to improve signal-to-noise ratio

and the nine parameters are calculated by using the combination of Newton iteration method and the least squares estimation method. The experimental results show that this method does not need additional equipment and devices

it can continuously update calibration parameters

and its compensation precision is prior to the two-step estimation method

it calibrates and compensates geomagnetic sensor error very well.