The nonlinear error of three-axis magnetometer is an important factor affecting its measurement precision

and the traditional error model for three-axis magnetometer only considering zero bias

axis nonorthogonal and sensitivity error

unable to realize the effective apartment and calibration of magneto measurement errors. By extending traditional error model

propose nonlinear error model for three-axis magnetometer. Adaptive particle swarm optimization（ APSO） algorithm is used for nonlinear error model parameters inversion. Experimental results show that the proposed method can effectively compensate the unlinear measuring error of three-axis magnetometer

and under the condition of larger scale of error parameters

APSO has better global search ability compared with the traditional calibration methods

which can significantly improve error parameter inversion precision and the convergence speed of algorithm.