Chaos is an unpredictable phenomenon that is considered in the nonlinear dynamical systems. In this paper, the chaos control in the nonlinear dynamic gear transmission system is considered. At first, the complex dynamics characters of gear transmission system are studied. Then, the existence of chaos in this nonlinear dynamic system is studied by phase portrait. The design process of sliding-mode control is divided into two steps: The first step is designing a sliding surface so that the plant restricted to the sliding surface has a desired system response, and the second step is constructing a switched feedback gains necessary to drive the plant’s state trajectory to the sliding surface. The stability of the proposed sliding surface is studied by the first theorem in this paper. Chaos control and stabilization of the gear transmission system states is studied based on the Lyapunov stability theorem. Also the stability of the closed loop system is investigated. The effects of the unknown controller parameters, system uncertainty, external disturbance and nonlinearity in the control input are fully taken into account. Appropriate adaptation laws are obtained to undertake the unknown parameters of the controller.  Finally, simulation results demonstrate the feasibility and robustness of the proposed controller.