Xanthate silane coupling agents ethyl xanthate propyl triethoxysilane (EXS) and octyl xanthate propyl triethoxysilane (OXS) with different alkyl chain lengths were prepared by phase transfer catalysis. Two xanthate-based silane coupling agents were used to modify silica and reinforce styrene-butadiene rubber, the processing properties, vulcanization properties, mechanical properties, wear resistance, compression properties, dynamic mechanical properties of styrene-butadiene rubber/silica composites were studied. At the same time, the composites modified with bis-[γ-(triethoxysilyl)propyl]tetrasulfide (Si-69) and end-capped mercaptosilane coupling agent (NXT) were prepared as comparison to explore the feasibility of the new coupling agents to replace NXT and Si-69 in rubber industry. The results show that the OXS-modified composites with long alkyl chains have the lowest Mooney viscosity, high processing safety but a slightly slower vulcanization speed, and the composites modified with EXS have a low processing safety but the vulcanization speed is faster; the mechanical properties, hardness and wear resistance of the composite modified with the new coupling agents are higher than those of the NXT modified composites, and the rolling resistance is lower; the silica modified with EXS has better dispersion in rubber matrix, and the higher interaction between filler and rubber.