A series of SiO₂ sols were synthesized with tetraethylorthosilicate (TEOS) and γ-methacryloxypropyltrimethoxysilane (MPS) as precursors via the acid-catalyzed sol-gel method. The stable SiO₂/TPGDA mixing dispersions were obtained by vacuum distillation process. Then the dispersions, reactive diluent and oligomers were mixed together to prepare UV-curable organic-inorganic hybrid coatings. The storage stability of hybrid coatings was discussed in detail. The chemical structure of SiO₂ sol and hybrid coatings was characterized by FT-IR. The effects of SiO₂ contents on thermal stability, flame retardancy and mechanical properties of hybrid coatings were investigated by the thermogravimetric analysis(TGA), microcalorimeter (MCC) and cone calorimeter test, etc. The results show that thermal stability and mechanical properties of hybrid coatings are improved significantly. With increased the content of SiO₂ sol from 0 to 50%, a increase of pencil hardness from 3H to 6H and a decrease of abrasion resistance from 0.0852g/100r to 0.0606g/100r are observed. The MCC test indicates that the hybrid coatings show excellent flame retardancy. The peak heat release rate (PHRR) and total heat release(THR) of the hybrid coatings with 40% SiO₂ contents decrease 41.98% and 41.12% compared with organic coating.