To meet the application of high toughness epoxy matrix, the core-shell nanoparticles with flexible polybutyl acrylate (PBA) as core and rigid poly(methyl methacrylate) (PMMA) as shell were used to toughen epoxy resin in the cooperation with epoxy-based POSS (G-POSS). The effects of toughening agents on the curing process and properties of the resin and the interaction between the epoxy matrix, core-shell nanoparticles and G-POSS were studied by DSC, viscosity analysis and mechanical properties testing, the microstructure of the cross section was observed by scanning electron microscopy (SEM). The results show that the addition of two parts of G-POSS and six parts of core-shell particles as toughening agent has little effect on the curing reaction of epoxy, and the glass transition temperature of cured resin increases slightly, the introduction of POSS makes up for the significant reduction in impact strength and bending strength caused by the addition of core- shell particles to epoxy, at the same time, the impact strength and fracture toughness of the modified resin are greatly increased by 367% and 362% compared with those of pure epoxy resin respectively. SEM analysis shows that the toughening mechanism of POSS is mainly the stress concentration induced by nanoparticles, forming plastic cavity and deformation. POSS also provides cubic structure and Si—O—Si flexible chain, dispersing and absorbing a large amount of energy brought by external forces, which retards the propagation of crack and improves the toughness of material.