The down fibers with excellent flame retardancy were fabricated by layer-by-layer self-assembly (LBL) modification, using phytic acid (PA) as poly-anionic and layer chitosan (CS) as poly-cationic layer. The structure and properties of the down fibers before and after modification were characterized by cold field emission scanning electron microscopy, thermogravimetric analyzer, micro combustion calorimeter, and microscopic confocal Raman spectrometer. The results show that the thermal stability and flame retardancy of the down fibers with 10 layers of LBL modification are significantly improved. The peak heat release rate (pHRR) and total heat release (THR) of the modified down fibers are 106.6 W/g and 12.2 kJ/g, respectively, which are reduced by 31.2% and 23.3% compared to 154.9 W/g and 15.9 kJ/g of the pristine down fibers. Meanwhile, the modified down fibers exhibit higher char residue and greater graphitization degree at high temperature. In addition, the overall structure of the modified down fibers remains almost undamaged and still retains excellent fluffiness, which will not affect the practical use of down fibers.