Polysilazane (PSZ) and its polymer-derived amorphous silicon oxycarbide (SiOC) ceramic were coated on aluminum nitride (AlN) by using a dip-coating method to allow moisture-crosslinking of PSZ on AlN, followed by heat treatment at 700 °C in air to convert PSZ into SiOC on AlN. The results from FTIR, XPS and SEM indicated that the surface of AlN was successfully coated by PSZ and SiOC film. It was found that the introduction of PSZ and SiOC film help improve in the interfacial adhesion between the modified AlN (PSZ/AlN and SiOC/AlN) and silicone rubber lead to the increase in the thermal conductivity of the composites since the thermal boundary resistance at the filler–matrix interface was decreased. However, the introduction of SiOC as an intermediate layer between AlN and silicone rubber could help increase the thermal energy transport at the filler–matrix interface rather than using PSZ. This result was due to the decrease in the surface roughness and thickness of SiOC film after heat treatment at 700 °C in air. Thus, in the present work, a SiOC ceramic coating could provide a new surface modification for the improvement of the interfacial adhesion between the thermally conductive filler and the matrix in which can enhance the thermal conductivity of the composites.