The material properties of hot mix asphalt (HMA) are modified by the amount and properties of the mineral filler (or filler) incorporated in the HMA. This paper focuses on the analysis of the filler effect on asphalt–aggregate interfaces of HMA based on thermodynamic properties (i.e., measurements of surface free energy, SFE, performed on asphalts, mastics (asphalt–filler combinations), and aggregates). Seven asphalts, three different mineral fillers added at different proportions, and six aggregates were assessed. The analysis was conducted in terms of energy parameters computed by using the SFE components of the materials evaluated. Corresponding results suggest that the inclusion of filler in the asphalt led to changes in the resistance to both fracture and moisture damage of the mastic–aggregate systems, and the wettability of the mastic over the aggregate as evaluated in terms of the energy parameters. Since these particular effects are not comprehensively captured based on conventional tests currently used for filler characterization—which mainly evaluate particle size, presence of harmful fines, and morphological properties, the HMA mix design can benefit from characterization of fillers and mastics in terms of the SFE and subsequent computation of the energy parameters included in this study.