The effect of molarity of alkali activator, manufactured sand (M-sand), and quarry dust (QD) on the compressive strength of palm oil fuel ash (POFA) and fly ash (FA) based geopolymer mortar was investigated and reported. The variable investigated includes the quantities of replacement levels of M-sand, QD, and conventional mining sand (N-sand) in two concentrated alkaline solutions; the contents of alkaline solution, water, POFA/FA ratio, and curing condition remained constant. The results show that an average of 76% of the 28-day compressive strength was found at the age of 3 days. The rate of strength development from 3 to 7 days was found between 12 and 16% and it was found much less beyond this period. The addition of 100% M-sand and QD shows insignificant strength reduction compared to mixtures with 100% N-sand. The particle angularity and texture of fine aggregates played a significant role in the strength development due to the filling and packing ability. The rough texture and surface of QD enables stronger bond between the paste and the fine aggregate. The concentration of alkaline solution increased the reaction rate and thus enhanced the development of early age strength. The use of M-sand and QD in the development of geopolymer concrete is recommended as the strength variation between these waste materials and conventional sand is not high.