Is it your brothers who refer to me as an elder? I will say, that where I am from, one should only call another 'father,' or 'grandfather,' if they receive permission from that person. Otherwise it is considered disrespectful. 'Uncle' is a more casual term, which they could consider.