Just wrap things up without any window dressing
It's often fine to just say you've got to go without any explanation, especially if you know the person already. They'll understand you've got things you need to do and won't be offended.
"I gotta run, good talking to you."
(Talking on the phone) "Well I'm gonna go. I'll talk to you later."
"Alright" (To agree with what they just said) "Anyway, take it easy man." (As you're heading off)
.