If you were there first, the coworker should go elsewhere.
If the coworker was there first, you could be polite and leave but the coworker should still go elsewhere. What kind of a weirdo has a personal conversation on the phone with other people around?