At the end of each number / agent row sits a settings cog icon button.
Click this to open the edit window for the number / agent in question.
A delete button can be found in the bottom left of this popup edit window.
Once clicked, it will prompt you to make sure your happy to delete. If confirmed, the number will be permanently deleted