What is the inverse term for a proxy?

The word you are looking for might be principal. Although the more usual pairing is principal-agent, I believe that principal is correct in such contexts as proxy voter, which is nearly what you have in your example.

I am not sure about this, but can we take delegator as the one who delegates?

According to Black's Law Dictionary Online, "principal" is the correct word for the person who delegates power to an agent or proxy. Black's was first published in Britain; now there are American editions as well. In American law schools, Black's is the traditional authority.