Should I say "have a good night" at 5:00 PM?

I don't see why it would be inappropriate to say 'have a good night', although I admit it does sound odd to my ears also. As an Australian English speaker and office worker, I'd rarely if ever heard anyone say this at the end of the work day, until a colleague from Canada arrived. He says it regularly to anyone leaving, even as early as 15:30 or 16:00.

Personally, I find 'have a good night' to be a strange thing to say on a regular weekday, when most people would typically be doing nothing worthy of note in the evening. If I knew a colleague was going to see a concert or a play, I might say 'have a great night' or similar. Otherwise, 'see you tomorrow' is perfectly reasonable in my opinion and about all you'll get from me.

Yes, though I think you're over-analysing it. By saying "have a good night", you're wishing them well for the entire remainder of the day, which is more informal than simply saying "see you tomorrow". An alternative would be "have a good evening".

"Enjoy the rest of the day/evening"

How about "Have a nice evening"?