What's a good opposite of "keep track"?

Actually, lose track of is quite a common phrase:

to lose contact with someone; to forget where something is. I lost track of all my friends from high school. Tom has lost track of his glasses again.

Yes, you can say 'lose track'. It doesn't sound funny.

"Lose track" is perfectly acceptable, in the sense of misplacing or losing contact information or forgetting what you meant to be doing, but in the sense of "something happened and [you] lost [a thought]", you might rather say you were distracted by other thoughts, or sidetracked by other activities.

A colloquial phrase for being lost track of is "dropping off the radar". [See my Update 1 comment]