What is the meaning of the subclause of 'goes off the deep end'?

This is one of several idioms related to the dangers of deep water. These include:

go off the deep end - lose one's temper, act rashly or get carried away with something
throw someone in at the deep end - put someone in a challenging position without preparation
in deep water - in trouble, in a difficult situation

Your example doesn't quite fit the usual meaning of the idiom. Presumably they mean that if a real-time program becomes overloaded or goes wrong it will make the system unresponsive.

"Going off the deep end" just means going out of control. Synonyms are "going haywire", "going kablooey", and so on.