What do I call doing something complex without writing/visual help?

Idiomatically,

The best programs we have could not break the encryption but you are saying he can do it in his head?

From Oxford Dictionaries Online Phrases, definition 29 (heads up @Julie Carter)...

in one's head - by mental process without use of physical aids

i.e. - without needing to use a pencil and paper for complex calculations.

I think you are referring to mental calculation:

• Executed or performed by the mind; existing in the mind: mental calculations.

Mental calculation:

• comprises arithmetical calculations using only the human brain, with no help from calculators, computers, or pen and paper. People use mental calculation when computing tools are not available, when it is faster than other means of calculation (for example, conventional methods as taught in educational institutions), or in a competitive context. Mental calculation often involves the use of specific techniques devised for specific types of problems. (Wikipedia)

Ngram calculate mentally

• What should I put (in mind), what should I memorise.

"He's such a brilliant student! He can calculate derivatives on the fly whereas his classmates have to write down all the steps in their notebooks."