What does “balls” mean as a reply word or interjection?

"Balls!", and the synonymous "Bollocks!" which is more common in this usage, are essentially just vulgar exclamations usually expressing annoyance regarding a situation, or rejection of something previously said.

In my experience Americans ¹ rarely use either of the above. In terms of "vulgarity" it probably falls somewhere between "Dammit!" and "Fuck [it]!", but exactly where is subjective. In terms of semantic content when used as an exclamation, there's no meaningful reference to testicles.

¹ See this earlier question for confirmation that balls/bollocks are primarily BrE. Hemmingway often wrote things like "Don't talk balls!", where semantically balls/bollocks = nonsense, but I don't think I've ever heard an American use "[Oh] balls!" as a standalone exclamation (I've encountered AmE [Oh] bollocks!" with various spellings, but it's relatively rare on that side of the pond).

As to "used casually in conversation", that really depends on the social context and the normal vernacular of the speaker. But my guess is that in Archer's context, Richard uses balls because it's slightly less offensive than bollocks, which is more common among speakers who swear habitually (but because Richard doesn't habitually swear he doesn't know his usage is slightly non-standard).

Also note that Florentina has to ask him to repeat the word because she wasn't certain what he'd said. That's partly because even British women rarely hear or use either word (it's a male thing! :)

A clue is given in the text you quote:

“Balls,” he repeated.

As he's repeated that, there must have been a mention immediately before it. The line which refers to Richard saying something immediately prior to that is

Richard swore uncharacteristically

which gives some indication of the type of exclamation. One can then go to a dictionary.

1. Testicles.
2. [mass noun] Courage or nerve.
3. [mass noun] British Nonsense; rubbish (often said to express strong disagreement).

ODO

It can be seen that sense 3 is getting towards being appropriate, although disagreement may not be entirely correct. Balls can also be used as a swear-word expressing extreme disappointment or annoyance.

Dictionary.com has a fuller definition, from Collins, showing it as an interjection.

balls (bɔːlz)

— pl n

1. the testicles
2. by the balls so as to be rendered powerless
3. nonsense; rubbish
4. courage; forcefulness

— interj

    5.  an exclamation of strong disagreement, contempt, annoyance, etc

usage Both its anatomical senses and its various extended senses nowadays have far less impact than they used to, and seem unlikely to cause offence, though some older or more conservative people may object. Interestingly, its use in the sense of courage is exactly paralleled in the Spanish term «cojones»

Collins' usage note indicates that it can be used more or less casually, but because of the anatomical reference it might be unwise to use it in more sensitive company.

Balls: it is an exclamation of disappointment (interjection).

Richard is disappointed by the fact that they can't defeat Thomas by acquiring 51% of the stock since he will issue 2 million new shares making it impossible to beat him.

Oh balls!!!

• An expression of dissapointment and or frustration. "Oh balls, I lost my chemistry homework..."

• (Expression) The exclamation of great discomfort or anger. Usage: Ryan: Hey Adam you broke it Adam: Oh balls...

• As an interjection, the word ‘balls’ is used to show a person’s frustration or distaste at an occurrence, idea, or situation. Though typically many interjections evolve to become more socially acceptable (i.e. ‘fudge’ from the more stigmatized curse word ‘fuck’), the word ‘balls’ does not appear as a tamed down version of a more stigmatized, less socially acceptable word. Etymology : Though originally a phrase referring to male testicles, the word ‘balls’ has evolved into an interjection that demonstrates frustration or distaste.

Source:http://neologisms.rice.edu/index

Source:http://onlineslangdictionary.com