What do you call someone who is affected by an action, whether positively or negatively?

You call them the affected— as a noun, describing the person. That is the word of choice employed by persons who prepare legal documents filed in a court-of-law. It is the ontologically correct word of choice for characterizing someone who has been influenced positively or negatively by the action of another absent any influence of their own.

It is generally used in a question posed to determine the impact or degree an influence has had on the subject, the questioner being presumed as already knowing the nature of the influence.

For example: "How are you affected by the accident?" This question implies that an accident influence the subject negatively; however, it leaves the degree of injury caused to the subject ambiguous.

That's where the fine line is. So, to leave the nature of the influence ambiguous, you simply describe a person affected, and, like all words, used it in a context that does not hint in one way or the other.

patient; one who, or that which, is passively affected; a passive recipient.

However, since the sense meaning "A person or animal who receives medical or veterinary treatment" is now so much more well known, I would avoid it unless in technical contexts where it is more common or at the very least when it clear opposition to the agent that is performing the action.

I would rephrase to use something like "those affected" or similar.

If you were open to borrowing a word from the discipline of Visual Impact Assessment, I'd suggest Receptor.

The standard text for Landscape and Visual Impact Assessment is the Guidance for Landscape and Visual Impact Assessment 3rd ed (not available in digital format) which gives the following definition

Visual Receptors Individuals and/or defined groups of people who have the potential to be affected by a proposal.

The more usual definition of Receptor is

an element of the nervous system adapted for reception of stimuli, eg a sense organ or sensory nerve-ending
(Chambers)

Frustratingly my mobile App version of Chambers also gives 'Receiver' as the first definition, which the online version omits.

"It is important to think of your business's most important client groups and whether they're being helped or harmed".