Hex, curse, spell, jinx, charm

Two points about HP as a source of English words: as a work of fiction, it has its own rules. For example, 'Charms' is a separate subject at Hogwarts, and the Basilisk is a large snake: neither applies (necessarily) in other contexts. And because it is set in a 'magic-heavy' environment, the author had to find different words for spells and casting them, in the same way that an English teacher encourages students to use different words for 'said'.

I am not sure that there is in fact any difference in the words when referring to spells; the witches in Macbeth say "Now the charm's wound up", which is certainly not a positive reference. But really the only person who can give a clear answer would be a witch or wizard...

Edit to expand on the last sentence: there is no 'correct' answer, any more than there is to 'What colour is a unicorn?' You can quote books and folklore, but ultimately the only sensible response is 'whatever was in the mind of the person imagining it'.

Here are the etymologies from most recent words to older (from etymonline.com). I recommend reading them as these words these days might have lost much of their delineation - today we label them under one category and often use them interchangeably.

jinx (n.) 1911, Amer.Eng., from 17c. jyng "a charm, a spell," originally "wryneck," a bird used in witchcraft and divination, from L. iynx "wryneck," from Gk. iynx. The verb is 1917 in Amer.Eng., from the noun. Related: Jinxed; jinxing.

hex (v.) 1830, Amer.Eng., from Pennsylvania German hexe "to practice witchcraft," from Ger. hexen "to hex," related to Hexe "witch," from M.H.G. hecse, hexse, from O.H.G. hagazussa (see hag). Noun meaning "magic spell" is first recorded 1909; earlier it meant "a witch" (1856).

charm (v.) c.1300, "to recite or cast a magic spell," from O.Fr. charmer, from L.L. carminare, from L. carmen (see charm (n.)). Related: Charmed; charming.

curse (n.) late O.E. curs "a prayer that evil or harm befall one," of uncertain origin, perhaps from O.Fr. curuz "anger," or L. cursus "course." Connection with cross is unlikely. No similar word exists in Germanic, Romance, or Celtic. The verb is O.E. cursian; meaning "to swear profanely" is from early 13c. Related: Cursed; cursing. Curses as a histrionic exclamation is from 1885. The curse "menstruation" is from 1930. Curse of Scotland, the 9 of diamonds in cards, is attested from 1791, but the origin is obscure.

In the actual meaning and use, they can all be used as verbs.

Jinx, hex and curse can have exactly the same meaning and it is negative; while charm is ambiguous: it can refer to both positive and negative.

The older words curse and charm have developed other, common meanings, to swear and to attract, respectively, while the younger words have only meanings related to magic and mysticism.

For more nuance:

• a person casts a hex on another person or an object in order to make them (or it) do something, or to limit them in some manner (a hex to make someone keep their mouth shut or to make a stone fly). This is usually in a negative sense.

• a person casts a charm to affect the properties of something or someone (to make someone love someone else, or to keep a door from closing. This can be negative or positive. It is closely related to a hex, possibly a hypernym (a superset concept). Both a hex and a charm are somewhat temporary in nature.

• a curse is a debilitating pronouncement. You break in to King Tut's tomb, you might be cursed to die of dysentery (a pronouncement that was promised to occur on some violation of a condition rather than it just happens no matter what).

• a jinx, well in modern AmE, it seems only to be used as a reflection that a stigma has been placed inadvertently somehow (not on purpose), by mentioning a bad occurrence word one is jinxing a situation leading the bad alternative more likely to happen ('knocking on wood' helps dispel a possible jinx). The most common usage is when you repeat the same word at exactly the same time as someone else, one of them gets to 'jinx' the other into silence (the other is not allowed to speak until allowed by the first).

This taxonomy/ontology of magical incantations is somewhat problematic, similar in nature to an ontology of ESP powers, in that it is an ontology mostly of psychological artifacts rather than testable physical phenomena.

As far as I know,

Charm mean a spell that affects the way the charmed person is, often psychologically, but in a intrinsic way.

Curse (as a spell) mean a spell that affects what would happen to the cursed person in some future or any time he does certain action.

Hex is a spell, but the type of spells only witches can cast. And I make a difference between wizard and witch. Often a wizard is seen as a person who uses or practices magic that derives from supernatural or occult source, while a witch has certain tendency to evil. That is because witch is a word instaured by the church (should get a citation somewhere).

Jinx is related to a type of curse placed on a person that makes them prey to many minor misfortunes and other forms of bad luck.

If I have time I'll edit this with some citations.