Is “Thisness” an established English word? What is the alternative expression that sounds more natural and familiar?

Thisness is a nonce word formed by adding the suffix -ness to the pronoun this. Normally pronouns can't take suffixes, which is why thisness is not really a formally accepted word. The meaning is clear enough, though, which is why the New Yorker was willing to use it in a quote.

If you're looking for a formally accepted alternative, you could use quiddity or haecceity. These words have the opposite problem of thisness: hardly anybody knows these words or understands what they mean, but they are nonetheless words which you can find in the dictionary.

Thisness is not a commonly used or understood word, as is evident from the cited text where it is first used in quote marks and then followed by two whole paragraphs to explain, with examples, what it really means.

It's rareness also explains why it is listed in the OED, which defines usage and lists many more words than in "everyday" dictionaries. This is not necessarily a bad thing; see my answer about therf werre eyght bokes for why you wouldn't want the OED as your spellchecker.

A synonym could be realism, believability, and antonymic phrase is breaking the fourth wall: for example when an actor in a play or television programme suddenly turns to us, the viewers, and addresses us directly -- it jolts our suspension of disbelief.

And finally, another thing that demonstrates its uncommonness is the difference in meaning between James Wood's personal definition of thisness ("the selection and elaboration of seemingly irrelevant detail is what makes fiction real") and that defined by the OED and Wikipedia (haecceity, "The quality of being ‘this’ (as distinct from anything else)").

The OED records thisness as meaning ‘The quality of being ‘this’ (as distinct from anything else)’ and gives haecceity (from Latin haec) as a synonym. The earliest citation for ‘thisness’ is dated 1643.

The 13th-century philosopher Duns Scotus has much to say on thisness (although, of course, he used the word haecceitas as he was writing in Latin). As far as I know, scholars writing on Duns tend to use both terms, thisness and haecceity, but the latter is far more technical.