What's the difference between "stared at" and "stared off at?"

Per comments from Carlo and Janus, normally we only stare, look, peer, glare, etc. at things we can actually see.

We don't normally think of seeing things like distance, darkness, gloom, murk, etc. These are really words for the medium through which we [might possibly] see something.

There's scope for a degree of uncertainty here though. Although we normally stare at the horizon (which can be "seen"), and stare into the darkness or stare into the distance (abstract things that can't be "seen" as such), in all cases the links show a few instances using the "wrong" preposition.

In some cases (He looked at/into her eyes; She stared at/into the flames), both prepositions are possible (using into implies trying to discern something within or beyond the eyes/flames).

There's little difference between stare into the distance and stare off into the distance. Arguably, including off (less commonly, out, away) emphasises that you're not looking at anything more relevant that might be nearby (such as a person who's talking to you). But it's a fine distinction.

Prepositions help the reader or listener understand the relationship (the logical relationship, the spacial relationship or the temporal relationship between words or phrases. Often as not that relationship is further clarified by the context of the rest of the sentence. Consequently, their informative value may be so slight that the selection of the "right" preposition is purely idiomatic. One stands "on line" to purchase a ticked to see a film in London, but he stands "in line" to purchase a ticket in the United States. Similarly, if I spend the night "in Mary's house" there is a suggestion that Mary is not present and I am making sure her house is inhabited while she is away. If I spend the night "at Mary's house" she, or at least her family, is there as well. If I find myself sleeping "with Mary," then there some suggestion that Mary and I are sleeping in the same bed and may have had sexual congress. When learning a 2nd language the subtle differences in the selection of prepositions are often the last things one learns.