Full-stack vs Full Stack, Back-end vs Back end, Front-end vs Front end

The usual principle is that an attributive adjective gets hyphenated (it was a hard-fought victory), but a predicative adjective does not (the victory was hard fought).

The punctuation usually follows the syllabic stress. Note that in hard-fought victory, there is only one stressed syllable in hard-fought, whereas in the victory was hard fought, there are two stressed syllables in hard fought.

So you'd have front-end system.

The noun phrase front end shouldn't be hyphenated: both words are stressed.