Do compound words have any logic?

It is interesting you should ask about logic when it comes to language. Sometimes logic may seem incompatible with language, but then it cannot be, since language is not chaotic. LinguisticPulse says:

Language is not governed by Logic in the formal sense, but even though it’s arbitrary we know it’s also not completely random. An individual language, like English, has particular consistencies about it.

M-W has a long article about compound words and I think there you will find an answer. I find this catchy sentence particularly helpful:

There aren't fast rules to forming compounds, but there are patterns.

These patterns may be included in what LinguisticPulse called consistencies.

The article speaks about the three ways of spelling compound words you mention:

Compounds are written in one of three ways: solid (teapot), hyphenated (player-manager), or open (which ranges from phrases such as off and on or little by little to combinations like washing machine—have a field day finding more). Because of the variety in formation, the choice among the styles for a given compound represents one of the most vexing of all style issues writers—and lexicographers—encounter.

So there is controversy about the matter, but let it not scare us.

For some terms, it is often acceptable to choose freely among open, hyphenated, and solid alternatives, even though the term has been used in English for an extended period (for instance, lifestyle, life-style, or life style). Although the styling that ultimately takes hold for a compound may be determined by nothing more than editorial and writerly preference, there are patterns of new compounds as they become established in the English language. Compound nouns, for instance, are usually written as one word; compound verbs are generally written as two; compound adjectives are often written with a hyphen. But note that we added "usually," "generally," and "often"—we're hedging.

The article goes on to tackle compounds in detail. It gives a pretty accurate idea about the dynamics of compounds formation in English.

Compound words are constructed according to a fundamental logic.

(CoGeL) [D] COMPOUNDING: adding one base to another, such that usually the one placed in front in some sense subcategorizes the one that follows; eg: blackbird, armchair, bott1e-feed […]

• long-term: the category is that of the terms; the subcategories are characterized by words such as "short", "mid" and others; therefore those terms come first: "short-term", "mid-term".

If you think carefully you see that a similar thinking process applies to all the following.

• snowball, mailbox, grandmother, basketball, skateboard, schoolhouse, peanut butter, ice cream

Two terms in the list, "mother-in-law" and "check-in", are not compounds. "Mother-in-law" is obtained through suffixation (suffix "-in-law") and "check-in" is obtained through so-called reconversion (of the verb "to check in")

Can I create my own compound words to be used in formal context? If so, is there any logic to figure out if my newly created compound word is closed-form, hyphenated or open compound word? And even the above list is not complete because you have to write "ice-cream cone", if I've understood correctly.

1/ Yes you can; however you have to do that in cases of sufficient necessity, when you discover a new concept, which, therefore has not yet been named. Do not forget, compounds have to be defined otherwise people do not know what they mean. So you can do that in a thesis , for instance, where the definition will be deduced from an explanation, or you can write a definition (not so usual).
2/ Of course, compounds can be used for compouding.
3/ There is no rule about hyphenation.