What is that feeling...?

Crestfallen
It is to be sad and disappointed in one go.

adjective
1. dejected; dispirited; discouraged.
Dictionary.com

Other options include:

Dejected
sad and depressed

adjective
1. depressed in spirits; disheartened; low-spirited
Dictionary.com

Disappointed
displeased due to a failing in expectations

adjective 1. depressed or discouraged by the failure of one's hopes or expectations
verb
to defeat the fulfillment of (hopes, plans, etc.); thwart; frustrate
Dictionary.com

In simple words, rejection is what is actually felt. But consider these terms:

Friendzoned (or brotherzoned, in this case)

Although, not a term for the feeling per se, the concept is, in popular culture, referred to as getting friendzoned, where your love for someone is not reciprocated, and instead they say something along the lines of "I consider you as a good friend, and nothing more".

Read about friend zone on Wikipedia, and similar slang expressions on Urban Dictionary.

You could say "His hopes were dashed."

4 a : (ruin, destroy) "the news dashed his hopes"
https://www.merriam-webster.com/dictionary/dash

I guess there isn't a universal one-word-fits-all answer to this question, as it would depend on how the individual person feels. I would assume that most people would feel some sort of displeasure, but that could be sadness, anger, depression, annoyance, etc. Their feelings could also be aimed at different people. It could be aimed at themselves (because they might feel they could have done something differently that would have changed this person's mind), it could be aimed at someone else, or it could not be aimed at anyone or anything. Therefore there are many options, below are just a few (you may also like to look up some of these words, and those given by other answers in a thesaurus to find other potential options):

• Frustrate verb, Frustration noun, Frustrated adjective

  Feeling annoyed or less confident because you cannot achieve what you want - Cambridge Dictionary

  Especially appropriate for this situation because it can be used to describe an emotion that cannot be expressed. It can be used as a verb to describe her actions with respect to him (she had frustrated his advances), as a noun to define his feelings (his pent-up frustration seethed inside him) or as an adjective to describe his feelings (his frustrated love for her).
  It is possibly less appropriate if he is feeling more sad or depressed and not at all annoyed, as frustrated implies some sense of annoyance.

• Unsatisfied adjective

  If you are unsatisfied with something, you are disappointed because you have not got what you hoped to get. - COBUILD Advanced English Dictionary

  It doesn't necessarily imply much sadness, depression or annoyance, more of a neutral way of expressing his displeasure at the situation. E.g. He was left unsatisfied with the knowledge that she wanted to be his sister.

• Disheartened verb or adjective

  If you are disheartened, you feel disappointed about something and have less confidence or less hope about it than you did before. - COBUILD Advanced English Dictionary

• Discouraged verb

  to deprive of courage or confidence - Merriam-Webster

  Dishearten and discourage can be fairly synonymous, and could be used to describe a lack of confidence or hope with regards to the situation. E.g. He was disheartened by her lack of interest.
  Discourage is also often used to describe one person attempting to dissuade another from doing something. E.g. She discouraged him from even attempting to woo her.

• Unrequited adjective

  If love that you feel for someone is unrequited, it is not felt in the same way by the other person - Cambridge Dictionary

  Not used to describe his feelings particularly, but more to describe the situation - of her not reciprocating his feelings. E.g. His unrequited love for her left him feeling frustrated.