"One can do his homework in the library"

In my opinion, using one in this sense is grammatical but awkward. I don't think it is entirely a matter of formality or that the usage has fallen out of practice (although, comparing one can, he can, she can, they can on ngrams does tell an intriguing story). I think the use of one is comparable to the use of the passive voice: both are clear and technically correct, but simply strained.

The best solution is to use a more specific word than one.

A student can do his or her homework in the library.

A touch of awkwardness remains. An easy resolution is to replace his or her with their (if you accept the singular they).

A student can do their homework in the library.

Alternatively, we can just make the subject plural. After all, the library should accommodate multiple students.

Students can do their homework in the library.

These days, one seldom uses the subject "one", meaning the impersonal "one", though one generally still understands when others use it. It sounds stilted and old-fashioned. It's too bad that it has gone out of use. I liked it.

"One" used to be archaic.

But these days, "one" has become important again, as a non-gendered pronoun. Third party singular pronouns are in short supply. If one is to avoid linguistic horrors such as singular "they", then any usable word is in play. "One" has the advantage of being singularly singular; that's its very name!

You are right, however; "one" has gotten a bad reputation, from a particular style of intentional misspeaking: addressing another in the third party (he, she, one) instead of the second-party (you) as one properly should.

Presuming you are talking to the person doing homework, then "One should do ones homework in the library" is stilted on purpose; that's the whole point of saying it that way. Proper would be:

• You can do your homework in the library -- remove the haughty formality and address the person directly. The "your" is superfluous, unless you actually mean to say "do your own homework, not Crabbe and Goyle's".

• One can do homework in the library -- but in this case we are speaking generally about third parties doing homework: say, to the Library Committee. This makes the second pronoun very redundant, so I've dropped it. This particular phrasing is gender neutral and singular by design. (It is distinct from "Students can do their homework in the library", because that implies multiples of students working together would be OK.)

• If we were speaking about a specific third party, we could just use gendered pronouns if that person has a gender. If our goal is to avoid gendered pronouns, then avert pronouns altogether (as I did in the italics just here).

In any case, one should feel free to use "one" as needed to refer to an actual third party; but make an effort to avoid expressions that make it sound stilted or overly formal.

I'd agree that the use of "one" sounds old-fashioned or formal. A more colloquial way of saying it would be

You can do your homework in the library.

or

They can do their homework in the library.

or even

Homework can be done in the library.

depending on your intent.

The first is a general statement that people in general can do their homework in the library. The second is more referring to one specific person, but without using that person's gender. It might be a response to a question such as, "Where can my friend do homework?"

The third version is simple passive voice.