Should the words "city"/"state"/"province" be capitalized (if not followed by the name of the city)?

In your second example, "city" should not be capitalized. Words for governmental or administrative units are only capitalized when they are used as part of a proper noun, such as the formal name of a city.

Your first example is correct so long as you're referring to the City of New York, as the formal name for New York. However, if I were simply using the word "city" to disambiguate and not as part of a formal name, I wouldn't capitalize it:

We're only going to the city of New York, not the rest of the state.

If I saw the City I would assume that what was meant was the City of London (which is not at all the same thing as London City). In other contexts, city should be spelled with a lowercase c.

City/state/province etc are qualifiers. On their own they don't specify anything specific and therefore aren't proper nouns. New York City is a vague virtual identity, meaning what it needs to mean to a particular writer. The "City" qualifier being a filler so it rolls off the tongue easier. Everyone knows New York is a city or else you can disambiguate in lowercase. It changes to uppercase when you have proper nouns like "New York City Council". Even in the case of the council only used when writing about THE New York City Council, as opposed to writing "New York city council and other city councils are considering the use of garbage cans." If we were to capitalize qualifiers referring to conceptual proper nouns, there would be capitals everywhere, like camels walking in every sentence. Making reading harder rather than easier. King George Whiting The Second went on an Abbreviated Holiday, spending his time in Elverston and Surrounding Area Park Range.