The sense of "inside diameter" is probably from the the Arabic meaning of "mold for casting bullets":

Caliber:

• 1560s, "degree of merit or importance," a figurative use from Middle French calibre (late 15c.), apparently ultimately from Arabic qalib "a mold for casting." Arabic also used the word in the sense "mold for casting bullets," which is the oldest literal meaning in English. Meaning "inside diameter of a gun barrel" is attested from 1580s. Barnhart remarks that Spanish calibre, Italian calibro "appear too late to act as intermediate forms" between the Arabic word and the French.

(Etymonline)

Caliber was originally used to mean a unit of weight for cannon shot. So, for example, you might have "12 pounds caliber". Much later it came to mean a bore size, but until about 1800 guns were always sized by the weight of the shot.

The analogy to human characteristics refers to weight, not size. So, someone or something of "high caliber" would be someone weighty or serious.