The Size of the Late Han Dynasty Harem

6,000 women.

Unfortunately, when imperial rulers pay too much attention to the pleasures of the forbidden garden, they tend to ignore the needs of their empire, and let bureaucrats take over. Not always a healthy career move (often resulting in serious power struggles).

That from War in Human Civilization by Azar Gat. I highly recommend it, and probably will be writing on it more extensively. Warning, though: the author writes more extensively on the impact of the horse than the impact of harems.