Unfortunately, when imperial rulers pay too much attention to the pleasures of the forbidden garden, they tend to ignore the needs of their empire, and let bureaucrats take over. Not always a healthy career move (often resulting in serious power struggles).
That from War in Human Civilization by Azar Gat. I highly recommend it, and probably will be writing on it more extensively. Warning, though: the author writes more extensively on the impact of the horse than the impact of harems.