Early on, it is absolutely ogres. To low level or under equipped characters, that can basically be a death sentence. By the late game, though, much of that threat is gone, to the point the game may use two or so of them at a time.
Late game, it was the enemy mages. Often had reaver as a specialty, and if Templar game was not on point, there were a few spells which were devastating to someone fueling damage with health loss.