Rhatigan, along with his comrades in Vimmark, have been driven insane from eating darkspawn flesh as a deranged ritual to better attune to their dark master's will, whom they seek to free from his prison; an odd desire for the Carta, whose members pride themselves on business savvy and brutality.The Grey Warden Janeka in fact contacted the Carta under Rhatigan control to capture Hawke and assist as armed support at the prison tower. Janeka also entrusted Rhatigan with The Hawke's Key to manipulate the prison's seals.
This battle isn't too hard if you play it right but you can take massive damage if you aren't careful. Note that upon entering his chamber you cannot return home to Kirkwall until the campaign is completed (although injuries can be healed by blowing the party summoning horn on Sashamiri's Floor, so stock well before entering and make your supplies last.
Second, the traps are deadly so be very careful where you're standing at all times. The bonus is that you can also use them to your advantage.
If you gather your party and then move immediately to the second level it will buy you some time. Position your party in the center and throw some immediate AOE spells on the snipers while your rogue flips the switch on the trap trigger. This trigger will activate the upper level buzzsaw traps that run the entire length of where the snipers are standing, cutting them to pieces. Activate the trigger repeatedly to deal far more damage than any circular or semi-circular area-of-effect attack possibly could. After enough hits they will transition to the lower ground if they haven't been eliminated, which will certainly happen on Nightmare difficulty, but it will least drive them out of a direct line-of-sight. The trap lever at the far end of the second level will trigger a spike trap that runs the length of their secondary position downstairs, so use this trap to finish off the Carta snipers.
With the bronto charging around the lower area and unable to follow you upstairs (at least for a time), only Rhatigan will be a problem. He fights like a true two-handed warrior, using Mighty Blow, Scythe, and Whirlwind to his advantage, particularly in the confined space of the upper walkway. Have a tank keep him occupied while the rest of your party picks apart any remaining snipers and the bronto from the relative safety of the ramp. Watch the bronto's movements carefully and run out to activate the lower level trap triggers, timing them to the point that the bronto will be running across their area-of-effect. The bronto should run in circles downstairs for a while but eventually he may successfully get upstairs, so be ready with freezing or slowing spells to keep him at bay long enough to finish off Rhatigan.