I only watched a really condensed version of the game with basically all breaks between plays being cut out. When did Brown go down? And how bad did it look?

I think Brown got hurt with 1:55 remaining on the 2nd quarter. It was the first play after the two minute warning and after Brees was awarded that late time out call on the previous play.

It was that 17 yard throw to Colston where Giordano, the FS from the Raiders, got hurt trying to defend the pass. The fact that Giordano was hurt probably caused the cameras to miss Brown hobbling off the field.

It seemed like Brown got tangled up with Evans trying to defend the Raiders' pass rusher, but it was kind of hard to really see since the camera moved away to follow Colston at the moment when Brown went down. On the next play where Graham fumbled, Harris was already on the field as the RT.

In any case, Brown was carted off to the locker-room, so I guess he couldn't put any weight on that ankle, but it was very hard to determine how severe the injury really was based on the TV feed alone.