He's a smaller QB with a weaker arm, odds are if he had to play most of his games outside in places with less than ideal weather or Sean Payton asked him to do things besides the short passing game, things that hide his physical limits, he wouldn't have had the same career. The other greats in the game haven't been so obviously limited.
You are underrating the hell out of Brees. His arm might not have been the best, but his deep ball accuracy in his prime was amazing. His pocket presence/mobility (underrated in general), anticipation, and accuracy are phenomenal. As far as pure accuracy goes, there haven't been that many QBs better.
Brees has a 90 QB Rating outdoors, and it would probably be even higher if you didn't include his early San Diego years when he was young. Weather has nothing to do with how good he is. Another thing that gets overlooked with Brees is that, compared to some other elite/great QBs, he never had the weapons they had. He had Jimmy Graham for some years and his best wide receiver was Marques Colston (who was solid but nothing special) but otherwise, he made the most out of throwing to guys like Devery Henderson, Lance Moore, etc.
As far as the question, no Brees isn't even close to the argument. But I do think as time passes, he will become ridiculously underrated. He was an amazing QB who should have won MVP in 2009, has always done his best with teams that otherwise would have been horrendous, and when he does get in the postseason, plays marvelous. I don't know why people are holding lack of victories or stuff like that against them. People have no issue calling Marino a top 5 QB and it isn't like he was making the playoffs every year. Brees balls out in the playoffs and it isn't like he didn't win a Super Bowl.
65%, 3539 yards, 7.6 Y/A, 24 TDs, 6 INTs, 100 QB Rating in 11 Postseason Games
He is a first ballot Hall of Famer. At least he should be. I just think he gets overlooked because he played alongside arguably the 2 greatest quarterbacks to date.