Good point. When a team does well in non-conference, but not in conference, there are two ways this is spun, depending on the viewpoint: (1) the team collapsed after non-conference; (2) the conference is really, really difficult.
If it is football season and the team at issue is the SEC, then the answer is always number 2. If it is basketball season and the team at issue is in the ACC, then the answer is also always number 2. If the team at issue is in the Pac 12, well… the notorious east-coast bias and lack of a major partnership with ESPN would suggest that the tendency would be go to with answer number 1.