Really can’t blame them. They’ve had long stretches were both football and basketball have been bad. Their school is ridiculously small (only about 7,000 undergraduates) which doesn’t help build up an alumni or fan base.
Not to mention that there are many other schools in California that people can grow up beings fan of and have a much better chance of attending.
They just have too much working against them. Though I suspect their recent success in football has improved things some what.