Joe, I think you're using diversity as a scapegoat, be careful about that.
USC and UCLA are nearly as diverse as St Johns. Attendance at college sports is like any other business though; you need to compete in the marketplace. Have a good product, market it well, set competitive prices, distribute it appropriately. We've had a terrible product for a decade, for a long time the athletic department was a mom and pop shop when it came to PR and marketing, and the campus isn't connected to the arena where we play most of our games. That's a recipe for failure.
I would imagine you could be very successful even if you have one or two of those things going against you, but not when you can't check any of the boxes.