OP, I transferred from a two year LAC affiliated with a mid sized university to a state flagship. I was one of three people (four if you include the professor) who shared my academic interests. To me a small school was suffocating. I went to a suburban high school of over 3,000 so it was a shock to go to a tiny, rural school.
Although my first year at the large state flagship was rough, I have few regrets about transferring. Even though the percentage of intellectual students is far lower at my school than LACs like Reed or Harvey Mudd, the fact that tens of thousands of people attend mean that the gross number is far higher.