In my experience, difficulty increases each year until senior, which is relatively easy. The reason for this can be seen in the rough layout of my alma mater’s (Penn State, admittedly a while ago) course requirements:
Freshman year is tough because it is a new school and a more rigorous pace and the students are newly independent, but it is basic math and physics and general education courses and really is not that hard.
Sophomore year gets harder because hardly anyone has ever seen this material before and it gets into a fair amount of depth. This was also the year that you get your first core EE course, and however hard the science and math were to learn, applying them is harder. A lot of people get weeded out this year, especially by that first EE course.
Junior year gets harder still, because there were 5 different core courses you had to take: Circuits, E&M, Controls, and Systems & Signals (all junior-level), as well as a Sophomore level CompE course. Each of the EE courses are harder than that intro EE course and they all require different skills, so almost everyone struggles in at least one of the four - at one point, the department realized that only about 1 out of 4 students were completing the series on their first try, although it was surmised that some who dropped one of the courses would have passed but chose to repeat it for a higher grade. Regardless, this series of courses was widely regarded as trying even for those who had “breezed” through the first two years.
Senior year… is easier. It is easier because your senior year is all about specialization, and no specializations require you to be good at everything so students largely choose specializations that match their strengths. And while some of the courses are hard, most are about applying what you already know so the added difficulty level just isn’t that high - you’ve already learned the hard stuff, this should be the stuff you enjoyed and want to do as a career.
Now, that assumes you take the courses in the suggested order, and may certainly vary between schools - it all depends on what is put into which courses in which years.
I don’t think there is a clear answer here. Some of the material gets harder simply because it is more complicated, and some gets harder because it is much more abstract - I knew a ton of people who struggled with E&M simply because you can’t see electromagnetic waves and are extremely hard to visualize. The workload increases and some higher level of intelligence is required for some of the material, but I don’t think that overall it can be pinned down to any one thing.
Also, as the courses get harder, the resources required to complete them increases, which means students need more of those resources to succeed - intelligence, time management, discipline, good study habits, etc. When students fail at higher levels, it can be because of deficiencies in any or all of those areas - smart kids who don’t study fail more often than kids of average intelligence who study well, for example. So students don’t struggle or fail because of one consistent reason, but rather because of a variety of individual reasons that mean they didn’t bring to college everything they needed to succeed.