There are a myriad theories that revolve around this one. To wit:
The ivy in the term is referring to the ivy-covered walls of the historic buildings of these institutions (See image)
One theory leads us to believe that over a century ago, an interscholastic athletic league was formed that comprised of Harvard, Yale, Princeton, and a fourth school that keeps changing with the stories . It was officially called "IV League"; IV standing for Roman numeral four.
Most agree though that the term came from the sports world when New York Tribune sportswriter in 1933 who while describing the inter-university football season wrote, "A proportion of our eastern ivy colleges are meeting little fellows another Saturday before plunging into the strife and the turmoil".
Source: Wikipedia, Google Images