A year or two ago I planted a Black Cherry tree, Prunus serotina. It has yet to flower and is only 5' tall but already it's attracting quite a bit of life.
I've actually read that these trees tend to be more popular with "pests" at this sage because trees of this size tend to be newer, representing the forest edge as it advances into meadow land. The lack of flowers means a lack of berries which are relished by birds. Adult trees of a flowering age get more bird attention because of the berries which perhaps makes young trees more tempting to certain bug species.
Another thought though is this might all be subjective observation. Younger trees still have their lower branches in tact and older trees form most of the new growth at the top, well out of view from the average person. Since caterpillars tend to target young growth more it could just be that we're not seeing it because it's up so high.