I think it's more a matter of the expectations for each type than the particular style, necessarily. With "popular" or genre fiction, especially of the best-seller type, it seems to me that those works are expected and meant a) to be fairly predictable, b) not to do much to challenge the status quo, and c) to end relatively happily. For instance, you could write the most well-written formula romance ever, but it can't end with both protagonists miserable or dead. When people read popular fiction (and they are often the same people who read "literary" fiction; I'm not constructing a hierarchy of audience) -- when people read popular fiction, they do so because they expect certain things and want a certain effect (they want to be comforted, or they want escape, or they want to be reassured, or they want to be scared in a safe way, or the want a cathartic good cry without too much misery, etc.) Thus writers of popular or genre fiction need to produce that effect. With so-called literary fiction, though, we're often reading for different reasons and with different expectations. Thus the works can be can be less predictable, less expected, can be unsettling or disturbing in ways that we don't usually want popular fiction to be.

There are exceptions, of course, but in general, I see the distinction as more one of what we expect each work to do and not necessarily about the complexity (or lack thereof) of style or plot or theme.