A mental model is based on a user’s belief, not the actual facts. Essentially, it is a model of what users think they know, not what actually exists. So a user's mental model of a website or application is basically how they think it should work, what they expect the various interface components to do, how they assume they can navigate around, etc. The site’s actual behavior may differ, and this is where users can get confused.
Users are now so accustomed to surfing the web, and visiting big sites like Amazon, Facebook, and Google, that they expect other sites to behave in the same way. They expect Search to be like Google. They expect user profiles to behave like Facebook, and so on.
This can raise a number of issues. First, depending on the software platform being used, the interface and user experience may be very different. A mobile app on Android might behave differently than one on the iPhone, for example. Depending on the project budget and timescales it may not be practical to do anything about it. It also may not be functionally appropriate to make these type of changes. But tools like NucliOS, the IguanaUIpack, and NetAdvantage Ultimate can give you a good start.
It’s important to remember, some aspects of users’ mental models will be correct, and those aspects we should try to conform to. Things like navigation layout, labeling of generic items, breadcrumbs, basic page layouts, and the like. Other intuitions might be wrong, or based on outdated principles. This is where, as developers, we should try to break out and innovate. It’s a tricky balancing act to get the correct mix of established and new, but that is what keeps UI and UX fresh… and us working hard!