Two factors were identified in a study several years ago (sorry, don't remember the reference):
1) The perceived complexity of an operation influences the user's expectations.
2) In a sequence of operations necessary to achieve a goal, the user grows increasingly patient.
No rocket science, but validated by the study.
This means that simple operations like moving a cursor, selecting an object, scrolling, etc. must be as fast as possible. Providing adequate feedback whenever instantaneous response is crucial.
Please let me know if you remember any clues on how to find the study you're referring to.