You might notice that your hard disk is constantly spinning on your Mac. If you check Activity Monitor, you might find out that the CPU is also being used up by the Finder, which typically doesn’t use a lot of CPU.
Turns out that the culprit could, in fact, be video files on your desktop. As you may know, OS X can play video files in their thumbnails and via Quick Look. To do this, it seems, your Mac needs to constantly be accessing the video file data, in order to have it ready to play at a moment’s notice. This eats up CPU cycles like nothing else on the desktop.