Interview with Matt Cutts more than hints to the fact that the speed at which your site loads may start to enter into the search ranking factors. You can test the speed of your site and get access to many free tools at the this Google site speed page. We especially like the Page Speed Firefox app.