Following is the list of various factors which can speed up your website.
CSS File size: - The CSS can be optimized by using shortcuts, grouping selectors with the same properties and values together.
HTML code: - A valid HTML/CSS will have smooth code in achieving better speed.
Images Size: - A well designed web page will have multiple images. It is better to remove unused images from the code. For other images, optimize them by converting them into .PNG. The use of jpeg files is advised when high quality photographs need to be displayed. Also, have the actual image size be same as the display size.
Javascript files size: - Java script files need to be compressed so that the size is reduced. If any JS file is required for a particular web page, include that file only on that page rather than on the general template.
Caching: - Caching is very important for dynamic websites where the page content is fetched from the database. Caching reduces the load time by saving the static version of the dynamic page and display that without any database query.

According to Google, if your site is well maintained with speed accuracy it adds another factor for search engine rankings.