Number of report items and number of returned data rows from the data source definitely affects the load time of the report. It doesn't matter that the data is retrieved for less time from your database, it still needs to be processed by the reporting engine.
You can optimize your report layout and the data you display by following the suggestions in Performance Considerations
Also Excel has some specifics and in order to export to Excel faster, you need to perfectly align all of your report items (e.g. by using a Table) so it does not generate extra columns/rows. Please check Excel Rendering
article - http://www.telerik.com/help/reporting/designing-reports-considerations-excel.html