Where is the site hosted? What database does it use and where is that hosted?
Is there anything in particular happening at these problem times?
Have you tried running any performance tools on the system? Windows performance monitor is included and can help to highlight problem areas.
Without knowing more about what your website does, what the code is like, where it is hosted, what else is running on the same server, what browser you are using, the database type and hosting environment, other workloads and what else may be occurring any answer is going to be guesswork.
Even with intimate knowledge of a system unless you are doing performance monitoring any answer is pretty much guesswork.
Intellectuals solve problems; geniuses prevent them.
-- Albert Einstein