A new approach to solve SharePoint’s painfully slow spin-up / start-up time for first request

Posted at: 09:55 on 03 April 2009 by Muhimbi

MrSlowThis article was originally posted on our CTO’s personal blog. Due to the tremendous amount of positive feedback we have decided to update the post and move it to the Muhimbi blog, which has more of a SharePoint focused audience. Please check out our PDF Converter for SharePoint, URL Shortener and other SharePoint products if your find this article helpful.

Ever since SharePoint 2007 was introduced I have been really disappointed with the 2 minute spin-up time of the sites and the 30 second wait when launching STSADM.

How could Microsoft have released this? I thought, What is wrong with these people? Why is there no uproar in the SharePoint Community at large?

The SharePoint community quickly came up with workarounds such as the SPWakeUp tool on Codeplex, but these tools don’t actually fix the underlying problem and are of no help at all when you are developing a project that requires a SharePoint restart between compilations.

Before you continue reading you should realise that there are multiple factors that affect initial load time. There is the ‘loading of the assemblies and associated pre-compilation’ wait time, which can be solved by faster hardware, and then there is the problem outlined below, which may or may not be present in your environment. Read on to find out.

As it turns out this problem does not occur on all SharePoint installations, it only happens under a certain combination of circumstances. Fortunately I have identified the root cause as well as a solution.
 

Symptoms for STSADM:

  • You start STSADM without any parameters
  • There is a delay of about 30 seconds
  • While you are waiting, and tearing your hair out because your deployment script has about 60 STSADM commands, there is no CPU activity, swapping or significant network traffic.

Symptoms for SharePoint sites:

  • You make the first request of the day, or the first request after recycling the app pool because you are developing assemblies that sit in the GAC.
  • There is a delay of about 2 minutes
  • While you are waiting, and tearing your remaining hair out because you know you have to do this at least 50 times today, there is no CPU activity, swapping or significant network traffic.

So, what is going on here? Quite a few SharePoint developers are complaining about this on the Internet - and we even encountered the problem on the training disk images used by a renowned SharePoint training provider - but no-one has a real solution.

After yet another night of Googling around I found the solution in a posting about SQL Server 2005, which appears under certain circumstances to suffer from the same problem as SharePoint 2007. (Note that in my particular situation the problem is caused by SharePoint being slow, not SQL Server.)

The problem is that when loading signed assemblies the .net Framework checks the Internet based Certificate Revocation List. As our servers have, like most secure environments,  no outgoing connections to the public Internet, the connection to crl.microsoft.com times out after what appears to be 30 seconds. It probably does this a couple of times in succession, causing a 2 minute wait when spinning up SharePoint.

After the timeout the assembly is still loaded and the software works as expected, though very slow every time a new signed assembly is loaded for the first time, which happens a lot. The worst thing is that no entries are written to the event log and no exceptions are thrown so you are left completely in the dark about why your application is so bloody slow.

There are a couple of workarounds, which one works best is for you to decide:

  1. Add crl.microsoft.com to your hosts file and point it to your local machine. Some people have reported success with this, but it didn't work for me. 
     
  2. Allow your servers to directly connect to crl.microsoft.com. If your environment dictates the use of a proxy server, configure it using proxycfg. 
     
  3. Disable the CRL check by modifying the registry for all user accounts that use STSADM and all service accounts used by SharePoint. Find yourself a group policy wizard or run the vbscript at the end of this posting to help you out. Alternatively you can manually modify the registry for each account:

    [HKEY_USERS\<userid>\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing]
    "State"=dword:00023e00     
     
  4. Download the CRLs and add them to the server manually (I haven't tested this, but it may work):
     
    1. Download: http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl
      http://crl.microsoft.com/pki/crl/products/CodeSignPCA2.crl 
       
    2. Add them:
      certutil -addstore CA CodeSignPCA.crl
      certutil -addstore CA CodeSignPCA2.crl
       
  5. Mark Russinovich has written a detailed post about this problem and provides a 5th option, which involves a manual change to each web application’s web.config file. 
     
  6. Brett Waring pointed me to an article that describes how to reduce the timeout using  a group policy.
     

We decided to go for Option 3 (disable CRL check) and life is good again, the improvement is significant.

     

VBScript to apply registry change:

The following script applies the registry change to all users on a server. This will solve the spin-up time for the service accounts, interactive users and new users.

const HKEY_USERS = &H80000003
strComputer = "."
Set objReg=GetObject("winmgmts:{impersonationLevel=impersonate}!\\" _
    & strComputer & "\root\default:StdRegProv")
strKeyPath = ""
objReg.EnumKey HKEY_USERS, strKeyPath, arrSubKeys
strKeyPath = "\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing"
For Each subkey In arrSubKeys
     objReg.SetDWORDValue HKEY_USERS, subkey & strKeyPath, "State", 146944
Next

With thanks to Nik Shaw for the script. 

Please read the comments section below, it contains some interesting replies including changes required for the various Windows versions as well as a PowerShell equivalent of the above mentioned VBScript  code.

 

Relevant links:

Labels:

34 Comments:

  • Cool blog!! Hope you guys keep posts coming.
    And thanks for the article - I'm going to test the solution later today - will be a godsend if it works.

    Regards,
    Greg O

    By Anonymous Anonymous, At 06 April, 2009 11:31  

  • I used Option 4 (downloading the CRL's) but I'm curious about others' success with the other offered options. Please post or email me. It still takes about 20 seconds to open a page in the morning now, but I'd like it to be immediate, since it IS an INTRANET!

    Thanks.

    Christine
    lisi@mydelraybeach.com

    By Anonymous Christine Lisi, At 16 April, 2009 19:18  

  • Hi Christine, to get rid of the 20 second wait you will need to use the warm up scripts referenced at the end of the article.

    One of the reasons the first request in the morning is slow is because the Application pools are recycled over night. Have a look in the IIS Manager for the exact time. If you schedule your warmup script shortly after that then you should not notice a delay after making your first request of the day.

    By Blogger Muhimbi, At 16 April, 2009 19:24  

  • This looks like it is going to be a lifesaver, thanks!!!

    By Blogger The Tim, At 22 April, 2009 22:44  

  • This is absolutely WONDERFUL! Our site times went from 2 minutes to 10 to 15 seconds consistently for the first site load. Subsequent site loads (with no IIS Reset) are 5 to 8 seconds.

    Thank you so much. You a truly a lifesaver.

    By Anonymous Anonymous, At 23 April, 2009 16:11  

  • does this happen for every user page load?
    or
    the first peroson who calls after recycle of iis.

    thanks
    Janu

    By Blogger Unknown, At 12 May, 2009 23:15  

  • Hi Sunil,

    If you are suffering from this problem, it appears to happen whenever a new signed .net assembly (piece of functionality) is loaded for the first time in the associated application pool since the Application Pool was reset (usually nightly or after a configurable timeout)

    So depending on the number of Application pools you run (Usually 1 per SharePoint Web Application) you may incur this sever delay several times a day.

    Once a page has been loaded, repeated requests will not incur this delay.

    By Blogger Muhimbi, At 13 May, 2009 08:35  

  • #3 definitely worked for me! Thanks

    By Anonymous Rachael Harralson, At 09 July, 2009 16:54  

  • Great info !
    I have a suggestion for a easier workaround:

    You can edit the machine.config (usually C:\windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config) instead of editing every web.config, and adding the following code :

    <configuration>
    <runtime>
    <generatePublisherEvidence enabled="false"/>
    </runtime>
    </configuration>

    By Anonymous Dove, At 22 July, 2009 10:54  

  • thanks!
    editing HOSTS works for both SQL Server Management studio slow start up and Sharepoint slow load. just adding a line as below to HOSTS and go!
    127.0.0.1 crl.microsoft.com

    By Anonymous Anonymous, At 04 November, 2009 05:33  

  • Editing the host worked. Sped up the load time dramastically.

    By Anonymous Anonymous, At 17 November, 2009 20:36  

  • how do you configure this option please?

    2.Allow your servers to directly connect to crl.microsoft.com. If your environment dictates the use of a proxy server, configure it using proxycfg.

    By Anonymous Anonymous, At 16 December, 2009 17:31  

  • Well, this very much depends on your network infrastructure, which I naturally know nothing about. You either update your firewall to stop blocking http requests originating from the server or use proxycfg.

    In order to use proxycfg, login to the server using a normal account. Configure the proxy in internet explorer so it allows proper access for the current user, then copy those settings using "proxycfg -u".

    If this is not a feasible option then you may want to consider some of the other solutions provided in this posting.

    For more details about proxycfg use Google.

    By Blogger Muhimbi, At 16 December, 2009 17:51  

  • Wow. Thanks for putting all this info in one place.
    I am in an environment where servers have no internet connection and the initial load time was about 40 seconds. I also used the registry key fix. The performance difference was immediate and dramatic. My all virtual sharepoint environment is running extremely fast after this fix. Users noticed it immediately.

    By Anonymous Anonymous, At 18 December, 2009 20:55  

  • im finding a little difficult to determine the "userid" for the user accounts, How do i know which is which?

    [HKEY_USERS\\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing]
    "State"=dword:00023e00

    Alternatively, if i use the VBS script to apply changes will that take care of it or do i need to edit script or do anything more?

    Thanks

    By Anonymous Anonymous, At 04 January, 2010 10:46  

  • Just use the script to set it for all accounts. Alternatively look at the other options provided in this posting.

    By Blogger Muhimbi, At 04 January, 2010 10:53  

  • The script has worked for me FANTASTIC!

    I copied and pasted in a notepad and saved it as vbscript.vbs and double clicked, no message came up, is this how it should be?

    can you please confirm i only needed to run this once or does this needs to be scheduled to run everyday?

    Thankyou!

    By Anonymous Anonymous, At 05 January, 2010 10:17  

  • You (should) only need to run it once.

    By Blogger Muhimbi, At 05 January, 2010 10:48  

  • We're behind a proxy here which uses an automatic configuration script. One of our servers would wait 30 seconds whenever we executed an stsadm command.

    Here's what I did to fix it: I went to IE > Internet Options > Connections > LAN settings > checked "Use automatic configuration script" and entered the address for the proxy server.

    Hopefully it will help you guys out if you're behind a proxy server and having this same issue.

    By Anonymous Anonymous, At 02 February, 2010 00:57  

  • Thanks for your help.
    ... downloading the crls worked for us.

    By Anonymous Anonymous, At 16 March, 2010 04:19  

  • Editing the machine.config (usually C:\windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config) instead of editing every web.config, and adding the following code :







    DId for Me.....
    Thanks ALOt! Great Info.....:)

    By Anonymous Anonymous, At 12 May, 2010 10:11  

  • I can confirm that this speeds SharePoint 2010 up as well, even for machines that can access crl.microsoft.com.

    By Blogger Muhimbi, At 03 September, 2010 15:50  

  • hosts file fix worked for us

    By Anonymous Anonymous, At 16 September, 2010 21:20  

  • how about the dove's recommendation?
    is it okay to modify the machine config for all WFE alone?

    -----------
    Great info !
    I have a suggestion for a easier workaround:

    You can edit the machine.config (usually C:\windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config) instead of editing every web.config, and adding the following code :







    By Dove, At 22 July, 2009 10:54
    --------------------

    By Anonymous Anonymous, At 11 November, 2010 16:40  

  • Sure, why not. Anything suggested in this post, as well as its comments, is worth considering.

    By Blogger Muhimbi, At 11 November, 2010 16:42  

  • Thanks for your information!
    You saved my day !
    The vbs solution works perfect!

    By Anonymous Anonymous, At 02 February, 2011 11:45  

  • vbs ran successfully on my 2 servers. It immediately presented browsing performance boost. Great thanks!

    By Anonymous Anonymous, At 13 February, 2011 08:15  

  • Updated script - My Win2008 servers didn't error out, but also didn't set the registry value.

    Dim arrSubKeys, strWmiObject, objReg

    const HKEY_USERS = &H80000003
    const strComputer = "."
    const strKeyPath = "\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing"

    strWmiObject = "winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\default:StdRegProv"
    Set objReg=GetObject(strWmiObject)

    objReg.EnumKey HKEY_USERS, "", arrSubKeys

    For Each subkey In arrSubKeys
    objReg.SetDWORDValue HKEY_USERS, subkey & strKeyPath, "State", 146944
    Next

    By Anonymous Anonymous, At 02 August, 2011 21:54  

  • Thanks for the revised script.

    By Blogger Muhimbi, At 03 August, 2011 08:00  

  • Thank you 3 worked perfectly

    By Blogger jon, At 19 September, 2012 07:26  

  • I face same problem on one of my customer cause SharePoint servers not connected to internet so i decide to use 3rd option so i make powershell script to modify the registry for all account and it change the registry setting.

    i wish it help .here the script:

    # change registery for all users on HKEY_USERS to disable Disable the CRL

    # add the HKEY_USERS path on powershell
    New-PSDrive HKU Registry HKEY_USERS

    # change the Reg Stat Key to be 146944 on all users on HKEY_USERS
    Get-ItemProperty "HKU:\*\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing\" | ForEach-Object {Set-ItemProperty -Path $_.PSPath -Name state -Value 146944}

    By Anonymous Anonymous, At 03 November, 2012 22:50  

  • Thanks. option 3 worked.

    By Anonymous Anonymous, At 12 February, 2013 15:20  

  • Thank you very much!!! Worked fine on sharepoint 2010...

    By Anonymous Anonymous, At 11 September, 2013 16:42  

  • I have tried tonnes of workaround to solve the first time load issue, just name it, i tried SPWakeup, IIS Timeout to 0, disable CRL checking thru Security Policies, HKEY bla bla bla. Nothing work.

    The only things that work for me is to increase the Timeout at 15 hives in web.config in layout folder to 24 hours. Now my application load with speed of light, of course with help of SPWakeup in the morning too.

    By Blogger Awang Hafiz, At 09 May, 2014 14:43  

Post a Comment

Subscribe to Post Comments [Atom]